Pervasive Computing - Pervasive 2011

Lecture Notes in Computer Science Commenced Publication in 1973 Founding and Former Series Editors: Gerhard Goos, Juris Hartmanis, and Jan van Leeuwen

Editorial Board David Hutchison Lancaster University, UK Takeo Kanade Carnegie Mellon University, Pittsburgh, PA, USA Josef Kittler University of Surrey, Guildford, UK Jon M. Kleinberg Cornell University, Ithaca, NY, USA Alfred Kobsa University of California, Irvine, CA, USA Friedemann Mattern ETH Zurich, Switzerland John C. Mitchell Stanford University, CA, USA Moni Naor Weizmann Institute of Science, Rehovot, Israel Oscar Nierstrasz University of Bern, Switzerland C. Pandu Rangan Indian Institute of Technology, Madras, India Bernhard Steffen TU Dortmund University, Germany Madhu Sudan Microsoft Research, Cambridge, MA, USA Demetri Terzopoulos University of California, Los Angeles, CA, USA Doug Tygar University of California, Berkeley, CA, USA Gerhard Weikum Max Planck Institute for Informatics, Saarbruecken, Germany

6696

Kent Lyons Jeffrey Hightower Elaine M. Huang (Eds.)

Pervasive Computing 9th International Conference, Pervasive 2011 San Francisco, USA, June 12-15, 2011 Proceedings

13

Volume Editors Kent Lyons Intel Labs, Intel Corporation 2200 Mission College Blvd. Santa Clara, CA 95052, USA, E-mail: [email protected] Jeffrey Hightower Google, Seattle 651 N 34th Street Seattle, WA 98103, USA E-mail: [email protected] Elaine M. Huang University of Zurich Department of Informatics Binzmühlestrasse 14 8050 Zurich, Switzerland, E-mail: [email protected]

ISSN 0302-9743 e-ISSN 1611-3349 ISBN 978-3-642-21725-8 e-ISBN 978-3-642-21726-5 DOI 10.1007/978-3-642-21726-5 Springer Heidelberg Dordrecht London New York Library of Congress Control Number: 2011929025 CR Subject Classification (1998): C.2, H.4, D.2, H.5, H.3 LNCS Sublibrary: SL 3 – Information Systems and Application, incl. Internet/Web and HCI

© Springer-Verlag Berlin Heidelberg 2011 This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, re-use of illustrations, recitation, broadcasting, reproduction on microfilms or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable to prosecution under the German Copyright Law. The use of general descriptive names, registered names, trademarks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. Typesetting: Camera-ready by author, data conversion by Scientific Publishing Services, Chennai, India Printed on acid-free paper Springer is part of Springer Science+Business Media (www.springer.com)

Preface On behalf the Program Co-chairs as well as the entire Organizing Committee, we welcome you to the proceedings of Pervasive 2011—the 9th International Conference on Pervasive Computing. This year’s conference was held in San Francisco, California, and marked the first time that this premiere forum for research in the field of pervasive and ubiquitous computing was held in the USA. Pervasive 2011 received a total of 93 submissions to the paper track, consisting of 70 full-length paper submissions (up to 18 pages) and 23 note-length submissions (up to 8 pages). A rigorous review process was conducted by a Program Committee of 27 recognized experts in the field of pervasive computing from 10 different countries and from both academia and industry. Every submission was evaluated in a double-blind review process by at least two Program Committee(PC) members and two external reviewers. In all, 174 external reviewers participated in the process in addition to the committee. The review phase was followed by an online discussion in which both the PC members and external reviewers took part. The final discussion and subsequent selection of technical program papers and notes happened during a two-day PC meeting in December 2010 in Seattle, Washington, hosted in part by Intel Labs Seattle through the use of their facility. Ultimately, 22 submissions were selected for inclusion in the technical program, among them 19 full papers and 3 notes, for a total acceptance rate of 23.7%. The selected papers are the work of 96 different authors from 34 international industrial and academic institutions. As in previous years, Pervasive 2011 showcased a wide range of research activities in addition to the technical program that is presented in this volume. This year’s categories of participation included a full-day of workshops prior to the technical program, as well as videos, demonstrations, and posters to accommodate the presentation and discussion of research in ways appropriate to its current state. Additionally, a Doctoral Consortium for senior PhD students occurred in conjunction with ISWC 2011, the co-located 15th International Symposium on Wearable Computers. The accepted submissions in these additional categories are not published in this volume, but can be found in the adjunct proceedings for Pervasive 2011. Pervasive 2011 was the direct result of the dedicated effort of numerous volunteers. We want to thank the Conference Committee members for their hard work and attention to detail in making sure each aspect of the conference came together. The Program Committee and reviewers worked diligently to assemble a terrific program. We also wish to thank the staff of events for their assistance with the management of the conference and our sponsors for helping make Pervasive 2011 a success. June 2011

Kent Lyons Jeff Hightower Elaine M. Huang

Organization

Conference Committee Conference Chair Program Co-chairs

Demos

Posters Workshops Videos Publications Publicity Local Arrangements Web Sponsorship

Kent Lyons, Intel, USA Jeff Hightower, Google, USA Elaine M. Huang, University of Zurich, Switzerland Tico Ballagas, Nokia Research Center, Palo Alto, USA Daniela Rosner, UC Berkeley, USA Oliver Amft, TU Eindhoven, The Netherlands Kurt Partridge, PARC, USA Mirco Musolesi, University of St. Andrews, UK Alexander Varshavsky, AT&T Labs, USA Daniel Roggen, ETH Zurich, Switzerland Gerd Kortuem, Lancaster University, UK Fahim Kawsar, Bell Labs, Belgium and Lancaster University, UK Andreas Bulling, University of Cambridge, UK Trevor Pering, Intel, USA Nirmal J. Patel, Georgia Tech, USA Thad Starner, Georgia Tech, USA

Program Committee Andreas Bulling Alexander Varshavsky Antonio Krger Bashar Nuseibeh Chris Schmandt Daniel Avrahami Florian Michahelles Frank Bentley Hao-Hua Chu James Scott Jens Grossklags Jin Nakazawa John Krumm Jon Froehlich Judy Kay Kay Connelly Kurt Partridge

University of Cambridge, UK AT&T Labs, USA DFKI and Saarland University, Germany Open University, UK MIT, USA Intel Labs, USA ETH Zurich, Switzerland Motorola, USA National Taiwan University, Taiwan Microsoft Research Cambridge, UK Penn State, USA Keio University, Japan Microsoft Research, USA University of Washington, USA University of Sydney, Australia Indiana University, USA PARC, USA

VIII

Organization

Leila Takayama Lena Mamykina Mike Hazas Minkyong Kim Nic Marquardt Patrick L. Olivier Rene Mayrhofer Shin’ichi Konomi Shwetak Patel Tico Ballagas

Willow Garage, USA Columbia University, USA Lancaster University, UK IBM T.J. Watson Research Center, USA University of Calgary, Canada Newcastle University, UK Upper Austria University of Applied Sciences, Austria Tokyo Denki University, Japan University of Washington, USA Nokia Research Center, Palo Alto, USA

Steering Committee A.J. Brush Hans Gellersen Anthony LaMarca Marc Langheinrich Aaron Quigley Hide Tokuda Khai Truong

Microsoft Research, USA Lancaster University, UK Intel Research, USA ETH Zurich, Switzerland University of St. Andrews, UK Keio University, Japan University of Toronto, Canada

Reviewers Wael Abd-Almageed Sharad Agarwal Manfred Aigner Fahd Albinali Swamy Ananthanarayan Lisa Anthony Lora Appel Ismail Arai Daniel Avrahami Tico Ballagas Luciano Baresi Aaron Beach Marek Bell Hrvoje Benko Frank Bentley Alastair Beresford Claudio Bettini Jon Bird Jan Borchers Gaetano Borriello Nick Brachet

Stephen Brewster Gregor Broll Leah Buechley Andreas Bulling Tiago Camacho Andrew Campbell Ricardo Chavarriaga Ling-jyh Chen Guanling Chen Yu-Chung Cheng Kunigunde Cherenack Mauro Cherubini Marshini Chetty Keith Cheverst Tanzeem Choudhury Marc Christie Hao-Hua Chu Jaewoo Chung Elizabeth Churchill Anthony Collins Kay Connelly

Sunny Consolvo David Cooper Scott Counts Landon Cox Florian Daiber David Evans Eyal de Lara Dave Dearman Anind Dey Travis Deyle Tawanna Dillahunt Sandra Dominikus Steven Dow Naranker Dulay Schahram Dustdar Nathan Eagle David Evans Benjamin Fabian Benedict Fehringer Steven Feiner Mirko Fetter

Organization

Laura Forlano Jodi Forlizzi Adrian Friday Jon Froehlich Raghu Ganti Lalya Gaye Sven Gehring Hans Gellersen Joy Ghosh Daniel Greenblatt William Griswold Jens Grossklags Svenja Hagenhoff Michael Haller Masahiro Hamasaki Raffay Hamid Mike Hazas Chantel Hazlewood Jennifer Healey Sumi Helal Urs Hengartner Steve Hodges Jaap-Henk Hoepman Jesse Hoey Eve Hoggan Paul Holleis Lars Erik Holmquist Gary Hsieh Polly Huang Bret Hull Masugi Inoue Stephen Intille Shamsi Iqbal Sibren Isaacman Giulio Jacucci Lee Joonhwan Wendy Ju Gerrit Kahl Eunsuk Kang Ashish Kapoor Stephan Karpischek Fahim Kawsar Judy Kay Ashraf Khalil Danish Khan

Sunyoung Kim Minkyong Kim Donnie Kim Jen King Mikkel Baun Kjærgaard Predrag Klasnja Andrew Ko Shin’ichi Konomi Vassilis Kostakos David Kotz Sven Kratz Christian Kray John Krumm Antonio Kr¨ uger Tsvi Kuflik Bob Kummerfeld Kai Kunze James Landay Nicholas Lane Marc Langheinrich Eric Larson Karin Leichtenstern Jonathan Lester Yang Li Kevin Li Lin Liao Zhigang Liu Clemens Lombriser Hong Lu Paul Lukowicz Markus L¨ ochtefeld Julie Maitland Lena Mamykina Jennifer Mankoff Natalia Marmasse Nicolai Marquardt Sergio Matos Yutaka Matsuno Rene Mayrhofer David McDonald Alexander Meschtscherjakov Florian Michahelles Daniel Michelis James Mickens

IX

Masateru Minami Anurag Mittal Iqbal Mohomed Mounir Mokhtari David Molyneaux Meredith Ringel Morris Ann Morrison Floyd Mueller Emerson Murphy-Hill Mirco Musolesi J¨ org M¨ uller Tatsuo Nakajima Jin Nakazawa David Nguyen Petteri Nurmi Bashar Nuseibeh Eamonn O’Neill Daniel Olguin-Olguin Patrick Olivier Wei Tsang Ooi Antti Oulasvirta Joseph Paradiso Chris Parnin Kurt Partridge Shwetak Patel Sameer Patil Don Patterson Nick Pears Thomas Pederson Matthai Philipose Andrew Phillips Gian Pietro Picco James Pierce Zach Pousman Bodhi Priyantha Daniele Quercia Ahmad Rahmati Ramesh Ramadoss Yvonne Rogers Daniel Roggen Stephanie Rosenthal Romain Rouvoy Dan Saffer Michele Sama Shunsuke Saruwatari

X

Organization

Quan Sasaki Takeshi Sato Andreas Savvides Bernt Schiele Chris Schmandt Albrecht Schmidt Stacey Scott James Scott Peter Scupelli Julian Seifert Mubarak Shah Yi Shang Pravin Shankar Elaine Shi Josh Smith Timothy Sohn Frank Stajano Thomas Strang Leila Takayama

Yuri Takhteyev Poorna Talkad Sukumar Desney Tan Anthony Tang Karen Tang Nick Taylor Thiago Teixeira Bruce Thomas Andrea Thomaz Tammy Toscos Koji Tsukada Joe Tullio Keisuke Uehara Ersin Uzun Jan Van erp Kristof Van Laerhoven Alexander Varshavsky Jo Vermeulen Nicolas Villar

Hongan Wang Evan Welbourne Kamin Whitehouse Stephen Whittaker Andy Wilson Jake Wobbrock Woontack Woo Oliver Woodman Michael Wright Kazuo Yano Koji Yatani Chuang-wen You Jaeseok Yun Lin Zhong Brian Ziebart John Zimmerman

Table of Contents

Practices with Smartphones Planning, Apps, and the High-End Smartphone: Exploring the Landscape of Modern Cross-Device Reaccess . . . . . . . . . . . . . . . . . . . . . . . . Elizabeth Bales, Timothy Sohn, and Vidya Setlur

1

Understanding Human-Smartphone Concerns: A Study of Battery Life . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Denzil Ferreira, Anind K. Dey, and Vassilis Kostakos

19

Sensing at Home, Sensing at Work Monitoring Residential Noise for Prospective Home Owners and Renters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Thomas Zimmerman and Christine Robson A Longitudinal Study of Pressure Sensing to Infer Real-World Water Usage Events in the Home . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jon Froehlich, Eric Larson, Elliot Saba, Tim Campbell, Les Atlas, James Fogarty, and Shwetak Patel Exploring the Design Space for Situated Glyphs to Support Dynamic Work Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fahim Kawsar, Jo Vermeulen, Kevin Smith, Kris Luyten, and Gerd Kortuem

34

50

70

Predicting the Future Learning Time-Based Presence Probabilities . . . . . . . . . . . . . . . . . . . . . . . . . John Krumm and A.J. Bernheim Brush

79

n-Gram Geo-trace Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Senaka Buthpitiya, Ying Zhang, Anind K. Dey, and Martin Griss

97

Location Sensing Autonomous Construction of a WiFi Access Point Map Using Multidimensional Scaling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jahyoung Koo and Hojung Cha

115

XII

Table of Contents

Identifying Important Places in People’s Lives from Cellular Network Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Sibren Isaacman, Richard Becker, Ram´ on C´ aceres, Stephen Kobourov, Margaret Martonosi, James Rowland, and Alexander Varshavsky NextPlace: A Spatio-temporal Prediction Framework for Pervasive Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Salvatore Scellato, Mirco Musolesi, Cecilia Mascolo, Vito Latora, and Andrew T. Campbell

133

152

Augmenting Mobile Phone Use Using Decision-Theoretic Experience Sampling to Build Personalized Mobile Phone Interruption Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Stephanie Rosenthal, Anind K. Dey, and Manuela Veloso SpeakerSense: Energy Efficient Unobtrusive Speaker Identification on Mobile Phones . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hong Lu, A.J. Bernheim Brush, Bodhi Priyantha, Amy K. Karlson, and Jie Liu Text Text Revolution: A Game That Improves Text Entry on Mobile Touchscreen Keyboards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Dmitry Rudchenko, Tim Paek, and Eric Badger

170

188

206

Pervasive Computing in the Public Arena Pervasive Sensing to Model Political Opinions in Face-to-Face Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Anmol Madan, Katayoun Farrahi, Daniel Gatica-Perez, and Alex (Sandy) Pentland Lessons from Touring a Location-Based Experience . . . . . . . . . . . . . . . . . . . Leif Oppermann, Martin Flintham, Stuart Reeves, Steve Benford, Chris Greenhalgh, Joe Marshall, Matt Adams, Ju Row Farr, and Nick Tandavanitj

214

232

Public Displays Hybrid Prototyping by Using Virtual and Miniature Simulation for Designing Spatial Interactive Information Systems . . . . . . . . . . . . . . . . . . . Yasuto Nakanishi, Koji Sekiguchi, Takuro Ohmori, Soh kitahara, and Daisuke Akatsuka

250

Table of Contents

Designing Shared Public Display Networks – Implications from Today’s Paper-Based Notice Areas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Florian Alt, Nemanja Memarovic, Ivan Elhart, Dominik Bial, Albrecht Schmidt, Marc Langheinrich, Gunnar Harboe, Elaine Huang, and Marcello P. Scipioni

XIII

258

Hands on with Sensing Recognizing the Use of Portable Electrical Devices with Hand-Worn Magnetic Sensors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Takuya Maekawa, Yasue Kishino, Yasushi Sakurai, and Takayuki Suyama 3D Gesture Recognition: An Evaluation of User and System Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Michael Wright, Chun-Jung Lin, Eamonn O’Neill, Darren Cosker, and Peter Johnson

276

294

Sensing on the Body Recognition of Hearing Needs from Body and Eye Movements to Improve Hearing Instruments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Bernd Tessendorf, Andreas Bulling, Daniel Roggen, Thomas Stiefmeier, Manuela Feilner, Peter Derleth, and Gerhard Tr¨ oster Recognizing Whether Sensors Are on the Same Body . . . . . . . . . . . . . . . . . Cory Cornelius and David Kotz Sensing and Classifying Impairments of GPS Reception on Mobile Devices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Henrik Blunck, Mikkel Baun Kjærgaard, and Thomas Skjødeberg Toftegaard Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

314

332

350

369

Planning, Apps, and the High-End Smartphone: Exploring the Landscape of Modern Cross-Device Reaccess Elizabeth Bales1 , Timothy Sohn2 , and Vidya Setlur2 1

University of California San Diego, 9500 Gilman Dr., La Jolla, CA 92093, USA 2 Nokia Research Center, 955 Page Mill Road, Palo Alto, CA 94304, USA [email protected],{tim.sohn,vidya.setlur}@nokia.com

Abstract. The rapid growth of mobile devices has made it challenging for users to maintain a consistent digital history among all their personal devices. Even with a variety of cloud computing solutions, users continue to redo web searches and reaccess web content that they already interacted with on another device. This paper presents insights into the cross-device reaccess habits of 15 smartphone users. We studied how they reaccessed content between their computer and smartphone through a combination of data logging, a screenshot-based diary study, and user interviews. From 1276 cross-device reaccess events we found that users reaccess content between their phone and computer with comparable frequency, and that users rarely planned ahead for their reaccess needs. Based on our findings, we present opportunities for building future mobile systems to support the unplanned activities and content reaccess needs of mobile users.

1 Introduction In the past several years the number of personal devices a user owns and interacts with has increased. Mobile phones, laptops, desktops, slates, and in-car navigation systems are becoming increasingly popular in the daily life of a user. In a previous study of multiple device usage, Dearman and Pierce found that users interact with as many as 5 personal devices a day [13]. With multiple devices, a user’s data often becomes fragmented based on the usage pattern and affordances of each device. A mobile phone will have history of phone calls, applications opened, and websites visited that are different than activity on another device. The fragmentation of digital activity creates a challenge for the user to transfer and reaccess content across their devices. Cloud computing has offered promise to enable consistent data access on any device. Services such as Evernote [4], synchronized bookmarks, Dropbox [3], and Chrome-tophone [2] all offer tools for users to transfer content from one device to another. These tools are designed to support planning practices, where a user recognizes information he will need later and saves it for easy reaccess. Users can sometimes forget the information they will need later, or choose not to plan ahead to preserve flexibility. These unplanned situations are often addressed by attempting to access web content by performing web query searches [24]. Web content is one of the primary sources of information today, especially as web applications that support productivity tasks are becoming increasingly popular. Both the K. Lyons, J. Hightower, and E.M. Huang (Eds.): Pervasive 2011, LNCS 6696, pp. 1–18, 2011. c Springer-Verlag Berlin Heidelberg 2011 

2

E. Bales, T. Sohn, and V. Setlur

computer and mobile phone are important devices in a user’s ecosystem that provide access to web content. There have been a number of studies analyzing the types of web content and searches that users perform both on their desktop and mobile devices [17,18], but few have studied reaccess patterns across these devices. The explosion of mobile applications has also added a new dimension of content reaccess because the same web content can be accessed through a web browser or a dedicated mobile application. In this paper we explore both the methods and content of web information reaccess among ones personal devices. We conducted a two week study with 15 users of high-end smartphones: iPhone, Android, N900. We used a combination of interviews, url logging, and a screenshot diary study to gather insights into cross-device reaccess patterns regardless of the method they used to access the content (e.g., web browser or mobile app). We measured cross-device reaccess by matching URLs and comparing timestamps to determine which access occurred first. This process required matching many of the URLs manually because mobile websites have different URLs than their desktop counterparts. We only considered two URLs a match if the content they referenced was the same. Our logging software captured over 123,497 web accesses on the computer and 3,574 web accesses on the mobile phone. Over the course of the study participants submitted 128 screenshots from in situ moments when participants noticed they were reaccessing content they had seen before. We captured over 1,200 cases where content was reaccessed on a device different from the original access device, with over 500 reaccesses originating on desktop and over 700 originating on the mobile device. The results of our study show that: – Cross-device reaccess, moving from computer to phone and from phone to computer, occurs with comparable frequency. – Reaccess is often unplanned. – Native applications are an important part of how users reaccess content. Informed by these results, we discuss several opportunities to support content reaccess among a user’s personal devices.

2 Related Work There are three areas that researchers have explored the types of content mobile users access. These can roughly be divided into information needs, search patterns, and crossdevice explorations. 2.1 Mobile Information Needs Studies on mobile information needs have used diary study methods to gather ecologically valid data about the types of content mobile users look for. Sohn et al. found that mobile users attempt to address many of their information needs through web access or other online resources that may have been previously seen [24]. In a similar study, Dearman et al. found that mobile users would often look to online resources to address their mobile information needs, but the process could sometimes be difficult and cumbersome [12]. Church and Smyth looked at the intent behind mobile information needs

Planning, Apps, and the High-End Smartphone

3

and found many information needs were related to finding PIM data, hinting that the data is related to content already seen by the user [10]. These diary studies hint at mobile users relying more upon connected resources through the cloud and understanding their re-access patterns would provider further insights into assisting mobile users in limited attention environments. 2.2 Search Behavior and Revisitation Patterns There have been a number of studies investigating web search behavior on both desktop and mobile devices. Many desktop studies have conducted query analysis on search logs reporting on query length and categorization [16,7]. Spink et al. conducted a longitudinal study of query behavior between 1997 and 2001 [25]. As smartphones have evolved over the years, users are accessing content through desktop and mobile web browsers. To investigate this trend, Kamvar and Baluja conducted a large-scale analysis of mobile search queries and found that mobile users with less featureful phones submitted shorter queries [17]. In a follow up study they found that iPhone users in particular behave differently than other smart phone users [18]. Their research revealed that iPhone users create search queries more like desktop computer users. We believe that this trend towards higher end smartphones being used more like computers alters how mobile users reaccess content across their devices and the type of content they reaccess. In addition to search behavior, studies have shown that web revisitation accounts for 58% [26] to 81% [11] of all desktop web site visits. Obendorf et al. found that 50% of desktop web revisits occurred within 3 minutes, while the other half took place much later [22]. Adar, Teevan, and Dumais looked deeper into the intent behind revisits [6] and found a variety of revisitation patterns. When studying how users reaccess content across devices, the analysis becomes multifaceted. A single piece of content can be accessed through a desktop URL, mobile URL, or a mobile application. As far as we are aware, few researchers have studied reaccess patterns across multiple devices, specifically when the content can be accessed through a web browser or mobile application on a high end smartphone [19]. 2.3 Cross-Device Interaction Researchers have explored how users manage their life with multiple devices. Dearman and Pierce conducted a study into how users interact with all their computing devices [13]. In a study of 14 Windows Mobile phone users, Kane et al. found that users frequently visit websites on both their phone and laptop/desktop machine, suggesting that sharing web history among these devices could be beneficial [19]. In a later study, Karlson et al. looked at situational constraints that mobile users face while using their device [20]. Participants were asked to take screenshots whenever they encountered a barrier on their mobile phone. They suggest the idea of decomposing tasks into subtasks so users can complete them across their devices based on their situational context. Neither of these studies looked at the effect of web reaccess on high-end smartphones and content that can be accessed through a mobile application. Several systems have created ways for users to plan ahead and share data between their mobile device and their desktop machine. The Context Clipboard uses a clipboard

4

E. Bales, T. Sohn, and V. Setlur

metaphor where users can place notes on the clipboard from their desktop and it is synchronized with their mobile device [15]. Gurungo uses the concept of mobile data types to identify key data that a user may want to access later on and sends the content to the device through Bluetooth [14]. There are also a number of commercial tools available today to support re-accessing content. These tools tend to support planned activities, where users knows ahead of time the content they will need later. Evernote [4] and Dropbox [3] both enable file sharing through the cloud. Googles bookmarks for maps and websites as well as the Chrome-to-phone extension let users synchronize data with their mobile device [2]. Firefox Home synchronizes bookmarks, tabs, and web history between desktop and mobile Firefox clients [5] supporting unplanned activites, where a user does not plan ahead for the content they need. The PIE system also supports unplanned activities by allowing users to search for files and documents on their devices [23]. We build upon this work by specifically exploring re-access patterns among high-end smartphone users. The high quality of smartphone interfaces and always-on connectivity have changed how phones are used today, with many phones being used more like desktops. We focus specifically on the frequency of cross-device reaccess by device type, the amount of preplanning users performed for content they reaccessed, and the role of mobile applications in content reaccess. The following sections describe our study design and results from our exploration of content re-access patterns.

3 User Study Gathering ecologically valid data from mobile users is challenging. We wanted to gather data from the moments of reaccess on a mobile device or on a computer, but placing an observer in the field to shadow a user can be time intensive. Logging methods are useful, but as mobile applications have become much more prevalent to access web information, the content remains siloed from the data-logging processes. As a result we used a hybrid approach of logging and a diary study to capture data in situ. Websites represent a majority of content users may want to access on their device, so we focused mainly on studying web content reaccess through a web browser or mobile application. The following sections describe our methods for obtaining ecologically valid data about the web content that user’s reaccess. 3.1 Participants We recruited 15 smartphone participants (7 iPhone, 4 Android, and 4 N900) through an advertisement on Craigslist from a city in the United States1 . Due to the sensitive nature of the data collected we experienced a relatively high attrition rate during our recruiting process. We also found it more difficult to recruit Android and N900 users compared to iPhone users which affected our overall recruitment numbers. Our Android users used a variety of phone models that run the Android software platform including the Nexus One, T-Mobile Cliq, and Motorola Droid. All iPhone participants used either the 3G or 3Gs models. Participants ranged in age from 22 to 50 years (µ: 35) and had 1

City is anonymized for submission.

Planning, Apps, and the High-End Smartphone

5

Fig. 1. Example of the same content accessed on through a native mobile application(left) and through the traditional web interface(right)

a wide variety of occupations including students, nanny, financial analyst, engineer, and freelance writer. We focused our recruiting on high-end smartphone users to understand how users of computer-like mobile phones manage content reaccess between their personal devices. Previous research found that users of high-end smartphones with computer-like capabilities behave differently than other smartphone users [18]. As a result, we chose three high-end smartphones with a modern web browser and an available set of mobile applications. All references to participants in this paper are anonymized with i1-i7 representing iPhone owners, a1-a4 representing Android owners, and n1-n4 representing n900 owners. 3.2 Procedure In order to gather in situ data from our participants, we used both a logging and screenshot capture method. The logging part of the study allowed us to observe the URLs that a participant visited on their desktop and some mobile devices. We developed a Chrome browser extension to log URL accesses on participants’ laptops/desktops. The extension logged the URL, timestamp, and page title each time a participant navigated to a webpage. We did not save any content from the web page due to privacy reasons. The data collected by the extension was automatically sent to a server in our research facility. We also used device specific methods, discussed later in this section, for extracting the URL history from each user’s mobile device so that it could be compared with the Chrome browser log data. With the explosion of applications available on a mobile device, users have multiple ways to access web-based content. Many applications act as native clients to web-based content and keep content in silos from other applications. It is difficult to observe content that may be accessed on a laptop through a web browser (e.g., Facebook website) and then on a mobile device through a specialized application (e.g., Facebook application). These types of reaccesses are also important, so we asked users to take screenshots when reaccessing content on their mobile device. Participants annotated these

6

E. Bales, T. Sohn, and V. Setlur

screenshots with additional comments about their reaccess event later through a nightly journal. Our study design is similar to the idea of snippets [9,8] , where users capture screenshots in the moment and annotate them in depth later on a PC. Figure 1 illustrates an example of the same content viewed through a dedicated mobile application and on the standard web interface. Participants attended a 1 hour in-office visit, filled out nightly online journals about their reaccess activity for the day, and participated in a semi-structured final interview two weeks after their start date. During the first in-office visit, we installed the Chrome browser and extension on the user’s laptop. N900 and iPhone users were instructed to take screen shots on their mobile devices and all users were instructed to take screenshots on their computer. Android phones do not have a screen capture function, so android users were asked to use a note application to document reaccesses on their mobile devices. We sent participants daily reminders with a link to an online journal where they could elaborate on their screenshots and reaccess stories they collected throughout the day. During the initial visit we performed a detailed walkthrough of the process of creating and annotating screenshots with the participants. We installed URL-logging software on the Android and N900 devices that uploaded the same information as the Chrome extension. This data was automatically sent to the server in our research facility. Because of device limitations on the iPhone (i.e., Mobile Safari does not allow browser extensions), we used an alternative method for collecting URL access events. iPhone participants sent us weekly phone data backups that would contain their URL history information. At the end of the two week study we conducted an exit-interview with the participants. The interview followed a semi-structured format and asked participants about their screenshots and reaccess patterns. Participation were compensated $80 USD at the end of the study.

4 Results and Observations We collected a total of 123,497 (µ: 6370 min: 775 max: 33892) web page visits on the computer and 3,574 (µ: 215 min: 28 max: 745) web accesses on the mobile phone. Of those webpage visits 14,642 were unique URLs on the computer and 260 unique URLs on the mobile phone. Table 1 shows a breakdown of average number of URLs accessed per user by device type. Android participants tended to browse more pages on the computer, while iPhone participants browsed more web pages on their mobile phone. Within-device reaccess of data, defined as reaccessing data on the device of original access was observed on all device types with iPhone users averaging 98.4, Android users averaging 9.8, and N900 users averaging 79 within-device reaccesses. Withindevice reaccess on the user’s personal computer averaged 6683.73 over all users. To study cross-device reaccess we matched the URL history from both devices to find access patterns. We considered two URLs a match when they accessed the same content, even if one was a mobile page and one was the full page (ex. m.cnn.com and www.cnn.com would be considered the same content even though one is the mobile URL and one is the standard URL). For password protected pages such as social

Planning, Apps, and the High-End Smartphone

7

Table 1. Total number of URLs access per user by device type Device iPhone

computer µ: 6370 min: 1482 max: 11667

mobile µ: 327 min: 31 max: 745

Android

µ: 14854 min: 775 max: 33892

µ: 182 min: 28 max: 449

N900

µ: 2169 min: 2527 max: 7344

µ: 142 min: 34 max: 309

networks and email we had to rely solely on the URL to determine if the content was the same. In total there were 1276 cross-device reaccesses (µ: 35, min: 4 max: 378), with 754 starting on the phone with reaccesses on the computer, and 522 starting on the computer with reaccesses on the phone. Table 4 shows the frequency and direction of reaccess for each of the smartphone participant classes, as well as the most common content reaccessed by direction. To gain a better understanding of what type of content users reaccessed we manually analyzed all the cross-device reaccessed URLS to identify the top categories. We categorized the reaccess events from the logs into website categories based on a scheme proposed by BBC [1] . The most frequent reaccesses were related to social network (e.g., Facebook) and news websites (e.g., New York Times). Information articles were also a common category of reaccess (e.g., Wikipedia). We also gathered temporal data about each logged reaccess event. In most cases the time to reaccess information varied between several minutes and several days (Table2). Most reaccesses were short term reaccesses, with the second access occurring on the same day as the initial access event. It is likely that the clustering of reaccesses in the short range time frame is influenced by memory, with reaccess that take place over a longer period of time being easy to forget to complete. This is especially likely when we take into account the methods participants used to remind themselves to reaccess data. WIth most participants using systems that depended on temporally affected interface, Table 2. Temporal Information of Cross-Device Reaccesses. Percentage represents percentile error. Time is displayed in hours:minutes:seconds. Device Direction Phone to Computer iPhone Computer to Phone

25% 02:38:19

50% 05:47:13

75% 06:21:05

90% 08:28:41

00:45:19

07:22:27

08:18:41

10:31:17

Phone to Computer

00:14:52

02:19:45

05:25:26

05:52:16

Computer to Phone

02:33:12

04:08:23

05:31:46

06:29:27

Phone to Computer

01:38:16

03:17:58

04:03:17

04:48:28

Computer to Phone

01:12:17

03:46:24

04:29:13

05:17:57

Android n900

8

E. Bales, T. Sohn, and V. Setlur

such as email inboxes, it is likely that if they waited too long to reaccess, the email would be pushed down off the main screen and forgotten. The following sections describe our observations around user’s current methods to synchronize content among their devices, the role of planning in reaccess behaviors, and the role of native applications in content reaccess. 4.1 Current Tools Do Not Adequately Support Cross-Device Reaccess We observed a variety of methods for sharing content among one’s personal devices. Through the nightly journals and interviews, we learned that our participants use many creative, sometimes cumbersome, methods to make their devices interact with each other. Participants expressed pride when sharing their clever syncing solutions. However, even those who were proud of their solutions noted that the methods were time intensive and often reserved for tasks where they could foresee an obvious return on their invested effort. Table 3 shows a list of the different practices employed by our participants as evidenced from their nightly journals, screenshots and interviews. The common theme among these practices was storing the information in a place for easy access later. Tools that synchronize easily across devices were used more heavily than others, but these methods were not particularly created for save and retrieval purposes (e.g., browser tabs). Email was a common place for our participants to put content they would need later. Many of our participants used a web-based email system that allowed them to access their data anywhere. In addition to emailing oneself links or files for later, users would also repurpose features to save content for reaccess. Marking an email as unread was a common example of repurposing a feature that was not necessarily meant for that purpose. One user also reported using the Facebook ‘like’ button to populate her ‘news feed’ with items she wanted to reaccess later. She knew that Facebook was easy to access from any device, and her news feed would be readily available to find the item she was looking for. The methods shared by our participants required some amount of planning to save the needed data in a place for later access. If user’s did not plan ahead, they would attempt to recreate web search queries in order to find the content they needed. Search can work effectively, but can also present additional hurdles when the technology does not behave the way the user expects. For example, Participant a3 encountered search results on his phone that were “completely different” than the results he got on his computer, making it hard for him to find the information that he wanted to reaccess. He expected the same results he had seen before, but the search engine he used displayed different results on the computer and mobile versions. Even if the user plans ahead, there is a high recovery cost when restarting a task or trying to find content previously seen. Bookmarks were one way to tag content to access later. However, as the number of bookmarks increases, users need to sift through large amounts of data to find their information. For some participants this lead to frustration. “I feel like bookmarks are buried, like I have thousands of bookmarks. I have bookmarks for car stuff, I have bookmarks for vegan stuff, I have bookmarks... ” (Participant a2) Some of these methods (e.g., browser tabs) act as a reminder tool to reaccess information later, which can be useful reinforcement. However, participants still need to

Planning, Apps, and the High-End Smartphone

9

Table 3. Methods for content reaccess shared by users. Many methods require the user to plan ahead for content reaccess. Method Email

Description Email applications automatically sync messages across devices. Users often depended on this feature to find content they had seen before.

Repurposing Features built for other purposes were overloaded by users to identify items for Features reaccess later. Common examples included emailing content to themselves and using the “mark unread” feature in email to mark a read email that the user wished to return to. Browser Tabs Leaving browser tabs open on the mobile device as a reminder to reaccess them on another device was a common user strategy for remembering to reaccess content. Paper

Paper for reaccess was used by several participants to help sync their devices. Informations was handwritten or printed, carried between the devices, and inputted on the second device to reaccess content.

Bookmarks

Shared bookmark systems were utilized by several users to share data between devices. Using these systems users could save a bookmark on one device and have it be available on their other device automatically.

Search

Unplanned reaccesses were frequently executed by entering search queries into another device.

manually enter the information into each device. Our participants expressed a need to overcome these challenges and have an easy method to reaccess their data. 4.2 Cross-Device Reaccess Happens in Both Directions We found that content reaccess occurs frequently in both directions between the mobile phone and computer (Table 4 ). Phones and computers have different strengths that influence reaccess patterns. Computers have large screen real estate, fast processing, and a high-speed network connection. Phones are locationally aware, always on, and ubiquitously connected. Phone to computer reaccess was often driven by technical barriers and participants decomposing their tasks among their devices. Computer to phone reaccess occurred due to contextual factors including location, time, and social context. We also found that the most convenient and accessible device was a factor in deciding which device to use for reaccessing content. In the remainder of this section we analyze the different reasons for each reaccess direction. Computer to phone: Need it at another location. Location was a prime contextual factor for motivating reaccess. Location affects the range of tasks the user can engage in, influences the external stimulus experienced by the user (which can act as a catalyst for reaccess), and often places constraints on which devices the user can interact with. Reaccess behaviors influenced by location often began on the computer and shifted to the smartphone as users realized they needed the information while mobile. This

10

E. Bales, T. Sohn, and V. Setlur

Table 4. Top three categories of cross-device reaccessed URLs, broken down by device and reaccess direction Device Top 3 Categories iPhone

News 27.6% Search/Portal 25.19% Social Networks 20.06%

Categories by direction

Phone to comp: News 73.74% Comp to phone: Social Networks 67.34% Android Social Networks 49.89% Phone to comp: Media/News 38.7% Social Networks 84.23% Information Articles 11.41% Comp to phone: Media/News 72.18% N900 Social Networks 56.02% Phone to comp: Mail 13.86% Social Networks 80.36% News 19.28% Comp to phone: Social Networks 71.93%

Phone to Comp µ: 31 min: 3 max: 193

Comp to Phone µ: 32 min: 1 max: 74

Total Reaccess µ: 72 min: 4 max: 267

µ: 20.5 µ: 10 µ: 28 min: 5 min: 3 min: 13 max: 166 max: 212 max: 378 µ: 11.5 µ: 17.5 µ: 29 min: 4 min: 7 min: 11 max: 53 max: 44 max: 97

frequently happened with maps and directions, where turn-by-turn directions are more useful due to the mobile nature of the device. Participant a1 shared this story of reaccess inspired by location. “it [restaurant] had good reviews and a lot of people were talking about it, so we actually went back friday the next week. and I looked it up (on phone) to see exactly what street it was.” Location-based reaccess also occurred when users recognized that information would be more useful at another place besides the point of original access. “Today, my girlfriend was interested in getting a new phone from sprint. I had heard about them having a few android phones, so I went online to read up on HTCs. I read a lot of information on my laptop before we left. While at the sprint store, she was curious also about HTC, but wanted different information. I went back to the same wiki and let her read, since she didn’t want me basically reading 2/3 of the wiki out loud as she perused cell phones.” (Participant i6) Participant i6 knew he would need the information he looked up on his laptop, but his ability to reaccess the content a particular location is what really made it valuable. Computer to Phone: Need it at a later time. Time was another contextual factor that motivated how users reaccessed content on their phones and computers. Participants would typically carry their phone while mobile and could rely on it being available at other times. In these types of reaccesses, users either did not have all the information they needed at the time when they started the task, the task was too long to complete at the initial access, or external events controlled the time at which they could finish the task. “In the morning, I felt like going to Chili’s for lunch so I went to the Chili’s website to find locations near me. I then repeated this on my phone when it was time for lunch so that I would have the address/map with me.” (Participant i4) The participant actually had to wait until the right time, here lunchtime, before he could act on the content he accessed. He wasn’t interested in knowing how to get to

Planning, Apps, and the High-End Smartphone

11

Chilli’s until lunchtime arrived, and waited until then to conduct his reaccess on his phone. Time constraints were another common reason to postpone a task and reaccess it later. Participant n3 shared an example of receiving a long email from a friend that contained a riddle. “A friend gave me a puzzle, like a really long thing, it was going to require a really long answer and it was going to require rereading and thinking about [...] I had glanced at [it] earlier on my computer when I was at work and couldn’t read it.” (Participant n3) When she first read the email she didn’t have time to think about what the answer might be. Later when she was at the airport waiting for a plane, and consequently had a lot of time, she revisited and answered the email on her phone. Computer to Phone: Show my Friends. Social factors was a third type of context that influenced mobile reaccess. Mobile reaccess influenced by social situations was reported 7 out of 15 participants. We define socially motivated reaccesses as any reaccess which prompted social interactions with a friend or colleague. In each of these instances, participants accessed a link, video, or picture they had seen on a device at an earlier time to share with another person. Spontaneous reaccess was common in this category, with many of the reaccesses inspired by conversations with friends. Inspired by his social context, Participant n4 related this situation where his reaccess of a recently watched video. “We were at a bachelor party and started playing foosball, so it kind of came up in conversation, and I was like, oh you gotta see this crazy foosball video! and I pulled it up. I googled ‘Nokia foosball I had remembered that they had spelled it funny, and so I was able to recreate that funny spelling on the google search and it came right up. (Participant n4) This reaccess was impossible for him to predict and he had to rely on his memory for the video’s name and search for it on the spot. Participants noted it was especially important that the content be found quickly, otherwise the conversation flow could be negatively impacted. Computer to Phone: Mobility Barrier. Although laptops are portable and travel frequently with their owners they are ergonomically difficult to use in settings where the user must stand or move frequently. They also have long boot up times, and are often difficult to access quickly. Participants would reach for their phone for convenience and speed depending on their current situation. Nylander et al. observed similar behavior in understanding the motivation for users to perform tasks on their mobile phone [21]. Participants experienced these mobility barriers, which influenced their choice of device when both were available. “I want to access something really quickly, don’t want to wait for computer to boot up [or there is ] no surface to put it on OR not a safe location to reveal I have a computer that someone might want to steal OR I’m actually walking/moving somewhere [or] I’m in a situation where using a computer would be ergonomically difficult (eg. remembering something I needed to do online, but already in bed) (this sounds like a weird use case, but it happens surprisingly often...” (Participant n3) Phone to Computer: Technical Barrier. When reaccessing content on the computer that was originally seen on the phone, technical barriers were the main influencing

12

E. Bales, T. Sohn, and V. Setlur

factors in a reaccess event. Although mobile phone technologies are fast approaching the capabilities of personal computers there are still some things that are impossible to do on today’s mobile smart phone. For example the iPhone is incapable of rendering Flash websites, thus users were forced to view it on their desktop or laptop computer. When users came to a website that looked “wrong” on their phone they often would revisit it later on another device to check and see it their phone was the problem. “I received a link to play a game and knew it would start immediately when I clicked it so I read that I got the msg but waited to get home to click the link.” (Participant a2) Although a2 was able to receive an invitation to play an online game through his email, his phone was not capable of running the game due to technical barriers. Motivated by a desire to beat his friend’s score, a2 delayed playing the game until he knew he was on a capable device. Phone to Computer: Decomposing Tasks. Participants would decompose tasks doing as much as they could on their mobile device and then following up later on their computer. Decomposition can occur because of barriers or mobile limitations, but can also happen when resources are more readily available at another location. Participant i2 shared that at the grocery store she accessed a recipe on her phone so that she could buy the correct ingredients. Later at home she accessed the recipe again from her laptop to assemble the ingredients into a meal. Both of the locations in this example have a specific function, the grocery store for selling food, and the kitchen for preparing it. Accessing the same content at both locations the user was able to complete her full task. Tasks can also be decomposed because they are ongoing over time. Participant i7 had an ongoing task of looking for a new apartment. When she saw an apartment complex that looked reasonable while she was commuting (as a passenger on public transit) she would conduct a brief search to find the price range and amenities to see if the place peaked her interest. Later when she had more time she would use her computer to look up more in depth information on the apartments such as reviews and neighborhood information. In this example i7’s physical location inspired a spontaneous access of data, however her location also imposed time and device constraints which limited her gathering of information. The cost benefits analysis of looking up basic information about the apartments on her phone was worth it, but doing more in-depth research on her phone was not. Once she determined, using her phone, to consider an apartment complex, she would wait until she was in the locational context of ‘home’ to peruse more details about the apartments at her leisure. 4.3 Unplanned Reaccess Behavior Planning ahead can be one of the easiest ways to expedite reaccessing content later. Easy access to directions for an event, phone numbers in an email, printing out a map, or bookmarking a page for later are all methods and practices our participants used to access their content. Despite these methods for planning ahead, participants communicated a general preference not to plan and would rather rely on internet connectivity to reaccess information they needed. Planning ahead was mainly reserved for important items, such as the map to an interview. Participant i5 said that she “wouldn’t preplan unless it’s a big date or a longer trip.”. The typical day to day activities did not involve much planning ahead.

Planning, Apps, and the High-End Smartphone

13

Unforeseen reaccess. It is often hard for users to recognize what information they will need to access later. Content was only accessed a second time if something changed in the participant’s original plan. The mobile convenience of the phone serves users well in these scenarios because they can often tweak their previous search queries to get the results they need. “what you actually need, like, when when you’re right in the situation, is not just information from earlier. It’s information of, like, highly contextual information if something changes. So if you try to go somewhere according to a plan or map that you had ahead of time but you get lost... and so you’re pretty close to where you were supposed to be, and so you need to change it a little bit.” (Participant n3) “I had to go to a wedding and so I just said great! I had to type it the thing[phone] from the paper invite, but then it was nice, I really did like it on my phone vs. paper because when I made wrong turns I could just restart it [...] I’d get to a light and just hit recalculate.” (Participant i3) In another example of unforeseen reaccess, Participant i6 was trying to walk his dog at a new location. “I used a website to locate a walking trail in San Mateo country. After I chose the destination, we headed with the dogs only to discover that the place was under construction. I quickly revisited the website with my iPhone, and decided on an alternate place to roam.” (Participant i6) The participant was not planning on using his phone once he arrived at the planned location but unforeseen circumstances forced him to change his plans. Since his web history was not shared between his devices, he had to redo a search query in order to find the website. Mobile connectivity was a crutch used by our participants to support their unforeseen reaccesses. Even if the participants had pre-planned their activities, the highly contextual nature of their circumstances and changes in plans made it difficult to anticipate the information needed later. Although it is difficult to predict what a user will need ahead of time, since user’s are reaccessing web content seen before, there may be opportunity to explore shortcuts to this content. Plan a little, find it later. Connectivity was an expectation for most of our users given their capable mobile devices. These expectations offer users the freedom to access content they might need on demand without having to completely plan ahead of time. When users would plan ahead, some would prime themselves with a small bit of information and rely on mobile connectivity to access more information while mobile. A common method for doing this was to do a search on the computer, such as visiting a website for initial ideas, but allowing final decisions to be made in a more fluid fashion as the day progressed. I went online to yahoo movies to look for a film that I’d like to see. I chose one, but didn’t select a specific time, since I was meeting someone for dinner first. When we went to the movie theater, I looked up times on my iPhone at the same website. The second search was for a different theater, so I was glad that I hadn’t settled on a time - life’s great when your schedule is flexible ;)”. (Participant i6) Participant i5 shared her all too familiar story of how limited preplanning and onthe-fly mobile research came together for her on a recent weekend. Although she had performed minimal pre-planning to get an idea of restaurants and clubs, she and her friends left the final decisions to the last minute, often changing their minds at the last second.

14

E. Bales, T. Sohn, and V. Setlur

“We had done some emailing, like mostly he[boyfriend] had emailed with them, so then it was like let’s meet up for dinner, and then they wanted to eat dinner while we wanted to be at the beach, so then it was like let’s have drinks later. We were going to meet them at a dance club and they were like ‘oh we’re not really in the mood for that... maybe something more low key,’ and then I was like ‘ok let me look up some other bars on my phone like through yelp’, and then they met us at the place.” (Participant i5) In another example, Participant a4 shopped initially online of shoes, but visited the brick and mortar store to browse and have the in-store experience. “She[wife] had to have these shoes and you could get them online but she wanted them today so I went out and got them for her.... I went to the store and I said ‘hey I need this shoe,’ and I read the description. They still asked me another question, and I said ‘well I don’t know, here it is, that’s what I need.’ ”[showed clerk webpage]. (Participant a4) Although a4 had looked at the shoe online on his computer at home, he accessed the mobile version of the page because he was not familiar with the product details. He relied on the content he was able to access on his phone in the store, to show the store clerk what he wanted, so that he would go home with the correct item. Plan for the long term. Proper planning was reserved for longer term activities, such as finding a job, applying for schools, finding a new apartment, or planning a big event. These longer term reaccess can be particularly difficult for users to handle because the time between the original access and the final reaccess can make it hard for the user to remember the details they need to locate the content. Planning for travel was a common longer term reaccess behavior. Purchasing flights and accommodation usually required many visits to the same websites to check prices before a final purchase. Once tickets were purchased the confirmation emails would be reference multiple times by the user as they made their final decisions about other elements of their trip. Finally, when on the trip, webpages and previously received confirmation emails would commonly be reference to help users navigate, check-in, and remember their schedule. “It’s usually in my email (flight confirmation numbers, hotel reservations, etc.), originally viewed via computer, and I then need to access it again while in transit (on my way to the airport to figure out which terminal to go to, at the counter of the hotel, etc.) [...] what I used to do was print or write this down and carry a piece of paper with all the information in one place. With the phone and a data plan, it was possible to look it up again in transit instead. ” (Participant n3) She did note problems with this method saying “This required logging in to my email and searching for the information, which may be spread out over several emails. I found it a bit of a frustrating experience because the internet access was always quite slow, and I needed to load many pages to get to the piece of information I needed”. In light of the troubles she experienced while having to locate travel related documents on her phone, often months after her original access she shared her vision for a more accommodating mobile solution. “what I really would prefer for that situation is to [...] have them all sent to my phone so that they ended up on one “page” accessible offline. Basically analogous to my printed consolidated piece of paper, only it’s easier to find because it’s on my phone, and the information can be collected as soon as I receive it, rather than right before the trip.”

Planning, Apps, and the High-End Smartphone

15

4.4 The Role of Applications Applications are at the center of how user’s interact with and user their smartphones. These applications are typically native portals into content that could be accessed through a web browser. However, we found many of the heavily trafficked webpages from the desktop absent from the mobile phone logs when the users also had a related native application installed on their device (e.g., Bank of America application). Native applications provide numerous benefits over web pages including better performance, use of sensors and actuators, and easy access from the phone interface. In order to capture data in third party applications, we asked participants to take screenshots whenever they found themselves reaccessing content on their mobile devices. Participants sent 128 screenshots over the course of the two week study, accompanied by a story of the moment of reaccess on the phone. 30 (23.4%) of the 128 screenshots were from applications. We expected more screenshots to be from applications given the plethora of applications that participants used. One possible reason is that frequently used applications are more conducive towards realtime content and not previously seen, static content. For example the BBC news application application for iPhone is designed towards consumption of new data, with the first screen the user is directed to presenting the most recent news stories. It may also be true that when users do engage in reaccess in these applications it is often as a subtask rather than a primary task making it harder for the user to recognize. For example, if a user goes to the Facebook application to see their friend updates and while browsing around decides to comment on a picture that she saw earlier, she may not consciously recognize this sub task as reaccess. “I would go on Facebook and say I feel like I saw this stuff three times ... I go on physical Facebook [on the computer] a lot less than I check the phone app [...] it’s probably 70-30 [iphone-computer]. (Participant i7)” She was aware that she was revisiting content she had seen before, however she never took a screenshot on her phone of any of these encounters. It is possible that although she was reaccessing information, the fact that the content was being “pushed” to her, instead of her actively retrieving it, caused her to not recognize it as a reaccess. Self reporting is one of the difficulties with gathering data in situ from mobile users with a diary study method. Participants indicated a general preference for interacting with native applications rather than mobile web pages. “if theres an app of something I definitely will do that, like, Ill look up products on the amazon app rather than going to amazon through safari. (Participant i5) There was also indication that the advantage native applications had over mobile web pages was slight with users others mentioning that, “if I have Safari open, I’m not going to close it to go to an app, I’ll get the mobile version anyway.” (Participant i7). Mobile web browsers are improving with the adoption of HTML5 that gives web applications access to local storage and on-device sensors. Application-centric smartphones also allow users to save bookmarks in their application screen, letting them live side-by-side with native applications. As the debate over native versus web applications continues, our results around reaccess suggests that users want to enable data to interact among their devices regardless of how they access it.

16

E. Bales, T. Sohn, and V. Setlur

5 Discussion Our investigation into reaccess habits among mobile users revealed the cumbersome workarounds used to find and reaccess information. We found that users would often use features that were made for different purposes as methods to find information later. Many tools, such as Context Clipboard, Evernote, and Dropbox, have attempted to address this problem by enabling easy capture and reaccess, such as saving a link to find later [15] [4] [3]. Although these tools are seamless and easy to use, they still require planning on the part of the user. Through our interviews and discussions with participants, they communicated a general attitude of only planning ahead for big trips and not for the more common reaccess tasks that occur in their everyday life. Sometimes our participants did not know what information they needed later, thus were not able to plan ahead effectively, and other times they expressed dislike of the rigidity imposed by preplanning. Based on these observations, we offer several opportunities to support content reaccess. First, several contextual factors were influential in computer to phone reaccess. Participants would often reaccess content previously seen on their desktop based on future location and time. This is an opportunity to identify content that a user may need later and use location and time context to present it at a relevant moment. Social context is also an opportunity to present previously seen information that can promote dialogue and help keep conversation flow moving. Second, the general attitude among users not to plan ahead presents a large design space to create tools to assist these unplanned reaccesses. Existing tools, such as Firefox Sync, have started the process by using cloud computing to enable the sharing of bookmarks and web history across multiple devices [5] . The next opportunity is exploring how to enable just-in-time access to this data without the burden of searching for it in a mound of data. Participant n4 said that when trying to search for information again while mobile he “[doesn’t] try very hard, if I don’t find it in the first or second search then I just give up.” Finally, breaking content free from mobile application silos can help assist with content reaccess. As applications have become the center of the mobile universe, we noticed signs that people prefer native applications. The content within the application is important, and having better reaccess tools to synchronize this content is essential. For example, after a user looks up directions on the computer, that content should automatically sync to their mobile phone. Many applications are locked in content silos that make it difficult to interact with other applications or devices. As applications continue to move forward, whether as native phone applications or web-based applications, synchronized content is the key to helping users effectively access their data and help support faster unplanned reaccess.

6 Conclusion We presented a two-week study of high-end smartphone users exploring cross-device reaccess patterns. Our analysis of web and mobile application content through logging and screenshots revealed that reaccess occurs with comparable frequency in both directions between the phone and computer. Participants also communicated a general

Planning, Apps, and the High-End Smartphone

17

attitude not to plan ahead for their reaccess needs, preferring to rely on the connectivity of their device. Based on these results, we suggested several areas of opportunity to support the unplanned activities of users. As more devices are introduced into the personal ecosystem, we believe there will be even greater opportunities to support quick, easy reaccess among these devices.

References 1. BBC. The top 100 websites of the Internet, http://news.bbc.co.uk/2/hi/technology/8562801.stm 2. Chrome-to-phone, http://code.google.com/p/chrometophone/ 3. Dropbox, http://dropbox.com 4. Evernote, http://evernote.com 5. Firefox Home, http://www.mozilla.com/en-US/mobile/home/ 6. Adar, E., Teevan, J., Dumais, S.T.: Large scale analysis of web revisitation patterns. In: CHI 2008: Proceeding of the Twenty-Sixth Annual SIGCHI Conference on Human Factors in Computing Systems, pp. 1197–1206. ACM, New York (2008) 7. Beitzel, S.M., Jensen, E.C., Chowdhury, A., Grossman, D., Frieder, O.: Hourly analysis of a very large topically categorized web query log. In: SIGIR 2004: Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 321–328. ACM, New York (2004) 8. Brandt, J., Weiss, N., Klemmer, S.R.: txt 4 l8r: lowering the burden for diary studies under mobile conditions. In: CHI 2007: Extended Abstracts on Human Factors in Computing Systems, pp. 2303–2308. ACM, New York (2007) 9. Carter, S., Mankoff, J.: When participants do the capturing: the role of media in diary studies. In: CHI 2005: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 899–908. ACM, New York (2005) 10. Church, K., Smyth, B.: Understanding mobile information needs. In: MobileHCI 2008: Proceedings of the 10th International Conference on Human Computer Interaction with Mobile Devices and Services, pp. 493–494. ACM, New York (2008) 11. Cockburn, A., Mckenzie, B.: What do web users do? an empirical analysis of web use. International Journal of Human-Computer Studies 54(6), 903–922 (2001) 12. Dearman, D., Kellar, M., Truong, K.N.: An examination of daily information needs and sharing opportunities. In: CSCW 2008: Proceedings of the 2008 ACM Conference on Computer Supported Cooperative Work, pp. 679–688. ACM, New York (2008) 13. Dearman, D., Pierce, J.S.: It’s on my other computer!: computing with multiple devices. In: CHI 2008: Proceeding of the Twenty-Sixth Annual SIGCHI Conference on Human Factors in Computing Systems, pp. 767–776. ACM, New York (2008) 14. Gonz´alez, I.E., Hong, J.: Gurungo: coupling personal computers and mobile devices through mobile data types. In: HotMobile 2010: Proceedings of the Eleventh Workshop on Mobile Computing Systems & Applications, pp. 66–71. ACM, New York (2010) 15. Harding, M., Storz, O., Davies, N., Friday, A.: Planning ahead: techniques for simplifying mobile service use. In: HotMobile 2009: Proceedings of the 10th Workshop on Mobile Computing Systems and Applications, pp. 1–6. ACM, New York (2009) 16. Jansen, B.J., Spink, A., Bateman, J., Saracevic, T.: Real life information retrieval: a study of user queries on the web. SIGIR Forum 32(1), 5–17 (1998) 17. Kamvar, M., Baluja, S.: A large scale study of wireless search behavior: Google mobile search. In: CHI 2006: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 701–709. ACM, New York (2006)

18

E. Bales, T. Sohn, and V. Setlur

18. Kamvar, M., Kellar, M., Patel, R., Xu, Y.: Computers and iphones and mobile phones, oh my!: a logs-based comparison of search users on different devices. In: WWW 2009: Proceedings of the 18th International Conference on World Wide Web, pp. 801–810. ACM, New York (2009) 19. Kane, S.K., Karlson, A.K., Meyers, B.R., Johns, P., Jacobs, A., Smith, G.: Exploring crossdevice web use on pCs and mobile devices. In: Gross, T., Gulliksen, J., Kotz´e, P., Oestreicher, L., Palanque, P., Prates, R.O., Winckler, M. (eds.) INTERACT 2009. LNCS, vol. 5726, pp. 722–735. Springer, Heidelberg (2009) 20. Karlson, A.K., Iqbal, S.T., Meyers, B., Ramos, G., Lee, K., Tang, J.C.: Mobile taskflow in context: a screenshot study of smartphone usage. In: CHI 2010: Proceedings of the 28th International Conference on Human Factors in Computing Systems, pp. 2009–2018. ACM, New York (2010) 21. Nylander, S., Lundquist, T., Br¨annstr¨om, A., Karlson, B.: “It’s just easier with the phone” – A diary study of internet access from cell phones. In: Tokuda, H., Beigl, M., Friday, A., Brush, A.J.B., Tobe, Y. (eds.) Pervasive 2009. LNCS, vol. 5538, pp. 354–371. Springer, Heidelberg (2009) 22. Obendorf, H., Weinreich, H., Herder, E., Mayer, M.: Web page revisitation revisited: implications of a long-term click-stream study of browser usage. In: CHI 2007: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 597–606. ACM, New York (2007) 23. Pierce, J.S., Nichols, J.: An infrastructure for extending applications’ user experiences across multiple personal devices. In: UIST 2008: Proceedings of the 21st Annual ACM Symposium on User Interface Software and Technology, pp. 101–110. ACM, New York (2008) 24. Sohn, T., Li, K.A., Griswold, W.G., Hollan, J.D.: A diary study of mobile information needs. In: CHI 2008: Proceeding of the Twenty-Sixth Annual SIGCHI Conference on Human Factors in Computing Systems, pp. 433–442. ACM, New York (2008) 25. Spink, A., Jansen, B.J., Wolfram, D., Saracevic, T.: From e-sex to e-commerce: Web search changes. Computer 35(3), 107–109 (2002) 26. Tauscher, L., Greenberg, S.: How people revisit web pages: empirical findings and implications for the design of history systems. International Journal of Human-Computer Studies 47(1), 97–137 (1997)

Understanding Human-Smartphone Concerns: A Study of Battery Life Denzil Ferreira1,2, Anind K. Dey2, and Vassilis Kostakos1 1

Madeira Interactive Technologies Institute, University of Madeira, Portugal Human-Computer Interaction Institute, Carnegie Mellon University, USA [email protected], [email protected], [email protected] 2

Abstract. This paper presents a large, 4-week study of more than 4000 people to assess their smartphone charging habits to identify timeslots suitable for opportunistic data uploading and power intensive operations on such devices, as well as opportunities to provide interventions to support better charging behavior. The paper provides an overview of our study and how it was conducted using an online appstore as a software deployment mechanism, and what battery information was collected. We then describe how people charge their smartphones, the implications on battery life and energy usage, and discuss how to improve users’ experience with battery life. Keywords: Large-scale study, battery life, autonomous logging, smartphones, android.

1 Introduction Sustainability and energy reduction have emerged as important topics in the social, political and technical agendas in recent decades. The ubiquitous computing research community, with its focus on both design and development of technological systems has had to systematically face a strain between sustainability and usability. On the one hand, users express an interest in adopting more sustainable products and behavior, but on the other hand, they do not wish to do so at the expense of their comfort. Hence it is important that solutions tackling energy reduction take into accounts users’ behavior and preferences before making an intervention. One area strongly related to ubiquitous computing research where substantial energy savings can be achieved by introducing more usable systems is smartphones. Cell phones are increasingly popular and diverse, with worldwide sales approaching 1.6 billion units, just last year [8]. Thanks to the rapid development of wireless technologies, smartphones allow users to be reachable anywhere [3]. As "convergent" devices, smartphones empower users with Internet access, music, audio and video playback and recording, navigation and communication capabilities. However, the growing functionality of smartphones requires more power to support operation throughout the day. Processing power, feature-sets and sensors are bottlenecked by battery life limitations, with the typical battery capacity of smartphones today being barely above 1500 mAh [5]. This is an important limitation because smartphones are increasingly regarded as a gateway to one’s daily life, K. Lyons, J. Hightower, and E.M. Huang (Eds.): Pervasive 2011, LNCS 6696, pp. 19–33, 2011. © Springer-Verlag Berlin Heidelberg 2011

20

D. Ferreira, A.K. Dey, and V. Kostakos

providing networking access to email, social networking, and messaging, making the management of battery life an important task. Despite the important limitations that battery life imposes on users, previous research has shown that existing battery interfaces present limited information, and, as a consequence, users develop inaccurate mental models about how the battery discharges and how the remaining battery percentage shown in the interface correlates to application usage [20]. In addition, users do not completely understand how they should charge their batteries to support their planned use of the phone. As a result, every year $22 million are spent in electric utility costs due to keeping cell phones plugged into outlets for more time than required, to maintain a full charge [8]. On average, cell phone power supplies use 0.2 watts when the charger is left plugged into an electrical socket and the phone is no longer attached, with less sophisticated power supply designs reaching 1 watt [8]. We argue that there exists potential in reducing the energy consumption of smartphones by better understanding users’ interactions with smartphones and providing better feedback. While previous studies have focused on the shortcomings of user interfaces in relation to battery life, there is a need to assess the real-world behavior of a large number of users in terms of when, how and how long they charge their batteries. By analyzing users’ battery charging behavior, we can assess the extent to which energy is being wasted, explore how often users demonstrate less than optimal charging behavior, how often they interrupt the charging cycle and when this is more likely to happen. We hypothesize that by conducting such a study we can identify design opportunities for reducing energy consumption, increasing battery life, and also predicting when intensive computational operations and long data transfers should be scheduled. This paper starts by giving an overview of related work and current state of the art on smartphone battery management, followed by a description of how was the study deployed and conducted using the Android Marketplace, and a discussion of implementation concerns. We then present the results and a discussion of users’ charging habits, how to tackle the issues of wasted energy and opportunistic processing on smartphones. We conclude with a discussion of how the results can affect the design of a future smartphone for an energy conscious world.

2 Related Work Most smartphones offer the possibility to add new applications, through distribution channels such as the Google Marketplace for the Android platform or App Store for the iPhone platform. These applications often take advantage of the sensors available, typically GPS and Internet connectivity to develop context-aware applications [10,5], accelerometer for motion tracking [18], Bluetooth for distance measurements from the device [15] and anomaly detection [3,19]. While devices are becoming increasingly mobile, many software developers have limited experience with energy-constrained portable embedded systems such as smartphones, which leads to unnecessarily power-hungry applications that rely on the operating system for power management. In addition, users struggle to determine which applications are energy-efficient, and typically users blame the operating

Understanding Human-Smartphone Concerns: A Study of Battery Life

21

system or hardware platform instead of unfortunate and unintentional software design decisions [21]. Rahmati et al. [16] coined the term Human-Battery Interaction (HBI) to describe mobile phone users’ interaction with their cell phones to manage the battery available. According to a survey they conducted, 80% of users take measures to increase their battery lifetime, and it can be expected that maximizing battery life will continue to be a key concern for users due to the major usability issues involved in this task. One approach to automatically deal with this issue is to rely on sensor data. For example, recent devices act proactively to reduce their power consumption, either by turning off the screen after a specific amount of time with no new interaction, switching to a lower processing speed (CPU scaling), or disabling wireless interfaces such as Bluetooth and WiFi when battery levels are low. These devices effectively take into account sensed data regarding battery levels, idle time, etc. Oliver et al. [7] highlighted the importance of using real user data collected from the world and how it can influence application development, by introducing the Energy Emulation Toolkit (EET) that allows developers to evaluate the energy consumption requirements of their applications against the collected data. As a result, by classifying smartphone users based on their charging characteristics, the energy level can be predicted with 72% accuracy a full day in advance. A study on the environmental impact of cell phone charging related to national energy consumption and power plant greenhouse gas emissions reveals that the energy consumed by cell phone charging has been reduced by 50% in the past years due to two technology shifts: increased usage of power management and low-power modes of battery chargers; and use of more efficient switch-mode power supplies [8]. Despite these efficiency gains, however, the US could save 300 million kWh in electricity per year, which amounts to $22 million in electric utility costs, or 216.000 tons of CO2 emissions from power plants. The study presented here complements Oliver’s study on user charging characteristics [7] and Rahmati et al.’s [16] study on how users consume battery in their devices. It aims to identify when, how, for how long and how frequently users recharge their devices’ batteries, in order to assess the extent to which energy savings can be achieved. At the same time, the collected information can be used to identify design opportunities in order to achieve such energy savings.

3 Study We conducted a study of battery charging behaviors with 4035 participants over a period of four weeks, during which anonymous battery information was collected from Android devices running Android 1.6 or higher. In total, more than 7 million data points of battery information were collected. The Open Handset Alliance Project “Android” is a complete, free, and open mobile platform, and its API provides open access to the device hardware, abstracted from each device’s manufacturer or brand [2, 13], therefore increasing the number of deployable devices. Although the study was conducted solely with Android devices, most of the results should be similar to other smartphone platforms with respect to battery information and user behavior over time [11].

22

D. Ferreira, A.K. Dey, and V. Kostakos

There was no monetary compensation given to the participating users. The developed application, OverCharged, which was developed to help users be more aware of their battery usage, was made available for free on the Google MarketPlace. The main function of the OverCharged application we developed is to inform participants of their smart phone’s current battery level, for how long the phone was running on battery and other miscellaneous information, such as temperature and voltage. As such, the users who downloaded the application and opted in to sharing their data are already concerned with the battery life on their mobile devices. Therefore, they may in fact be atypical users, and our sample may not be representative of what all smartphone owners would do. Nonetheless, our study does serve as the first large collection of battery usage. During the study, users had the option to opt-in to sharing their battery data anonymously in order to contribute to a better understanding of battery usage patterns. The application captured charging activity, battery level, device type, temperature, voltage and uptime: • Charging activity captured when the user charged his device, either through USB or an AC outlet. • Battery level reflects the remaining battery and how long it took to discharge or charge. • Device type is the manufacturer, device board, model, Android version and build and the carrier. • Temperature of the battery, both Celsius and Fahrenheit. • Voltage available in millivolts (mV). • Uptime is the amount of time the device was on until being turned off or rebooted. The combination of charging activity and battery level allows for the identification of events such as “unplugged not full”, “charged just unplugged”, “finished charging”, “charging” and “running on battery”, defined as follows: • Unplugged not full: when the user stopped charging, even though the battery was not fully charged. • Charged just unplugged: when the user unplugged the charger and the battery is fully charged. • Finished charging: the moment when the battery is fully charged. • Charging: when the battery starts charging. • Running on battery: when the battery is the only power source. 3.1 Implementation Polling a device’s state can reduce battery life [10, 12]. The Android API is eventdriven, hence gathering the data had a negligible impact on regular battery life. By programming a BroadcastReceiver attached to an Android Service running in the background, whenever the Android OS broadcasts ACTION_BATTERY_ CHANGED, the following battery information was recorded: battery level, battery scale (maximum level value), battery percentage, battery technology (i.e. Li-ion), health rating of the battery, whether the phone was plugged to AC/USB, whether the

Understanding Human-Smartphone Concerns: A Study of Battery Life

23

phone is charging, temperature, voltage, uptime and usage uptime, battery status (charging, discharging, full and not charging) and phone events related to battery (fully charged and user just unplugged, charging, finished charging, running on battery, unplugged when not fully charged). As highlighted by Oliver [10], a large-scale user study distributed across the globe requires the use of UTC timestamps. We captured the UNIX timestamp on the participant’s device time zone, which results in consistent times across different time zones (i.e., 8pm is the same for different users at different time zones). These timestamps were used across all data collection and analysis operations. The application was programmed to start automatically when the device was turned on or rebooted. A small icon in the notification bar at the top of the screen kept users informed that data was being collected and allowed users to view further information [Figure 1].

Fig. 1. Notification bar information

3.2 Device Distribution Of the approximately 17000 people that were using the application at the time the study was conducted, 4035 opted in to participate on our study. After the installation of the application from the MarketPlace, if the user opted in to participate in our study, the application captured device details including device board, service carrier, manufacturer, model, Android version and Android build. Recent Gartner worldwide mobile device sales reports [7, 19] do not place HTC as the leading sales manufacturer. Originally producing primarily Windows Mobile phones, HTC has changed their focus to Android devices, by manufacturing the Google Nexus One and EVO 4G more recently. Of the phones used by our

24

D. Ferreira, A.K. Dey, and V. Kostakos

participants, HTC devices and Sony Ericsson devices were the most popular (44.6% and 29.8% respectively). In third place were Motorola devices with 14.8%, followed by Samsung with 7.5% [Table 1]. Furthermore, Google’s statistics claim that Android 2.1 is the most popular version with 41.7%, while in our study we saw that 33% of phones used this version [Table 2]. One surprise in the collected data is that Android 1.6 (Donut) is the leader with 36% of the participating devices using it. Table 1. Most popular platforms recorded during the study

Platform

Distribution

HTC

44.6%

Sony Ericsson

29.8%

Motorola

14.8%

Samsung

7.5%

Table 2. Google’s official Android distribution, as of September 1, 2010 [1]

Platform

API Level

Popularity (Source: Google)

Popularity (Source: Study)

Android 1.5

3

12.0%

-

Android 1.6

4

17.5%

36%

Android 2.1

7

41.7%

33%

Android 2.2

8

28.7%

31%

3.3 How Do Users Manage Battery Life? Users mostly avoided lower battery levels, with the daily average of the lowest battery percentage values being 30%. This is likely due to the fact that the Android devices’ battery icon turns yellow at 30%, and prompts the user with a textual notification to charge the smartphone by the time it reaches 15%. The visualization in Figure 2 shows the average battery available at different hours of the day, across all the users, and how frequently the percentage was observed, when the battery was not being charged. Each bubble represents a different day of the study, for a given hour (with a bubble created only when there were at least 1000 datapoints for the selected day-hour combination). Hence, the visualization contains three dimensions (Percentage, Time and Frequency), with frequency (low to high) highlighted both by size (small to big) and color (light yellow to dark red). The most frequent battery averages are above the 30% battery level.

25

Battery percentage

Understanding Human-Smartphone Concerns: A Study of Battery Life

Hour of the day (0-23)

Fig. 2. Average battery levels during the day (when not charging)

Battery percentage

On average the lowest average battery level was 65% at midnight, while the highest was 74% at 5AM. We expected that battery levels would be lowest at the end of the day, and the results confirmed it. The average battery percentage is 67% across all users throughout the day [Figure 3].

Hour of the day (0-23)

Fig. 3. Average battery levels throughout the day for the whole population

Despite the small variation of hourly battery levels across the whole population, individual users exhibited varying charging patterns. Some prefer to charge for short amounts of time throughout the day, while others allow the battery to discharge and charge it for longer periods of time until full [Figure 4].

D. Ferreira, A.K. Dey, and V. Kostakos

Battery percentage

26

Hour of the day (0-23)

Fig. 4. Battery level during a single day for three different users

Battery percentage

The data reveals two major charging schedules: one between 6PM and 8PM, with the majority of users initiating charging when the battery levels are at 40%, and another charging schedule between 1AM and 2AM, with a majority initiating charging when battery is at 30%. Another frequent charging event happens at 8AM, with battery levels at 80% on average [Figure 5].

Hour of the day (0-23)

Fig. 5. Average battery levels during the day at the moment when charging begins

The majority of the charging instances occur for a very small period of time (up to thirty minutes) or between one to two hours, which is the average required time to recharge completely a battery (left side of the graph). [Figure 6].

27

Frequency

Understanding Human-Smartphone Concerns: A Study of Battery Life

Amount of charging time

Fig. 6. Charging duration (amount of time the phone remains plugged in)

Frequency

As expected, a lot of charging instances happen overnight, for 14 hours or more (right side of Figure 6). The average charging time across the whole population is approximately 3 hours and 54 minutes, but there is certainly a bimodal distribution, with the majority of charging instances lasting less than 3 hours. By charging time, we mean the time since the user plugged his device to charge until unplugged from the outlet. Most charging instances start between 5PM and 9PM, while the least popular time to begin charging is from 3AM to 8AM [Figure 7], although the data in Figure 6 shows that it is likely that phones are being charged during this time.

Hour of the day (0-23)

Fig. 7. Charging schedule (times when users have their phones plugged in)

3.4 How Much Energy Do Users Waste? Overall, in 23% of the charging instances, the phone is unplugged from the charger (USB and AC) within the first 30 minutes after the battery is fully charged, while in the remaining 77%, the phone is plugged in for longer periods thus leading to energy waste. On average, users keep the phones plugged for 4 hours and 39 minutes after charging has been completed [Figure 8].

D. Ferreira, A.K. Dey, and V. Kostakos

Frequency

28

Amount of time

Fig. 8. Time until unplugged after the battery is full

Frequency

Monitoring when the device has finished charging, we calculated how long the user took to unplug the device from the charger (USB and AC). The amount of time is greater as expected during the night, starting most often at 11PM and lasting until 8AM [Figure 9].

Hour of the day (0-23)

Fig. 9. Overcharging schedule

3.5 How Does Charging Happen?

Frequency

As predicted, for longer charging periods AC is the preferred choice for phone charging. For short charges (30 minutes or less), USB charging is much more frequent. On average, users charge their phones 39% of the time using USB, and 61% of the time using AC [Figure 10].

Amount of time

Fig. 10. Amount of time charging with USB (red) vs. AC (blue)

Understanding Human-Smartphone Concerns: A Study of Battery Life

29

In Figure 10, blue represents AC, and red is USB charging. The initial pair on the left represents charging between 0-30 minutes, in which charging is mostly USB for this specific period of time. AC charging has two peaks, one between 1-3h of charging time and 14 hours or more for overnight charging. 3.6 How Often Is the Phone Rebooted/Turned Off? Uptime is the time elapsed before the phone is rebooted or turned off. In our study, all participants’ devices are on for at least up to a full day [Figure 11]. The results show that the likelihood of having a device on for up to two days is 33%, 18% for up to three and 11% for up to four days.

Fig. 11. Uptime in days

4 Discussion The large-scale study described here was conducted in order to assess the extent to which energy is being wasted, explore how often users demonstrate less than optimal charging behavior, how often they interrupt the charging cycle and when this is more likely to happen. We hypothesized that by conducting such a study we could identify design opportunities for reducing energy consumption, increasing battery life, and also predicting when intensive computational operations and long data transfers should be scheduled. Previous studies have shown that users have inadequate knowledge of smartphone power characteristics and are often unaware of power-saving settings on smartphones [16]. Users should be provided with options on how to better manage the remaining battery, and, to some extent, automated power features can also help them use the device as intended [12, 20]. Most smartphones alert the user that they need to be charged when the battery reaches critical levels [16,17], but do not notify the user when it has finished charging. For instance, explicitly notifying the user that the device is running low on battery is something Android does when the battery is at 15%. Battery management requires user intervention in two respects: to keep track of the battery available so that users can decide how to prioritize amongst the tasks the

30

D. Ferreira, A.K. Dey, and V. Kostakos

device can perform; and to physically plug the device to the charger and surrender its mobility [17]. There is an opportunity to optimize which functionality should remain active based on the user’s lifestyle and battery charging habits, improving the humanbattery interface (HBI) with the user. Each user is unique and as such, the optimization system must learn and adapt to the user. The results show important differences between users’ behavior and preferences, but also highlight common patterns that can be useful in understanding aggregate behavior and developing software that taps into those behaviors. The findings of this study show that users • demonstrate systematic but at times erratic charging behavior (mostly due to the fact that charging takes place when the phones are connected to a PC); • mostly choose to interrupt their phones’ charging cycle thus reducing battery life. • aim to keep their battery levels above 30% due to an automatic ambient notification; and • consistently overcharge their phones (especially during the night); 4.1 Users’ Charging Habits The study shows that users charged throughout the day resulting in erratic charging patterns and disrupted charging cycles that can reduce the lifetime of the battery [Figure 4]. A potential design opportunity exists here, whereby erratic charging behavior can be avoided by implementing a timer threshold that will prevent batteries from charging for short periods of times, e.g., for less than 5 minutes. The results [Figure 10] demonstrate that charging using USB could be triggered by command from the user (a feature already seen with some HTC Sense® devices) or if the battery percentage available is below 30%. Interrupted charging cycles [Figure 11] leads to the necessity of battery calibration (drain the battery until depletion and fully charging it). The “memory effect”, is a term loosely applied to a variety of battery ills [9]. From Corey’s research [5], overcharging, over discharge, excessive charge/discharge rates and extreme temperatures of operation will cause the batteries to die prematurely. Users in this study consistently kept the battery from reaching lower levels, with an average lower percentage of 30% of battery power by charging throughout the day (e.g., plugging their devices to the car dock for navigation at 8AM [Figure 5] or charging while transferring files). Software updates and backup routines could take these moments to run power intensive operations only if the user has his phone plugged in for more than 30 minutes, since according to the results, there is a very high probability the user will charge for at least 1-2 hours. 4.2 Avoiding Energy Waste Another problem that our study highlights in relation to charging duration is the amount of time the users keep their phones connected unnecessarily. In the past, charging a battery for a long period of time would damage the battery from overheating and overvoltage [4, 5, 21]. Modern Li-ion and Li-poly batteries come from the manufacturer prepared to interrupt charging as soon as they are fully charged

Understanding Human-Smartphone Concerns: A Study of Battery Life

31

[14], but this still results in unnecessary power consumption. This study shows that this happens frequently, which suggests that manufacturers should make an effort to improve their chargers to cutoff the charging as soon as the battery is full or after some time in cases where the phone is being powered directly from the charger. In addition, there is a design opportunity to give feedback to users the moment they plug in their phone – they usually look for confirmation that the phone is charging. At that moment feedback could be provided to change users’ behavior. For example, we can predict when a “plugged in” event is likely to result in a long power consumption session, specially if it happens around 11PM. At that moment a message could inform the user that “your phone will be fully charged in X minutes”, prompting them to remember to unplug it, to minimize the time when the phone is plugged in when it is already fully charged. The combination of erratic charging and unnecessary charging observed in this study shows that users appear to have two types of charging needs: short bursts of charging to get through the day, and long charging periods during the night. One mechanism to reconcile these two distinct requirements is to allow for batteries to have a “slow-charge” mode, whereby they do not charge as fast as possible, but charge at a rate that will reduce the amount of unnecessary charging. A rule of thumb can be derived from [Figure 6], which suggests that an effective rate for “slowcharge” rate could kick-in after 30 minutes and aim for a full charge in 4 hours (the average overcharging length). A more sophisticated approach could incorporate a learning algorithm on the smartphone or even the battery itself. 4.3 Opportunistic Processing on Smartphones In terms of identifying opportunities for intensive operations on the smartphone, the results suggest that there exists an important 30-minute threshold once charging begins. If a charging session lasts more than 30 minutes, it is very likely that it will last for a substantially longer period. Charging that uses AC is also an indicator that the user will be likely to charge for a longer period of time. Combined, the 30-minute threshold and AC power source provide a good indication as to when applications should perform power intensive operations on smartphones: large data transfers, computationally intensive activities, etc.

5 Conclusion More than ever, industry and academic research have an opportunity to resolve numerous issues and conduct studies using published applications to support users’ needs. Marketing and mobile phone manufacturers study a variety of user needs, focusing on the design of new handsets and/or new services [15]. Using automatic logging, in which software automatically captures user’s actions for later analysis provides researchers with the opportunity to gather data continuously, regardless of location or activity the user might be performing, without being intrusive. Asking users to anonymously collect battery information using a Google Marketplace application was a success: at the time of writing, 7 million battery

32

D. Ferreira, A.K. Dey, and V. Kostakos

information points and 4000 participating devices from all over the world were loaded into our database from which the battery charging patterns were explored. The results provide application developers and manufacturers with information about how the batteries are being charged by a large population. The design considerations highlight how can we improve users’ experience with their battery life and educate them about the limited power their devices have. We look forward to seeing the next generation of smartphones, that learn from the user’s charging routines and changes their operation and charging behavior accordingly.

Acknowledgements We thank all the anonymous participants that contributed for the study using our application. This work was supported in part by the Portuguese Foundation for Science and Technology (FCT) grant CMU-PT/HuMach/0004/2008 (SINAIS).

References 1. Android Developer Dashboard (September 1, 2010), http://developer.android.com/resources/dashboard/ platform-versions.html 2. Android OS (2011), http://www.android.com (last accessed February 24, 2011) 3. Buennemeyer, T.K., Nelson, T.M., Clagett, L.M., Dunning, J.P., Marchany, R.C., Tront, J.G.: Mobile Device Profiling and Intrusion Detection using Smart Batteries. In: Proceedings in the 41th Hawaii International Conference on System Sciences (2008) 4. Byrne, J.A.: The Proper Charging Of Stationary Lead-Acid Batteries (Your Battery Is Only As Good As How You Charge It.). In: Battcon 2010 (2010) 5. Corey, G.P.: Nine Ways To Murder Your Battery (These Are Only Some Of The Ways). In: Battcon 2010 (2010) 6. Cuervo, E., Balasubramanian, A., Cho, D., Wolman, A., Saroiu, S., Chandra, R., Bahl, P.: MAUI: Making Smartphones Last Longer with Code Offload. In: MobiSys 2010, San Francisco, California, June 15-18 (2010) 7. Gartner Research – Gartner Says Worldwide Mobile Device Sales Grew 13.8 Percent in Second Quarter of 2010, But Competition Drove Prices Down (August 12, 2010), http://www.gartner.com/it/page.jsp?id=1421013 8. Gartner Says Worldwide Mobile Device Sales to End Users Reached 1.6 Billion Units in 2010; Smartphone Sales Grew 72 Percent in 2010 (February 9, 2011), http://www.gartner.com/it/page.jsp?id=1543014 9. McDowall, J.: Memory Effect in Stationary Ni-CD Batteries? Forget about it! In: Battcon 2003 (2003) 10. Oliver, E.: The Challenges in Large-Scale Smartphone User Studies. In: International Conference On Mobile Systems, Applications And Services, Prec. 2nd ACM International Workshop on Hot Topics in Planet-scale Measurement, San Francisco, California (2010) 11. Oliver, E.: A Survey of Platforms for Mobile Networks Research. Mobile Computing and Communications Review 12(4) (2008) 12. Oliver, E., Keshav, S.: Data Driven Smartphone Energy Level Prediction. University of Waterloo Technical Report No. CS-2010-06 (April 15, 2010)

Understanding Human-Smartphone Concerns: A Study of Battery Life

33

13. Open Handset Alliance, http://www.openhandsetalliance.com (last accessed February 24, 2011) 14. Ostendorp, P., Foster, S., Calwell, C.: Cellular Phones, Advancements in Energy Efficiency and Opportunities for Energy Savings. NRDC 23 (October 2004) 15. Patel, S.N., Kientz, J.A., Hayes, G.R., Bhat, S., Abowd, G.D.: Farther Than You May Think: An Empirical Investigation of the Proximity of Users to Their Mobile Phones. In: Dourish, P., Friday, A. (eds.) UbiComp 2006. LNCS, vol. 4206, pp. 123–140. Springer, Heidelberg (2006) 16. Rahmati, A., Qian, A., Zhong, L.: Understanding Human-Battery Interaction on Mobile Phones. In: MobileHCI 2007, Singapore, September 9-12 (2007) 17. Ravi, N., Scott, J., Han, L., Iftode, L.: Context-aware Battery Management for Mobile Phones. In: Sixth Annual IEEE International Conference on Pervasive Computing and Communications (2008) 18. Reddy, S., Mun, M., Burke, J., Estrin, D., Hansen, M., Srivastava, M.: Using Mobile Phones to Determine Transportation Modes. ACM Transactions on Sensor Networks 6(2), article 13 (February 2010) 19. Schmidt, A.D., Peters, F., Lamour, F., Scheel, C., Çamtepe, S.A., Albayrak, S.: Monitoring Smartphones for Anomaly Detection. Mobile Network Applications (2009) 20. Truong, K., Kientz, J., Sohn, T., Rosenzweig, A., Fonville, A., Smith, T.: The Design and Evalution of a Task-Centered Battery Interface. In: Ubicomp 2010 (2010) 21. Zhang, L., Tiwana, B., Dick, R.P., Qian, Z., Mao, Z.M., Wang, Z., Yang, L.: Accurate Online Power Estimation And Automatic Battery Behavior Based Power Model Generation for Smartphones. In: CODES+ISSS 2010, Scottsdale, Arizona, USA, October 24-29 (2010) 22. Zheng, P., Ni, L.M.: Spotlight: The Rise of the Smart Phone, vol. 7(3). IEEE Computer Society, Los Alamitos (March 2006) 23. Zhuang, Z., Kim, K., Singh, J.P.: Improving Energy Efficiency of Location Sensing on Smartphones. In: MobiSys 2010, San Francisco, California, June 15-18 (2010)

Monitoring Residential Noise for Prospective Home Owners and Renters Thomas Zimmerman and Christine Robson Mobile Computing Research Group IBM Research - Almaden 650 Harry Road, San Jose CA 95120 {tzim,crobson}@us.ibm.com

Abstract. Residential noise is a leading cause of neighborhood dissatisfaction but is difficult to quantify for it varies in intensity and spectra over time. We have developed a noise model and data representation techniques that prospective homeowners and renters can use to provide quantitative and qualitative answers to the question, “is this a quiet neighborhood?” Residential noise is modeled as an ambient background punctuated by transient events. The quantitative noise model extracts noise features that are sent as SMS text messages. A device that implements the noise model has been build, calibrated and verified. The qualitative impact of sound is subjectively assessed by providing one-minute audio summaries composed of twenty 3-second sound segments that represent the loudest noise events occurring in a 24 hour sampling period. The usefulness and desirability of the noise pollution monitoring service is confirmed with pre- and post-use surveys. Keywords: Location-based services, mobile devices, sensors.

1 Introduction Excessive, unwanted or disturbing sound in the environment is called noise pollution. Noise affects our physical and mental health, increasing blood pressure and stress, damaging hearing and disturbing sleep [1]. Unlike chemical pollution which can be measured with a single value (e.g. parts per million) sound is difficult to quantify because it is multidimensional, varying in intensity and spectra. Similarly, the impact of noise on humans is complex. Some noises are pleasant, like flowing water, while others are annoying like car alarms, screeching breaks and people arguing. We experience noise pollution in all aspects of our lives; at work, at home and on holiday. Noise in the workplace is regulated to protect employees against loss of hearing and other injuries [2]. The parks department is very concerned with maintaining an acceptable soundscape and uses a combination of human observers and instrumentation to measure the impact of noise pollution such as jet flyovers and recreation vehicles (e.g. snowmobiles and water craft) [3]. In this paper, we are concerned with residential noise. In annual surveys conducted by the Department of Housing and Urban Development for the past three decades, noise has been identified as the leading cause of neighborhood dissatisfaction, with K. Lyons, J. Hightower, and E.M. Huang (Eds.): Pervasive 2011, LNCS 6696, pp. 34–49, 2011. © Springer-Verlag Berlin Heidelberg 2011

Monitoring Residential Noise for Prospective Home Owners and Renters

35

traffic and aircraft noise leading the list [4]. We have developed noise monitoring hardware, analysis algorithms, visual and auditory representations to help people evaluate the noise environment of a home, providing a quantitative objective answer to the question, ``is this a quiet neighborhood?'' When a person enters into a contract to purchases a house, a home inspection is performed to inform the buyer of any structural, electrical, plumbing and roofing problems. Too often, however, residents learn of unexpected noise pollution on the first night of occupancy. With this need in mind, we set out to create a device that could be used as part of a home inspection to provide an assessment of residential noise pollution. We began with a survey to understand how noise pollution affects residents in our area and a review of current laws and literature on noise pollution. We designed and built a device to monitor noise pollution in response to these problems. Using noise samples from three representative houses, spanning the spectrum of quiet to noisy neighborhoods, we developed a noise model to characterize residential noise and a means to compress noise events of an entire day (24 hours) into a one minute auditory summary. Our design minimizes data collection, transmission and storage requirements to utilize low-cost and low-power components, while maintain sufficient measurement accuracy. We conducted a user study to measure the effectiveness of visual and auditory presentations of the collected noise data. Our results show that people prefer to compare homes by the audio summaries rather than visual representation of noise data.

2 Background 2.1 Related Work Noise control has its US roots in the establishment of the National Environmental Policy Act of 1969 and the Noise Control Act of 1972. At that time the EPA testified before Congress that 30 million Americans are exposed to noise pollution [5]. The "Green Paper on Future Noise Policy" [6] published in 1996 set the ground work for noise policy in Europe. As more people live in cities, there are more noise sources and greater pressure from residents to control noise. Several organizations maintain web sites with a wealth of information on noise pollution (e.g. www.nonoise.org, www.acousticecology.org, www.noisefree.org) to raise awareness and reduce noise pollution. Government agencies [7] have noise monitoring and education programs, including interactive noise maps [8, 9]. Sound meters have been combined with GPS receivers to populate a database to create sound maps of a location [10]. Airports have permanent noise monitoring equipment to measure noise produced by arriving and departing aircraft [11, 12]. The FAA is required to make noise exposure maps available to the public via the Internet [13]. Using the WebTrack tool [14] we are able to correlate the air traffic noise detected at our noise pollution study locations with departing and arriving aircraft. The site also has real time sound pressure levels from fixed location monitoring stations on the arrival and departure corridors around the airport.

36

T. Zimmerman and C. Robson

Noise pollution has motivated municipalities to take an active role in monitoring and reducing noise to protect their citizens. The European Directive 2002/49/EC [15] to monitor, inform, address and develop a long term strategy to reduce offending noise, has stimulated much noise pollution research and development activity in Europe. The Directive stimulated the MADRAS project [16], creating a database of samples of noise pollution sounds used by researchers studying noise pollution. One project [17] developed an automatic noise recognition system using a hidden Markov model to classify transportation noise events (car, truck, moped, aircraft and train) with higher accuracy than human listeners. Smart phones have been used to enable citizens to contribute to noise pollution maps [18, 19, 20, 21, 22]. Monitoring noise pollution with smart phones have several problems as reported by Santini et al. [23] including location of the phone (e.g. hand, pocket or backpack), modification of the detected sound by phone hardware and firmware (e.g. noise cancellation, low-pass filtering, automatic gain control), and power consumption limiting continuous monitoring duration. Previous works typically presents and compares noise pollution as a single number. As we will show through surveys, the impact of residential noise is too subjective and complex to be represented by a single metric. Our goal is to create a device to continuously and accurately monitor noise pollution, and data presentation methods to enable prospective home owners and renters to effectively evaluate noise pollution at several residential locations. To achieve these goals we decided to build our own microprocessor-based audio capture system to send noise analysis results as SMS text messages. We chose to build a custom system rather than use a smart phone to control the audio performance characteristics of the device, including sample rate, dynamic range, resolution and accuracy. We developed several graphical representations and a novel audio summarization technique which we evaluated with user studies. 2.2 Noise and Hearing The human ear detects minute changes in air pressure as sound. The ear is incredibly sensitive, with a dynamic range (the difference between the thresholds of hearing and pain) spanning 13 orders of magnitude (Table 1). To accommodate the large dynamic range, sound pressure level (SPL) is measured in logarithmic units of decibels (dB), with 0 dB defined as the threshold of hearing. A +3 dB change doubles the SPL and is the minimum increase in loudness perceivable by humans. A +10 dB increase is perceived as the doubling of loudness [24]. This implies that our noise sensor must have a large dynamic range (e.g. 30 dB to 90 dB) but with low resolution (1-3 dB). Sound varies in amplitude and frequency. Spectrum refers to the frequency components that make up a sound. White noise has energy spread equally across all frequencies. Pink noise, also called 1/f noise, has a power spectral density inversely proportional to frequency, and is produced by flowing water and distant highway traffic (Fig 1, left). Motorcycles, propeller airplanes and helicopters are noise sources that appear prominently in our noise sampling sites and have a strong periodic component. Compressed gas emanated from the motorcycles’ exhaust system and the

Monitoring Residential Noise for Prospective Home Owners and Renters

37

Table 1. Sound pressure levels (in decibels) of common loudness references Decibels 0 20 30 50-70 75 80 90 95 100 110 120-140 180

Example Threshold of hearing Rustling leaves Whisper, quiet library Normal conversation at 3 to 5 feet Loud Singing Telephone dial tone Train whistle at 500 feet, motorcycle, lawnmower Subway at 200 feet Diesel truck at 30 feet Jack hammer, rock concert, boom box Pain, gun blast, jet engine at 100 feet Death of hearing tissue

chopping of air by propellers produce distant frequencies in the noise spectra (Fig. 1, right). While previous work uses sophisticated signal processing combined with statistical models to automatically detect and categorize these noise sources [17], our noise detection algorithm is simple enough to run on a low power 8-bit microprocessor. Instead of automatically identifying the noise source, we record short segments (e.g. 3 seconds) so humans can judge for themselves the subjective impact of noise. 0

0

-10

Amplitude (dB)

Amplitude (dB)

-10 -20 -30 -40 -50 -60 -70

Airplane Bird Chirp Motorcycle

-20

-30

-40

Highway

Jet

-50 1

2

Frequency (kHz)

3

4

Airplane Motorcycle 100

200

300

400

500

Frequency (Hz)

Fig. 1. Spectra of several noise sources (left), sampled at 8 kHz, covering a frequency range from 100 Hz to 4 kHz. Periodic noise sources (right) are particularly disturbing.

3 Research Methodology The goal of our research is to develop an inexpensive, convenient and effective means of monitoring and comparing residential noise pollution. Our approach is to interview potential users to understand the significance of the problem, take some real-word measurements, build a system and verify it in the field, then test our solution with users. The paper is organized in the order we use to carry out our research: •

Conduct a survey to determine the importance of noise pollution on choosing a place to live and what noise sources are most disturbing

38

T. Zimmerman and C. Robson

• • • • •

Select three homes that represent a diversity of noise pollution environments and collect several days of audio recordings Determine a noise model that captures the salient noise features to efficiently quantify noise pollution Build hardware to measure salient noise features and verify in the field Design noise pollution data presentation methods Conduct a user survey to determine the effectiveness of noise pollution presentation methods.

4 Data Collection 4.1 Noise Pollution Survey We conducted a survey of 82 people to determine if noise pollution is perceived as an important factor in choosing a place to live, and what noise pollution sources cause problems. The subjects were adult technical and administrative staff, male and female, from our research laboratory. Practically everyone (93% of respondents) indicated that a quiet neighborhood is important in selecting a place to live (55% indicated ``very important''). We classified people into ``primarily apartment or townhouse residents'' and ``primarily stand-alone house residents'' based on their reported residence history. People who have lived primarily in houses care more about noise then people who live in apartments (100% vs. 88%), and many feel it is ``very important'': (64% vs. 49%). About 2/3 of both apartment-renters and house-dwellers remember discovering noises in a new residence that they didn't know about before hand (65%, 64%). House-dwellers are a little more likely to read and buy a noise survey (read: 77% vs. 68% ; buy:87% vs. 74%), and would pay more ($30 vs. $25 on average). Table 2. Survey results of which noise sources annoy residents

Noise Source Loud music Yard noise Traffic Parties Children Babies Construction Arguments Pets

Apartment or Townhouse Resident 55% 43% 43% 49% 20% 31% 45% 33% 33%

House Resident 64% 48% 42% 24% 9% 18% 30% 42% 42%

The most problematic times for noise are nights, followed by evenings and weekend mornings. The results of the relative annoyance and type of noise sources reported by survey participants are listed in Table 2. Neighbors' loud music is the biggest noise complaint for both groups, followed by yard services and traffic noise.

Monitoring Residential Noise for Prospective Home Owners and Renters

39

Traffic noise is the dominant noise pollution source detected in our monitoring since it happens every day. Leaf blowers are the biggest write-in noise complaint. Most people (52%) do nothing in response to noise pollution. Only about 1 in 5 people have confronted a neighbor over a noise problem. One in four house-dwellers called the police over a noise incident, and the same percentage of apartment-renters has either called the police or the building superintendent/security. A further 4% have contacted the humane society over pet noises. A surprising 4% of respondents (both apartment and house dwellers) reported retaliating against loud neighbors by making noise themselves. The variety of noise sources and the impact on individual demonstrates the subjective nature of noise pollution, and foreshadows an important lesson we learned by running our experiments and surveys; the impact of noise pollution must be evaluated subjectively by the individual. 4.2 Selection Monitoring Sites In order to refine our understanding of what data should be collected to evaluate residential noise pollution, we made preliminary recordings at five locations. We listened to the recordings and selected three houses to represent a diversity of noise pollution environments. House A is located on a residential street far from any expressway or major road, representing a suburban location. It is a very quiet location punctuated by an occasional vehicle. House B is located in a large county park, down an unpaved road and represents a very rural setting. It is next to a creek, providing a source of pink noise (what is considered “white noise” in casual usage). We chose House A and B to study the relative impact of transient noise juxtaposed with background noise. A transient noise event (e.g. a car passing by) at house A may be perceived as more disturbing than the same magnitude event at house B since house A has a lower average sound level. However those noise events may not be as disturbing as a constant higher level of background noise. House C is located in a dense suburban development within a few blocks of two freeways, providing a variety of human activity and transportation noise sources, as would be encountered in an urban environment. 4.3 Field Recordings For each of the three houses field recordings are made using an external electret microphone and laptop computer. The microphone has a ten foot cable, allowing it to be placed outside a window, while the laptop is located inside, plugged into the mains, to enable many hours of continuous recording. The microphone is chosen for its small size (6 mm diameter), low cost (region = entire world; Assign all geo-tags to root; PartitionSimple( root ); PartitionSimple( Partition cp ) { IF( cp->geo-tags region = quadrant i of cp->region; Set p->label = Concatenate( cp->label, i ); FOR EACH geo-tag gt IN cp IF (gt lies within p) THEN Assign gt to p; END PartitionSimple( p ); i = i + 1; END } While there are partitioning schemes such as Hierarchical Triangular Maps which would result in a more even spacial partitioning, these scheme require greater computational cost at run-time. Furthermore such partitioning schemes do not provide significant performance improvements.

n-Gram Geo-trace Modeling

101

A

AA

AB A

AC

AAA

AD

AAB

AA AB AAC AAD AADA ADB AC ADCADADD Fig. 1. The map on the left displays a partition based on the frequency of visits by the user. The image on the right shows an example of how labels are assigned to partitions in a density based partitioning scheme.

2.2

Time and Direction Features

We also extract features other than location information for geo-trace modeling. – direction of displacement. The direction of displacement of a geo-tag gi is the angle of the directional vector gi − gi−1 , i.e., the direction the user moves from his previous location to the current location. The direction is 0 if gi − gi−1 points to North and 90 if East. Similar to quantizing locations, we also quantized direction of displacement into B sectors where B > pj, pi may be used as pij.

Autonomous Construction of a WiFi Access Point Map

121

Δ = 2⋅ pj

Fig. 1. At a point, a smartphone measures received signal strengths from two AP, Yi and Yj. The dissimilarities pi and pj are measured distances, and two circles show locatable positions of AP i and j, respectively. Then, the candidate of pij lies between pi-pj and pi+pj. 2×pj is the extent of possible errors of the dissimilarity pij.

On the estimation of the dissimilarity, we can predict the extent of possible errors. In the figure, the error of an estimated dissimilarity is 2×pj at worst. This means that error is possibly proportional to the smaller one between pi and pj. In the estimation of dissimilarity, another issue should be considered. In the beginning we assumed that we could get many scans in various positions. As a result, we have several scans including the same APs. Figure 2 illustrates the situation. Four scans are gathered at positions m1, m2, m3, and m4, and their dissimilarities measured by the smartphone are (pj(1), pi(1)), (pj(2), pi(2)), (pj(3), pi(3)), and (pj(4), pi(4)), respectively. Considering we estimate dissimilarity pij from them, estimating four dissimilarities, pij(1), pij(2), pij(3) and pij(4) and averaging them is one possible solution. However, since most scans are not measured on the shortest path connecting two APs, its value is usually larger than a real distance. Hence, we select and use a radio scan that is the most proper to infer a real distance from. Measurement positions m1 and m3 compared, pj(1) is equal to pj(3), but pi(1) is larger than pi(3). Then, m3 is selected as a candidate. Compared with m2, pj(2)+pi(2) is smaller than pj(3)+pi(3), then m2 is newly selected. Since m2 and m4 are on the shortest path, pj(2)+pi(2) is similar to pj(4)+pi(4). Hence, m2 or m4 may be equally selected. Until now we did not consider that received signal strengths are disrupted by noise. In reality, the measured signal strength is not always consistent with the expected distance. Hence, in the selection of a radio scan, we take an approach to reduce an expected error. One clue is the extent of possible errors shown in Figure 1. It is proportional to the value of the smaller one of dissimilarities between APs and a smartphone. Therefore, we finally select the radio scan with the smallest dissimilarity value, which in this case is m2.

122

J. Koo and H. Cha

pj(3)

Yj

m3

pj(2) pi(2) pj(1)

m2

m1

pj(4)

m4 p i(4)

pi(3)

Yi

pi(1)

Fig. 2. At several points the received signal strengths from two APs, Yi and Yj are measured. The dashed line shows the shortest path between the two APs. Here, pj(2) 15) Low Show Me (ratings spread out but most < 15)

Low Medium (recognition accuracy (recognition accuracy between 71-80%) between 81-90%)

Select Open Delete

Pick Up

Move Back Zoom Out

Zoom In Close Search

1. Drop: this gesture should be retained in its current form as both the user and system performance are high. 2. Select, Open and Delete: these gestures are regarded by users as an excellent match to their corresponding tasks. However, the medium system recognition rates indicate that work needs to be undertaken to improve the system. 3. Pick Up: similarly, Pick Up should be retained due to its high user rating and work should be undertaken on improving the system. 4. Move Forward and Show Me: participants gave these gestures a medium and low rating respectively, indicating that these gestures were only a reasonable or low match to the task being performed. However, both these gestures have high system recognition rates. Therefore, it is recommended that these gestures should be retained and the user should be encouraged to learn the gestures. In the case of Show Me, this is further corroborated by Study 1 where, setting aside the simple Point gesture as discussed above, the Show Me gesture chosen was easily the most popular gesture generated for this task. Show Me is sufficiently abstract a task that it is unsurprising that Study 2 participants did not rank it highly. It seems likely, again corroborated

3D Gesture Recognition: An Evaluation of User and System Performance

311

by the findings of Study 1, that they would have had similar or greater concerns with any other gesture chosen to perform this task. 5. Move Back and Zoom Out: the generalized design recommendations suggest that either the gesture or the system could be altered. However, based on the mirrors of these gestures (Move Forward and Zoom In) being retained in their current form, it would seem sensible to recommend that the Move Back and Zoom Out gestures should be retained in their current form and improvements made to the gesture recognition system. 6. Zoom In: although the system recognition rate for Zoom In was low, participants reported that the gesture was a reasonable match to the action being performed. Therefore, it is recommended that improvements are made to the system rather than altering the gesture. 7. Close and Search: these gestures should be rejected as participants did not regard them as matching their tasks well and the system recognition rate was poor.

7 Conclusions and Future Work In this paper we have reported a series of empirical studies and system development undertaken to investigate the use of gestures as an interaction technique in pervasive computing environments. In phase 1, participants were asked to generate gestures that we categorized based on the degree of consensus and the number of different gestures generated by participants. Additionally, we discovered that many of the gestures generated by participants were performed in 3D. Therefore, in phase 2, we implemented a computer vision based 3D gesture recognition system and applied it in a further study in which participants were trained on the archetypal gestures derived from phase 1. Participants were asked to perform tasks using these gestures. From this study we were able to collect data on both user performance and preferences and system performance. Finally, we explored the trade off between the requirement for gestures to support high system performance versus the requirement for gestures to support high human performance and preference, deriving design recommendations. Deriving user-generated gestures, as we did in phase 1, enabled us to define an archetypal gesture set for specific types of interactions in pervasive computing environments. The advantage to this approach is that we are able to define gestural interactions that are considered natural and intuitive, based on user expectations and preferences and the degree of consensus amongst participants. However, considering only the user requirements for gestures when implementing a gesture recognition system for use in pervasive computing environments excludes from the equation the needs of the system. Therefore, we proposed a method by which we could compare both user performance and preference and system performance. The resulting general design recommendations indicate where the archetypal gestures can remain unchanged, where adjustments need to be made to the gesture performance by the user, where development effort is needed to improve a recognition implementation and where a potential gesture could be rejected.

312

M. Wright et al.

We illustrated the application of these general recommendations to our particular gesture set and system implementation. As part of our future work we wish to define a framework that designers can employ to add new gestural interactions to our archetypal gesture set for new tasks. This framework should not only take into account how to generate gestures for particular tasks but also the practicalities of gesture recognition and interaction. For example, the technology used to recognize gestures (e.g. computer vision with 2D or 3D cameras, accelerometers etc) and the context of the interaction. Furthermore, we plan to identify further gestures using this framework and evaluate them with a range of gesture recognition systems for pervasive computing environments. The aim is to compare these different systems, exploring the trade off between user and system performance. From these studies, we aim to provide insights into the types of gestural interactions that work well – and poorly – for different recognition technologies in different contexts.

References 1. Accord_Statistics_Library, http://www.crsouza.com 2. Kendon, A.: Current Issues in the Study of Gesture. In: The Biological Foundations of Gestures: Motor and Semiotic Aspects, pp. 23–47. Lawrence Erlbaum, Mahwah (1986) 3. Ramamoorthy, A., Vaswani, N., Chaudhury, S., Banerjee, S.: Recognition of Dynamic Hand Gestures. Pattern Recognition 36(9), 2069–2081 (2003) 4. Keskin, C., Erkan, A., Akarun, L.: Real Time Hand Tracking and 3D Gesture Recognition for Interactive Interfaces using HMM. In: ICANN/ICONIPP 2003, pp. 26–29 (2003) 5. Kray, C., Nesbitt, D., Dawson, J., Rohs, M.: User-Defined Gestures for Connecting Mobile Phones, Public Displays and Tabletops. In: MobileHCI 2010, pp. 239–248 (2010) 6. Efron, D.: Gesture and Environment. Morningside Heights. King’s Crown Press, New York (1941) 7. McNeill, D.: Hand and Mind: What Gestures Reveal about Thought. University of Chicago Press, Chicago (1992) 8. Chen, F., Fu, C., Huang, C.: Hand Gesture Recognition Using a Real-Time Tracking Method and Hidden Markov Models. Image and Vision Computing 21(8), 745–758 (2003) 9. Poggi, I.: From a Typology of Gestures to a Procedure for Gesture Production. In: International Gesture Workshop 2002, pp. 158–168 (2002) 10. Wobbrock, J.O., Morris, M.R., Wilson, A.D.: User-Defined Gestures for Surface Computing. In: CHI 2009, pp. 1083–1092 (2009) 11. Oka, K., Sato, Y., Koike, H.: Real-Time Fingertip Tracking and Gesture Recognition. IEEE Computer Graphics and Applications, 64–71 (2002) 12. Elmezain, M., Al-Hamadi, A., Appenrodt, J., Michaelis, B.: A Hidden Markov ModelBased Isolated and Meaningful Hand Gesture Recognition. Electrical, Computer, and Systems Engineering 3(3), 156–163 (2009) 13. Nielsen, M., Störring, M., Moeslund, T.B., Granum, E.: A Procedure for Developing Intuitive and Ergonomic Gesture Interfaces for HCI. In: Camurri, A., Volpe, G. (eds.) GW 2003. LNCS (LNAI), vol. 2915, pp. 409–420. Springer, Heidelberg (2004) 14. Wu, M., Balakrishnan, R.: Multi-Finger and Whole Hand Gestural Interaction Techniques for Multi-User Tabletop Displays. In: UIST 2003, pp. 193–202 (2003) 15. Mistry, P., Maes, P., Chang, L.: WUW - Wear ur World - A Wearable Gestural Interface. In: CHI 2009, pp. 4111–4116 (2009)

3D Gesture Recognition: An Evaluation of User and System Performance

313

16. Malik, S., Ranjan, A., Balakrishnan, R.: Interacting with Large Displays from a Distance with Vision-Tracked Multi-Finger Gestural Input. In: UIST 2005, pp. 43–52 (2005) 17. Fikkert, W., van der Vet, P., van der Veer, G., Nijholt, A.: Gestures for Large Display Control. In: Kopp, S., Wachsmuth, I. (eds.) GW 2009. LNCS, vol. 5934, pp. 245–256. Springer, Heidelberg (2010) 18. Wu, Y., Huang, T.: Vision-Based Gesture Recognition: A Review. In: Braffort, A., Gibet, S., Teil, D., Gherbi, R., Richardson, J. (eds.) GW 1999. LNCS (LNAI), vol. 1739, pp. 103–115. Springer, Heidelberg (2000)

Recognition of Hearing Needs from Body and Eye Movements to Improve Hearing Instruments Bernd Tessendorf1 , Andreas Bulling2 , Daniel Roggen1, Thomas Stiefmeier1 , Manuela Feilner3 , Peter Derleth3 , and Gerhard Tr¨ oster1 1

Wearable Computing Lab., ETH Zurich Gloriastr. 35, 8092 Zurich, Switzerland {lastname}@ife.ee.ethz.ch 2 Computer Laboratory, University of Cambridge 15 JJ Thomson Avenue, Cambridge CB3 0FD, United Kingdom {firstname.lastname}@acm.org 3 Phonak AG, Laubisr¨ utistrasse 28, 8712 St¨ afa, Switzerland {firstname.lastname}@phonak.com

Abstract. Hearing instruments (HIs) have emerged as true pervasive computers as they continuously adapt the hearing program to the user’s context. However, current HIs are not able to distinguish different hearing needs in the same acoustic environment. In this work, we explore how information derived from body and eye movements can be used to improve the recognition of such hearing needs. We conduct an experiment to provoke an acoustic environment in which different hearing needs arise: active conversation and working while colleagues are having a conversation in a noisy office environment. We record body movements on nine body locations, eye movements using electrooculography (EOG), and sound using commercial HIs for eleven participants. Using a support vector machine (SVM) classifier and person-independent training we improve the accuracy of 77% based on sound to an accuracy of 92% using body movements. With a view to a future implementation into a HI we then perform a detailed analysis of the sensors attached to the head. We achieve the best accuracy of 86% using eye movements compared to 84% for head movements. Our work demonstrates the potential of additional sensor modalities for future HIs and motivates to investigate the wider applicability of this approach on further hearing situations and needs. Keywords: Hearing Instrument, Assistive Recognition, Electrooculography (EOG).

1

Technology,

Activity

Introduction

Hearing impairment increasingly affects populations worldwide. Today, about 10% of the population in developed countries suffer from hearing problems; in the U.S. even 20% adolescents suffers from hearing loss [22]. Over the last generation, the hearing impaired population grew at a rate of 160% of U.S. population growth [13]. About 25% of these hearing impaired use a hearing instrument (HI) to support them in managing their daily lives [13]. K. Lyons, J. Hightower, and E.M. Huang (Eds.): Pervasive 2011, LNCS 6696, pp. 314–331, 2011. c Springer-Verlag Berlin Heidelberg 2011 

Recognition of Hearing Needs from Body and Eye Movements

315

Over the last decade considerable advances have been achieved in HI technology. HIs are highly specialised pervasive systems that feature extensive processing capabilities, low power consumption, low internal noise, programmability, directional microphones, and digital signal processors [10]. The latest of these systems –such as the Exelia Art by Phonak– automatically select from among four hearing programs. These programs allow the HI to automatically adjust the sound processing to the users’ acoustic environment and their current hearing needs. Examples of hearing need support include noise suppression and directionality for conversations in noisy environments. Satisfying the users’ hearing needs in as many different situations as possible is critical. Already a small number of unsupported listening situations causes a significant drop in overall user satisfaction [14]. Despite technological advances current HIs are limited with respect to the type and number of hearing needs they can detect. Accordingly, only 55% of the hearing impaired report of being satisfied with the overall HI performance in common day-to-day listening situations [14]. This is caused, in part, by the fact that adaption is exclusively based on sound. Sound alone does not allow to distinguish different hearing needs if the corresponding acoustic environments are similar. We call this limitation the acoustic ambiguity problem. 1.1

Paper Scope and Contributions

In this work we investigate the feasibility of using additional modalities, more specifically body and eye movements, to infer the hearing needs of a person. As a first step toward resolving the acoustic ambiguity problem we focus on one particular listening situation: the distinction between concentrated work while nearby persons have a conversation from active involvement of the user in a conversation. The specific contributions are: 1) the introduction of context-aware HIs that use a multi-modal sensing approach to distinguish between acoustically ambiguous hearing needs; 2) a methodology to infer the hearing need of a person using information derived from body and eye movements; 3) an experiment to systematically investigate the problem of acoustically ambiguous hearing needs in an office environment, and 4) the evaluation of this methodology for automatic hearing program selection. 1.2

Paper Organisation

We first provide an overview of the state-of-the-art in HI technology, introduce the mechanisms that allow HIs to adapt to the user’s hearing needs, and discuss the limitations of current systems. We then survey related work and detail our methodology to infer the user’s hearing need from body and eye movements. We describe the experiment, discuss its results, and provide a brief outlook on the technical feasibility of integrating body and eye movements into HIs.

316

B. Tessendorf et al.

Fig. 1. Components of a modern behind-the-ear (BTE) HI [19]

2

Hearing Instrument Technology

Figure 1 shows the components of a modern behind-the-ear (BTE) HI. HIs are also available in smaller form factors. E.g., Completely-in-the-Canal (CIC) devices can be placed completely inside the user’s ear canal. Current systems include a DSP, multiple microphones to enable directivity, a loudspeaker, a telecoil to access an audio induction loop, and a high-capacity battery taking up about a quarter of the HI housing. HIs may also integrate a variety of other accessories such as remote controls, Bluetooth, or FM devices as well as the user’s smart phone to form wireless networks, so-called hearing instrument body area networks (HIBANs) [3]. These networking functionalities are part of a rising trend in higher-end HIs. This motivates and supports our investigation of additional sensor modalities for HIs that may eventually be included within the HI itself, or within the wireless network controlled by the HI. A high-end HI comprises two main processing blocks as shown in Figure 2. The audio processing stages represent the commonly known part of a HI. It performs the traditional audio processing function of the HI and encompasses audio pickup, processing, amplification and playback. The second processing block is the classifier system. It estimates the user’s hearing need based on the acoustic environment of the given situation, and adjusts the parameters of the audio processing stages accordingly [12]. The classification is based on spectral and temporal features extracted from the audio signal [4]. The classifier system selects the parameters of the audio processing stages from among a discrete set of parameters known as hearing programs. The hearing programs are optimised for different listening situations. Most current high-end HIs distinct four hearing programs: natural, comprehensive hearing (Speech), speech intelligibility in noisy environments (Speech in Noise), comfort in noisy environments (Noise), and listening pleasure for a source with high dynamics (Music). The hearing programs represent trade-offs, e.g. speech intelligibility versus naturalness of sound,

Recognition of Hearing Needs from Body and Eye Movements

317

Fig. 2. Bottom: audio processing stages of the HI, from microphone pick-up to amplified and processed sound playback. Top right: classification of the acoustic environment based on sound to adjust the parameters of the audio processing stages. Top left: the extension proposed in this paper. Body and eye movement data are included in the classification system to select the appropriate hearing program. (Figure extended from [10]).

or omnidirectional listening versus directivity. The automatic program selection allows the hearing impaired to use the device with no or only a few manual interactions such as program change and volume adjustment. Adaptive HIs avoid drawing attention to the user’s hearing deficits. Users consider automatic adaption mechanisms as useful [4]. Further technical details on HI technology can be found in [10, 21]. 2.1

The Acoustic Ambiguity Problem

HIs select the most suitable hearing program according to the user’s acoustic environment. The current acoustic environment is used as a proxy for the user’s actual hearing need. This approach works well as long as the acoustic environment and hearing need are directly related. This assumption does not hold in all cases and leads to a limitation we call the acoustic ambiguity problem: Specifically, in the same acoustic environment a user can have different hearing needs that require different hearing programs. A sound-based adaption mechanism cannot distinguish between these different hearing needs. Therefore, it is important to not only analyze the acoustic environment but to also assess the relevance of auditory objects [23]. The challenge here is not the effectiveness of the dedicated hearing programs but rather automatically adapting the hearing program to the specific hearing need, rather than to the acoustic environment. The following story illustrates the acoustic ambiguity problem: Alice suffers from hearing impairment and works in an open office space. Alice is focused on her assigned task when Bob enters the office space to talk to a

318

B. Tessendorf et al.

colleague sitting next to Alice. Alice’s HI senses speech in noise and optimizes for speech intelligibility. She now has a hard time focussing on her work, as the HI adapts to the distracting conversation that occurs around her. Then Bob starts talking to Alice. She now needs the HI to support her interaction with colleagues in the noisy environment. Alice doesn’t like to select hearing programs manually and desires a robust automatic adaption to her current hearing need. In the first case, the HI user takes part in a conversation, in the second case, the user could be concentrated on her work and and experiences the conversation as noise. The key challenge in this example is to assess the relevance of speech in the acoustic environment to the HI user. The HI needs to choose between a hearing program that optimizes speech intelligibility and a hearing program treating the speech as noise for user comfort. In both situations, the HI detects the same acoustic environment and thus cannot select a suitable hearing program in both of the cases. A possible strategy is a static “best guess”choice based on a predefined heuristic rule. It could favor speech intelligibility over comfort in noise as social interaction is generally considered important. Other typical situations in which state of the art classification systems fail include listening to music from the car radio while driving or conversing in a cafe with background music [10]. 2.2

Vision of a Future HI

We envision the use of additional modalities to distinguish between ambiguous hearing need requirements in the same acoustic environment. These modalities will be included within the HI itself, or within a wireless network controlled by the HI. Wireless networking functionalities are now starting to appear in higherend HIs. These new sensors need not be specifically deployed for HIs: they may be shared with other assistive technologies, such as systems designed to detect falls or to monitor physiological parameters. Thus, we see the HI as one element included in a broader set of ambient assisted living technologies. Wearable and textile integrated sensors have become available and sensor data from a mobile phone that may be carried by an individual can be used. We believe the next step in HI technology is to utilize this infrastructure to improve HI performance.

3

Related Work

Various sensor modalities have been proposed to detect social interaction, conversation, or focus of attention from wearable sensors. In [8] body-worn IR transmitters were used to measure face-to-face interactions between people with the goal to model human networks. All partners involved in the interaction needed to wear a dedicated device. In [11] an attentive hearing aid based on an eye-tracking device and infrared tags was proposed. Wearers should be enabled to “switch on” selected sound sources such as a person, television or radio by looking at them. The sound source needed to be attached with a device that catched the attention of the

Recognition of Hearing Needs from Body and Eye Movements

319

hearing aid’s wearer so that only the communication coming from the sound source was heard. In [7] different office activities were recognised from eye movements recorded using Electrooculography with an average precision of 76.1% and recall of 70.5%: copying a text between two screens, reading a printed paper, taking hand-written notes, watching a video, and browsing the web. For recognising reading activity in different mobile daily life settings the methodology was extended to combine information derived from head and eye movements [6]. In [18] a vision-based head gesture recognizer was presented. Their work was motivated by the finding that head pose and gesture offer key conversational grounding cues and are used extensively in face-to-face interaction among people. Their goal was to equip an embodied conversational agent with the ability to perform visual feedback recognition in the same way humans do. In [9] the kinematic properties of listeners’ head movements were investigated. They found a relation of timing, tempo and synchrony movements of responses to conversational functions. Several researchers investigated the problem of detecting head movements using body-worn and ambient sensors. In [1] an accelerometer was placed inside HI-shaped housing and worn behind the ear to perform gait analysis. However, the system was not utilised to improve HI behavior. Capturing the user’s auditory selective attention helps to recognise a person’s current hearing need. Research in the field of electrophysiology focuses on mechanisms of auditory selective attention inside the brain [24]. Under investigation are event-related brain potentials using electroencephalography (EEG). In [17] the influence of auditory selection on the heart rate was investigated. However, the proposed methods are not robust enough yet to distinguish between hearing needs and are not ready yet for deployment in mobile settings. All these approaches did not consider sensor modalities which may be included in HIs, or assumed the instrumentation of all participants in the social interactions. In [26], analysis of eye movements was found to be promising to distinguish between working and interaction. Head movements were found to be promising to detect whether a person is walking alone or walking while having a conversation. However, the benefit of combining modalities was not investigated. Moreover, the actual improvement in hearing program selection based on the context recognition was not shown.

4 4.1

Experiment Procedure

The experiment in this work was designed to systematically investigate acoustically ambiguous hearing needs in a reproducible and controllable way, still remaining as naturalistic as possible. We collected data from 11 participants (six male, five female) aged between 24 and 59 years, recruited from within the lab. The participants were normal hearing and right handed without any known attributes that could impact the results.

320

B. Tessendorf et al.

Table 1. Experimental procedure to cover different listening situations and hearing needs. The procedure was repeated eight times with the different office activities mentioned above and with the participants being seated and standing. Time Slot [min] Situation 1 2 3 4 5

Participant and colleague are working Disturber and colleague converse Disturber and participant converse Disturber and colleague converse Colleague and participant converse

Hearing Need work work conversation work conversation

The experiment took place in a real but quiet office room. The participant and an office colleague worked in this office. A third person, the disturber, entered the office from time to time to involve them in a conversation. The participants were given tasks of three typical office activities: Reading a book, writing on a sheet of paper, typing text on a computer. The participants had no active part in controlling the course of events. They were instructed to focus on carrying out their given tasks and to react naturally. This assures that the resulting body and eye movements are representative for these activities. The experiment was split in one minute time slots each representing a different situation and hearing need (see Table 1). In the first minute, the participant worked concentrated on his task. In the second minute, the participant tried to stay concentrated while the office colleague was talking to the disturber. In the third minute, the participant was interrupted and engaged in a conversation with the disturber. In the fourth minute, the disturber talked to the colleague again. In the fifth minute, the participant and the colleague had a conversation. This procedure was repeated eight times with the office activities mentioned above and with the participants being seated and standing. The total experiment time for each participant was about 1 hour. We then assigned each of these activities to one of the following two hearing needs. Conversation includes situations in which the participant is having a conversation. The HI is supposed to optimize for speech intelligibility, i.e. the hearing program should be “Speech in Noise” throughout. Figure 3 shows all four combinations of sitting and standing while talking to the conversation partners. Work includes situations in which the participant is carrying out a work task. This case covers situations in which no conversation is taking place around him and situations in which two colleagues are having a conversation the participant is not interested in. The HI is supposed to be in a noise suppression program called “Noise”. Figure 4 shows the participant work sitting and standing. Figure 5 shows the participant work in speech noise for the sitting case only. 4.2

Performance Evaluation

We investigate how accurate we can distinguish the two classes conversation and work. The hearing programs we declared as optimal for each of the situations

Recognition of Hearing Needs from Body and Eye Movements

321

Fig. 3. Situations with the hearing need Conversation, including all four combinations of sitting and standing conversation partners. The HI is supposed to be in a program optimizing for speech intelligibility (Speech In Noise).

Fig. 4. Situations with the hearing need Noise for the case Work. Working tasks include reading a book, writing on a sheet of paper, and typing a text on the computer. The participant works sitting and standing.

Fig. 5. Situations with the hearing need Noise for the case Work in Speech Noise. The participant tries to focus on his working task while two colleagues are having a conversation. Only the sitting case is shown here.

322

B. Tessendorf et al.

served as ground truth: Speech In Noise for conversation and Noise for work. It is important to note that the Noise program is not optimized for supporting the user with concentrated work, but is the best choice among the available hearing programs in our conversation cases. Robust detection of working situations would enable to augment existing HIs with a dedicated program and sound signal processing strategies. For evaluation we compare for each signal window whether the classification result corresponds to the ground truth. We count how often classification and ground truth match in this two-class problem to obtain an accuracy value. In addition, we obtained as a baseline the classification result based on sound. To this end, we analysed the debug output of an engineering sample of a HI1 . 4.3

Data Collection

For recording body movements we used an extended version of the Motion Jacket [25]. The system features nine MTx sensor nodes from Xsens Technologies each comprising a 3-axis accelerometer, a 3-axis magnetic field sensor, and a 3-axis gyroscope. The sensors were attached to the head, the left and right upper and lower arms, the back of both hands, and the left leg. The sensors were connected to two XBus Masters placed in a pocket at the participants’ lower back. The sampling rate is 32 Hz. For recording eye movements we chose Electrooculography (EOG) as an inexpensive method for mobile eye movement recordings; it is computationally light-weight and can be implemented using on-body sensors [5]. We used the Mobi system from Twente Medical Systems International (TMSI). The device records a four-channel EOG with a joint sampling rate of 128 Hz. The participant wore it on a belt around the waist as shown in Figure 6. The EOG data was collected using an array of five electrodes positioned around the right eye as shown in Figure 6. The electrodes used were the 24mm Ag/AgCl wet ARBO type from Tyco Healthcare equipped with an adhesive brim to stick them to the skin. The horizontal signal was collected using two electrodes on the edges of both eye sockets. The vertical signal was collected using one electrode above the eyebrow and another on the lower edge of the eye socket. The fifth electrode, the signal reference, was placed away from the other electrodes in the middle of the forehead. Eye movement data was saved together with body movement data on a netbook in the backpack worn by the participant. We used two Exelia Art 2009 HIs from Phonak worn behind the left and the right ear. For the experiment we modified the HIs to use them for recording only the raw audio data rather than logging the classification output in real-time. With the raw audio data the HI behavior in the conducted experiment can be reconstructed offline. Using the same noise for overlay gives equal conditions for each participant to rule out different background noise as an effect on the resulting performance. Moreover, it is possible to simulate for different acoustic environments, e.g. by overlaying office noise. Another advantage of recording 1

This work was carried out in collaboration with a hearing instrument company.

Recognition of Hearing Needs from Body and Eye Movements

323

Fig. 6. Sensor setup consisting of HIs (1), a throat microphone (2), an audio recorder (3), five EOG electrodes (h: horizontal, v: vertical, r: reference), as well as the Xsens motion sensors placed on the head (4a), the upper (4b) and lower (4c) arms, the back of both hands (4d), the left leg (4e), two XBus Masters (4d), and the backpack for the netbook (5)

raw audio data is the possibility to simulate the behavior with future generation of HIs. We used a portable audio recorder from SoundDevices to capture audio data with 24 bit at 48 kHz. Although not used in this work, participants also wore a throat microphone recording a fifth audio channel with 8 bit at 8 kHz. Based on both sound recordings we investigate detection of active conversation based on own-speech detection in future research. Data recording and synchronisation was handled using the Context Recognition Network (CRN) Toolbox [2]. We also videotaped the whole experiment to label and verify the synchronicity of the data streams.

5 5.1

Methods Analysis of Body Movements

We extract features on a sliding window on the raw data streams from the 3-axis accelerometers, gyroscopes and magnetometers. For the magnetometer data we calculate mean, variance, mean crossing and zero crossing. For the gyroscope data we additionally extract the rate of peaks in the signal. For the accelerometers data we calculate the magnitude based on all three axes. Based on a

324

B. Tessendorf et al.

parameter sweep we selected a window size of 3 seconds and a step size of 0.5 second for feature extraction. 5.2

Analysis of Eye Movements

EOG signals were first processed to remove baseline drift and noise that might hamper eye movement analysis. Afterwards, three different eye movement types were detected from the processed EOG signals: saccades, fixations, and blinks. All parameters of the saccade, fixation, and blink detection algorithms were fixed to values common to all participants. The eye movements returned by the detection algorithms were the basis for extracting different eye movement features using a sliding window. Based on a parameter sweep we set the window size to 10 seconds and the step size to 1 second (a more detailed description is outside the scope of this paper but can be found in [7]). 5.3

Feature Selection and Classification

The most relevant features extracted from body and eye movements were selected with the maximum relevance and minimum redundancy (mRMR) method [20]. Classification was done using a linear support vector machine (see [15] for the specific implementation we used). We set the penalty parameter to C = 1 and the tolerance of termination criterion to  = 0.1. Classification and feature selection were evaluated using a leave-one-participant-out cross-validation scheme. The resulting train and test sets were standardised to have zero mean and a standard deviation of one. Feature selection was performed solely on the training set. 5.4

Analysis of Sound

We used the classification output of commercial HIs as a baseline performance for sound based classification. We electrically fed the recorded audio stream described in section 4.3 back into HIs and obtained the selected hearing programs over time with a sampling rate of 10 Hz. To simulate a busy office situation we overlaid the recorded raw audio data with typical office background noise. In silent acoustic environments without noise, the hearing instrument remains mainly in the Clean Speech program for both the working and the conversation situation. We focus on the scenario with noise: The HI needs to decide wether optimizing for speech is adequate or not. 5.5

Data Fusion

To combine the unimodal information from the different motion sensors we used a fusion approach on feature-level. We built a feature vector comprising features from each of the sensors. To combine the multimodal information from body movement, eye movement, and sound we used majority voting as a standard fusion method on classifier-level. When there was no majority to make a decision we repeated the most recent decision. In this way, we suppress hearing program changes based on low confidence.

Recognition of Hearing Needs from Body and Eye Movements

6

325

Results and Discussion

6.1

Analysis of the Different Modalities

We first evaluated the performance of the different modalities. Accuracies are given for the two-class classification problem comprising active conversation and working while colleagues are having a conversation. Figure 7 shows the accuracies for distinguishing the hearing needs using sound, body movements, eye movements, and combinations of these modalities averaged over all participants. The results for body movements are based on sensors attached to all nine body locations whereas the results for sound-based adaption are based on the classification output of the HIs. The limited recognition accuracy of 77% for adaption based on sound is a consequence of the acoustic ambiguity problem that has been provoked in this scenario. The sound based analysis does not distinguish between relevant and irrelevant speech. The HI optimizes for speech in both of the cases described in section 4.1: When the participant is having a conversation and also when the colleagues are having a conversation. As can be seen from Figure 7, recognition based on body movement data from all available movement sensors (placed at head, back, arms, hands, leg) achieves the best performance with an accuracy of 92%. Adaptation based on eye movement performs slightly worse with 86% accuracy. Looking at combinations of different modalities shows that the joint analysis of body and eye movements has an accuracy of 91%, sound and body movement results in 90% accuracy, and combination of sound and eye movements yields 85% accuracy. Complementing body movements with eye movements or sound results in a lower standard standard deviation, meaning more robustness across different users. First results suggest the inclusion of movement sensors additionally to sound into the HI. 6.2

Analysis of Body Locations

Accuracy [%]

Based on these findings we selected body movements for further analysis. We investigated on which body locations the movement sensors provided the highest

100 90 80 70 60 50

77%

Sound

86%

Eye Mov.

92%

91%

90%

85%

89%

Body Mov.

Body Mov. Eye Mov.

Sound Body Mov.

Sound Eye Mov.

Body Mov. Eye Mov. Sound

Fig. 7. Accuracies for distinguishing the hearing needs in our scenario based on sound, eye movements, body movements (placed at head, back, arms, hands, leg), and all possible combinations. Results are averaged over all participants with the standard deviation indicated with black lines.

326

B. Tessendorf et al.

100

Accelerometer Gyroscope Magnetometer Fusion

Accuracy [%]

90 80 70 60 50

head

back

rua

rla rha lua lla Position of sensors on the body

lha

leg

Fig. 8. Accuracies of adaption based on individual sensors (Accelerometer, Magnetometer, Gyroscope) for each of the 9 locations on the body: head, back, left upper arm (lua), left lower arm (lla), left hand (lha), right upper arm (rua), right lower arm (rla), right hand (rha), and leg. Results are averaged over all participants with the standard deviation indicated with black lines.

accuracies. Figure 8 provides the accuracies for adaption using individual sensors (Accelerometer, Magnetometer, Gyroscope) as well as using sensor fusion on feature level for each of the nine body locations averaged over all participants. Figure 8 shows that from all individual body locations the sensor on the head yields the highest performance with accuracies between 72% for the gyroscope and 85% for the accelerometer. It is interesting to note that fusing the information derived from all three sensors types at the head does not further improve recognition performance (see first group of bars in Figure 8). For all other body location, sensor fusion consistently yields the best recognition performance. Single sensors placed on other body locations perform considerably worse with accuracies ranging from 59% (magnetometer on the left upper arm, lua) to 73% (accelerometer on the right hand, rha). These sensors may still prove beneficial if combined with other sensors located at other parts of the body. The higher utility of analyzing sensors on the right arm (rha, rla, rua) can be explained by the fact that all participants were right handed. 6.3

Further Analysis of the Head Location

As shown in the previous sections, the head is the most relevant individual body location. The sensors at this location are also the most promising with respect to a later implementation into a HI. Figure 9 shows the accuracies for distinguishing the hearing needs based on sound, head movements, eye movements, and all possible combinations. As can be seen from Figure 9, from the three individual modalities, an accuracy of 86% was achieved using eye movements. Moreover, the standard deviation is lower than the one for head movements that yields an accuracy of 84%. From all individual modalities, eye movement analysis performs best. From all combinations

Accuracy [%]

Recognition of Hearing Needs from Body and Eye Movements

100 90 80 70 60 50

77%

Sound

86%

Eye Mov.

327

84%

87%

85%

85%

88%

Head Mov.

Head Mov. Eye Mov.

Sound Head Mov.

Sound Eye Mov.

Head Mov. Eye Mov. Sound

Fig. 9. Accuracies for distinguishing the hearing needs in our scenario based on sound, eye movements, head movements, and all possible combinations. Results are averaged over all participants with the standard deviation indicated with black lines.

of two modalities (bars 4–6 in Figure 9), joint analysis of head and eye movements perform with 87%. The combination of all three modalities yields the highest accuracy of 88%. Adaption based on eye movements (86%) outperforms adaption based on head movements (84%). As described in section 5, eye movement analysis requires a three times larger data window size (10 seconds) than body movement analysis (3 seconds), leading to a larger classification latency. The joint analysis of body and eye movements combines the more long-term eye movement analysis, and more short-term body movements and yields an accuracy of (85%). Taking into account movement from all body location corresponds to the idea of leveraging the HIBAN described in section 2. Sensing head and eye movements corresponds to the idea to eventually integrate all sensors into the HI. The HIBAN approach leads to higher than the stand-alone approach at the cost of additional locations on the body that have to be attached with a sensor. The two cases represent a trade-off between accuracy and required number of body locations attached with sensors. Hearing impaired can decide to take the burden of wearing additional sensors to benefit from better hearing comfort. Besides, smartphone and on-body sensors are more and more likely to be available. As shown, the system functions stand-alone with reduced performance. 6.4

Individual Results for Each Participant

To further investigate the large standard deviation for head movements we additionally analysed the individual recognition performance for each participant. Figure 10 shows the accuracy of choosing the correct program for adaption based on sound, head movements, eye movements, and their fusion on feature level for each individual participant. This analsyis reveals that for four participants eye movements performed best, for the remaining 7 participants head movements performed best. Eye movements provide more consistent high performances for all participants between 82% and 91%. Results for head movements were less consistent. In particular participant 9 and 10 showed reduced accuracies of 61% and 74%. A possible reason for this can be a displaced sensor, e.g. caused by the user adjusting the cap. For eye movements the variability is smaller in the given

Accuracy [%]

328

B. Tessendorf et al.

Sound Eye Mov. Head Mov. Head Mov., Eye Mov., Sound

100 90 80 70 60 50

1

2

3

4

5

6 7 Participant #

8

9

10

11

Fig. 10. Accuracy for adaption based on sound, head movements, eye movements and their combination, individually for each participant

data set. Sound adaption compares worse to body and eye movement adaption since this scenario intentionally contains acoustically ambiguous hearing needs. 6.5

Integrating Body and Eye Movement Sensing into a HI

The integration of additional sensor modalities is within reach of future HIs driven by the trend of HIBANs. HIs may in some cases be one of many deployed ambient assisted living technologies. Thus, wearable and textile integrated sensors, as well as the user’s smart-phone may become part of the HIBAN. Future HIs can also take advantage of additional sensors that are already deployed for other purposes (e.g. motion sensing for fall detection). This reduces the user burden of utilizing multiple sensors while improving his auditive comfort. Whenever HiBANs are not available, sensors could also be completely integrated into the HI itself to provide a stand-alone solution. Low power accelerometers with small footprints are available for integration into a HI. EOG is an inexpensive method for mobile eye movement recording. These characteristics are crucial for future integration of long-term eye movement data into future HIs in mobile daily life settings. EOG integration into a HI could follow integration achievements of EOG into glasses [7] or headphones [16]. 6.6

Limitations

Although we significantly enhanced the distinction of two ambiguous auditory situations, our multimodal context recognition approach remains a proxy to infer what is essentially a subjective matter: the subjective hearing need of a person. Thus, even a perfect context recognition would not guarantee that the hearing need is detected correctly all the time. Ultimately, this would require capturing the user’s auditory selective attention. Our evaluation is based on the recognition accuracy compared to the objective ground truth defined in the scenario. However, to assess the actual benefit experienced by the user, a more thorough user study with hearing impaired will need to be carried out.

Recognition of Hearing Needs from Body and Eye Movements

329

We currently investigated a single ambiguous auditory situation. Nevertheless there are a large number of other ambiguous situations for current hearing instruments. Our objective is to identify the smallest subset of additional sensor modalities which can help to distinguish a wide range of currently challenging auditory situations. Thus, this work in an office scenario is an exemplary proof-of-concept approach. It still needs to be shown that the approach can be generalised and that one can resolve ambiguity in a sufficient number of other situations to justify the inclusion of additional sensor modalities within HIs. The office scenario we chose may be a limitation. We chose a specific office work situation, but a variety of other office situations are thinkable, e.g. with more conversation partners and different activities. For this proof-of-concept study it was necessary to choose a trade-off between variety and control to collect data in a reproducible manner for multiple participants. After the experiment we went through a short questionnaire with each participant. The general feedback was, that the sensor equipment was found to be bulky, but overall the participants felt that they were not hindered to act natural. Overlaying background noise as described in section 5.4 may be a limitation. We overlaid one typical office background noise. Many different kinds and intensities are thinkable. In some cases, the performance of the sound-based HI might be better. However, the performance based on body and eye movement is independent of the present sound. As a further potential limitation the participants may not act the same as they would if there is actual background noise. 6.7

Considerations for Future Work

There are a large number of other challenging situations that are faced by current HIs, e.g. listening to music from the car radio while driving, reading a book in a busy train, or conversing in a cafe with background music. This motivates the investigation of additional modalities, acoustic environments, and hearing situations in future work. A critical issue will be the trade-off in improving context awareness in HIs while minimising the burden caused by additional sensors. Possible additional sensor modalities are the user’s current location, proximity information, or information from other HIs or the environment. Based on the promising results achieved in our proof-of-concept study, we plan to deploy our system in further real-life outdoor scenarios to study the benefit in everyday life experienced by the user.

7

Conclusion

Hearing instruments have emerged as true pervasive computers and are fully integrated into their user’s daily life. In this work we have shown that multimodal fusion of information derived from body and eye movements is a promising approach to distinguish acoustic environments that are challenging for current hearing instruments. These results are particularly appealing as both modalities can potentially be miniaturised and integrated into future HIs.

330

B. Tessendorf et al.

Acknowledgments. This work was part funded by CTI project 10698.1 PFLSLS ”Context Recognition for Hearing Instruments Using Additional Sensor Modalities”. The authors gratefully thank all participants of the experiment and the reviewers for their valuable comments.

References 1. Atallah, L., Aziz, O., Lo, B., Yang, G.Z.: Detecting walking gait impairment with an ear-worn sensor. In: International Workshop on Wearable and Implantable Body Sensor Networks, pp. 175–180 (2009) 2. Bannach, D., Amft, O., Lukowicz, P.: Rapid prototyping of activity recognition applications. IEEE Pervasive Computing 7, 22–31 (2008) 3. Biggins, A.: Benefits of wireless technology. Hearing Review (2009) 4. B¨ uchler, M., Allegro, S., Launer, S., Dillier, N.: Sound Classification in Hearing Aids Inspired by Auditory Scene Analysis. EURASIP Journal on Applied Signal Processing 18, 2991–3002 (2005) 5. Bulling, A., Roggen, D., Tr¨ oster, G.: Wearable EOG goggles: Seamless sensing and context-awareness in everyday environments. Journal of Ambient Intelligence and Smart Environments 1(2), 157–171 (2009) 6. Bulling, A., Ward, J.A., Gellersen, H.: Multi-Modal Recognition of Reading Activity in Transit Using Body-Worn Sensors. ACM Transactions on Applied Perception (to appear, 2011) 7. Bulling, A., Ward, J.A., Gellersen, H., Tr¨oster, G.: Eye Movement Analysis for Activity Recognition Using Electrooculography. IEEE Transactions on Pattern Analysis and Machine Intelligence 33(4), 741–753 (2011) 8. Choudhury, T., Pentland, A.: Sensing and modeling human networks using the sociometer. In: ISWC, p. 216. IEEE Computer Society, Washington, DC, USA (2003) 9. Hadar, U., Steiner, T.J., Clifford Rose, F.: Head movement during listening turns in conversation. Journal of Nonverbal Behavior 9(4), 214–228 (1985) 10. Hamacher, V., Chalupper, J., Eggers, J., Fischer, E., Kornagel, U., Puder, H., Rass, U.: Signal processing in high-end hearing aids: State of the art, challenges, and future trends. EURASIP Journal on Applied Signal Processing 18(2005), 2915– 2929 (2005) 11. Hart, J., Onceanu, D., Sohn, C., Wightman, D., Vertegaal, R.: The attentive hearing aid: Eye selection of auditory sources for hearing impaired users. In: Gross, T., Gulliksen, J., Kotz´e, P., Oestreicher, L., Palanque, P., Prates, R.O., Winckler, M. (eds.) INTERACT 2009. LNCS, vol. 5726, pp. 19–35. Springer, Heidelberg (2009) 12. Keidser, G.: Many factors are involved in optimizing environmentally adaptive hearing aids. The Hearing Journal 62(1), 26 (2009) 13. Kochkin, S.: MarkeTrak VIII: 25-year trends in the hearing health market. Hearing Review 16(11), 12–31 (2009) 14. Kochkin, S.: MarkeTrak VIII: Consumer satisfaction with hearing aids is slowly increasing. The Hearing Journal 63(1), 19 (2010) 15. Lin, C.J.: LIBLINEAR - a library for large linear classification (February 2008) http://www.csientuedutw/~ cjlin/liblinear/ 16. Manabe, H., Fukumoto, M.: Full-time wearable headphone-type gaze detector. In: Ext. Abstracts of the SIGCHI Conference on Human Factors in Computing Systems, pp. 1073–1078. ACM Press, New York (2006)

Recognition of Hearing Needs from Body and Eye Movements

331

17. Molen, M., Somsen, R., Jennings, J.: Does the heart know what the ears hear? A heart rate analysis of auditory selective attention. Psychophysiology (1996) 18. Morency, L.P., Sidner, C., Lee, C., Darrell, T.: Contextual recognition of head gestures. In: ICMI 2005: Proceedings of the 7th International Conference on Multimodal Interfaces, pp. 18–24. ACM, New York (2005) 19. Naylor, G.: Modern hearing aids and future development trends, http://www.lifesci.sussex.ac.uk/home/Chris_Darwin/BSMS/ Hearing%20Aids/Naylor.ppt 20. Peng, H., Long, F., Ding, C.: Feature selection based on mutual information: criteria of max-dependency, max-relevance, and min-redundancy. IEEE Transactions on Pattern Analysis and Machine Intelligence 27(8) (2005) 21. Schaub, A.: Digital Hearing Aids. Thieme Medical Pub. (2008) 22. Shargorodsky, J., Curhan, S., Curhan, G., Eavey, R.: Change in Prevalence of Hearing Loss in US Adolescents. JAMA 304(7), 772 (2010) 23. Shinn-Cunningham, B.: I want to party, but my hearing aids won’t let me? Hearing Journal 62, 10–13 (2009) 24. Shinn-Cunningham, B., Best, V.: Selective attention in normal and impaired hearing. Trends in Amplification 12(4), 283 (2008) 25. Stiefmeier, T., Roggen, D., Ogris, G., Lukowicz, P., Tr¨ oster, G.: Wearable activity tracking in car manufacturing. IEEE Pervasive Computing 7(2), 42–50 (2008) 26. Tessendorf, B., Bulling, A., Roggen, D., Stiefmeier, T., Tr¨ oster, G., Feilner, M., Derleth, P.: Towards multi-modal context recognition for hearing instruments. In: Proc. of the International Symposium on Wearable Computers (ISWC) (2010)

Recognizing Whether Sensors Are on the Same Body Cory Cornelius and David Kotz Department of Computer Science, Dartmouth College, Hanover, NH, USA Institute for Security, Technology, and Society, Dartmouth College, Hanover, NH, USA

Abstract. As personal health sensors become ubiquitous, we also expect them to become interoperable. That is, instead of closed, end-to-end personal health sensing systems, we envision standardized sensors wirelessly communicating their data to a device many people already carry today, the cellphone. In an open personal health sensing system, users will be able to seamlessly pair off-the-shelf sensors with their cellphone and expect the system to just work. However, this ubiquity of sensors creates the potential for users to accidentally wear sensors that are not necessarily paired with their own cellphone. A husband, for example, might mistakenly wear a heart-rate sensor that is actually paired with his wife’s cellphone. As long as the heart-rate sensor is within communication range, the wife’s cellphone will be receiving heart-rate data about her husband, data that is incorrectly entered into her own health record. We provide a method to probabilistically detect this situation. Because accelerometers are relatively cheap and require little power, we imagine that the cellphone and each sensor will have a companion accelerometer embedded with the sensor itself. We extract standard features from these companion accelerometers, and use a pair-wise statistic – coherence, a measurement of how well two signals are related in the frequency domain – to determine how well features correlate for different locations on the body. We then use these feature coherences to train a classifier to recognize whether a pair of sensors – or a sensor and a cellphone – are on the same body. We evaluate our method over a dataset of several individuals walking around with sensors in various positions on their body and experimentally show that our method is capable of achieving an accuracies over 80%.

1 Introduction Mobile sensing of the human body is becoming increasingly pervasive with the advent of personal devices capable of processing and storing large of amounts of data. Commercial devices like the FitBit [7] and BodyBugg [1] allow a person to collect nearly continuous data about his or her health. The FitBit, for example, allows a person to track one’s own fitness and sleeping patterns by wearing an accelerometer on the waist. Typically these devices are highly specialized, end-to-end solutions, but we imagine the sensors in these products becoming commodities and inter-operating with a device most people carry with them everyday: cellphones. A person could wear several sensors of varying types (e.g., blood pressure monitor, pulse oximeter, pedometer, blood glucose meter). Because of the physiological requirements, or comfort, these sensors will K. Lyons, J. Hightower, and E.M. Huang (Eds.): Pervasive 2011, LNCS 6696, pp. 332–349, 2011. c Springer-Verlag Berlin Heidelberg 2011 

Recognizing Whether Sensors Are on the Same Body

333

necessarily be attached at different locations on the body. We imagine these sensors wirelessly communicating with a person’s cellphone, which would store and aggregate all data coming from the sensors. In fact, this scenario is feasible today, and there are purchasable medical and fitness sensors capable of communicating to cellphones via Bluetooth. There are many security issues, not to mention privacy issues, with this scheme. How does the cellphone authenticate valid sensors? How do sensors discover the presence of the cellphone, without exposing their own presence? How does the user pair sensors with the cellphone? What types of encryption are employed to maintain confidentiality and integrity? How do we balance privacy and usability? We focus our attention on one specific challenge: how can we verify that a suite of sensors are attached to the same person? Suppose Alice and Fred, a health-conscious couple living together, each decide to buy a fitness-monitoring sensor. The instructions indicate that each should “pair” their respective sensor with their own cellphone. Pairing ensures, through cryptographic means, that a sensor is only able to communicate with a specific cellphone. One day, when Alice and Fred go for a run, Alice unknowingly wears Fred’s sensor, and Fred wears Alice’s sensor. As they run, thereby remaining in communication range, Fred’s cellphone will be collecting data about Alice, but labeling the data as Fred’s and placing it in Fred’s health record, and vice versa. This problem, a result of the one-to-one pairing model, is even more likely as the number of sensors grows. The implicit assumption when pairing is that the sensors paired with a cellphone will not be used by anyone else but the user of the cellphone. Our goal is to make life easier for people like Alice and Fred. Although Alice and Fred buy identical sensor devices, Alice should be able to strap on either device and have her cellphone recognize which device is attached to her, automatically creating the phone-device association without an explicit pairing step. Similarly, if Alice and Fred jointly own another sensor device, either may use the sensor at any time, and again the correct cellphone should detect which body is wearing the sensor and receive the data into the correct person’s health record. To achieve this vision requires two core problems to be solved. First, Alice’s phone must be able to determine which sensors are attached to Alice’s body, ignoring sensors that may be in radio range but not attached to Alice. Second, the phone and sensor devices must be able to agree on a shared encryption key, to secure their communications; ideally, this should require no user assistance and be more secure than in most “pairing” methods today. In this paper we specifically address the first challenge, leaving the second challenge to future work. There are existing solutions that address the second challenge, but it is unclear if those solutions can be applied for accelerometers that are not intentionally shaken together [13]. To address the first challenge, the sensor device must somehow attest (to the cellphone) which body is wearing the sensor at the current time. Ideally, the phone would analyze the data coming from the sensors to see whether it identifies the wearer by some biometric measure. However, not all types of sensors, or sensor locations, produce data that is suitable for biometric identity verification. Thus we propose the following compromise: every sensor device will include an accelerometer sensor in addition to

334

C. Cornelius and D. Kotz

its primary sensor (ECG, blood pressure, etc.). Accelerometers are cheap, so this is a relatively inexpensive addition; instead of biometric identity verification with a wide variety of sensor data, sensor placement, and usage conditions, we only need to find correlations for the accelerometer data that answers the question: are the devices in a given set all attached to the same body? We recently [6] formalized this problem as the “one-body authentication problem,” which asks: how can one ensure that the wireless sensors in a wireless body area network are collecting data about one individual and not several individuals? We identified two variants of this problem. The strong version of this problem requires identifying which person the sensors are attached to, whereas the weak version of this problem simply requires determining whether the sensors are on the same body. We noted how existing solutions do not necessarily solve the problem and called for further research. Thus, we now aim to provide a solution to the weak one-body authentication problem; given such as solution, one might solve the strong-body problem for one of the sensors in a set, and be able to extrapolate the verification to all of the sensors on the body. Our paper is organized as follows. In the next section we describe our model. In the third section we briefly describe our approach and hypothesis as to why we believe our approach will work. In the fourth section we describe, in detail, our method. In the fifth section we describe the data we collected as well as our collection method. In the sixth section we evaluate our method. In the final sections, we discuss related work and distinguish our work from earlier approaches, and provide some discussion about our method’s limitations and about some potential future work.

2 Model We imagine a world where personal health sensors are ubiquitous and wirelessly connect to a user’s cellphone. Thus, there are two principle components in our system: – One mobile node (e.g., the user’s cellphone) per user. – Many sensor nodes (e.g., blood glucose, pedometer, electrocardiography). We assume that mobile nodes communicate wirelessly with sensor nodes. Sensor nodes are also capable of communicating wirelessly with mobile nodes but have limited computational resources relative to the mobile nodes. Additionally, sensor nodes have the ability to detect when they are attached to a user (although they will not know to whom). The sensor node might contain a circuit that is completed, for example, when the user straps a sensor node onto their body and the two ends of a necklace or wriststrap come into contact. Finally, we also assume each sensor node, and the mobile node, has an accompanying triaxial accelerometer of the same type (so that their readings may be directly compared). Since accelerometers are tiny, cheap, and require little energy to operate, this is a reasonable assumption1 . 1

The Freescale MMA845xQ line of accelerometers, for example, cost $0.95 (in quantities of 100K) and consume “1.8 microamps in standby mode and as low as 6 microamps in active mode” [8].

Recognizing Whether Sensors Are on the Same Body

335

2.1 Binding “Binding” occurs when a user wishes to use a sensor node. The following happens: 1. The user straps the sensor node to their body, thereby turning it on. 2. The sensor node detects that it was applied, and broadcasts its presence. 3. The mobile node receives the broadcast, thereby binding it with the sensor node, and labels that sensor node as unauthenticated. Binding is like pairing, but without the need for user intervention. In a pairing scenario, the user is usually required to enter a shared key on one of the devices. Binding does not have this requirement. When a sensor node is bound to a mobile node, the sensor node enters an unauthenticated state. 2.2 Authentication “Authentication” is a process, initiated by the mobile node, for verifying which of the mobile node’s bound sensor nodes are on the same body. Once a sensor node is authenticated, the mobile node will record sensor data from that node; until then, the data will be ignored. (As it may take some time for authentication to succeed, in some implementations the mobile node may buffer the incoming data received between the moment of binding and the moment of authentication, recording the data only once authentication is assured. This “retroactive authentication” of the early data is feasible because of our assumption that a sensor node can detect its own attachment and removal; if a sensor node is moved from one body to another before it was authenticated on the first body, the unbinding and rebinding events will clear the buffer on the first body’s mobile node). To achieve authentication, our protocol requires an algorithm that is able to decide whether two streams of data are originating from sensor nodes on the same body. That is, given a stream of accelerometer data from a sensor node, the algorithm examines the correlation between a sensor node’s data stream and the mobile node’s data stream, with the requirement that the two streams should correlate well only when both the mobile node and the sensor node are on the same body. The algorithm should return true if and only if the two data streams are well correlated and false otherwise. We present the details of our algorithm in Section 4. Procedure 1 provides an overview of the process for the mobile node to authenticate sensor nodes. Because our method depends on recognizable acceleration events, our algorithm performs authentication only when the user is walking. The mobile node records acceleration data using its internal accelerometer for t seconds. Simultaneously, it asks the other sensor node to send it acceleration data for the same duration. The duration required depends on the level of confidence desired; a shorter duration may lead to more incorrect results (false positives and false negatives), but a longer duration makes the approach less responsive after the person first puts on the sensor. It then runs our algorithm, called AreCorrelated, to determine whether its internal acceleration data correlates with the sensor node’s acceleration data. Only when the accelerometer data correlates well does the mobile node begin to record that sensor node’s other sensor data (e.g., electrocardiography data).

336

C. Cornelius and D. Kotz

Procedure 1. Authenticating sensor nodes, from the mobile node’s perspective

1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: 15: 16: 17: 18: 19: 20: 21: 22: 23: 24:

Notation: B: set of bound sensor nodes, initially empty Ai : acceleration data from sensor node i, where i = 0 is the mobile node’s acceleration data. Record(t): read mobile node’s accelerometer for t seconds Recv(b, t): read sensor node b’s accelerometer for t seconds AreCorrelated(x, y): determine whether acceleration data x and y ————————————————– while { true } do if b := NewSensorNodeDetected() then B := B ∪ b { Mark sensor node b as unauthenticated } end if for b ∈ B do if Disconnected(b) or Timeout(b) then B := B \ b else if d := RecvData(b) and IsAuthenticated(b) then RecordData(b, d) { Save b’s data d in our health record } end if end for if UserIsWalking() then for b | b ∈ B and not IsAuthenticated(b) do { The next two lines are accomplished in parallel } A0 := Record(t) Ab := Recv(b, t) if AreCorrelated(A0 , Ab ) = true then { Mark sensor node b as authenticated } { Tell sensor node b to send sensor data } end if end for end if end while

2.3 Unbinding Unbinding occurs when a user removes a sensor node. In the ideal case, the following happens: 1. The user unstraps the sensor node from their body. 2. The sensor node detects that it was removed and notifies the bound mobile node of this fact. 3. The mobile node acknowledges this notification, thereby unbinding it with the sensor node. 4. Upon receipt of this acknowledgement (or upon timeout), the sensor node turns off. A sensor node may lose power or go out of range of the mobile node, during this process or prior to the user unstrapping the sensor node. Thus, the mobile node periodically pings each sensor node (not shown in Procedure 1); if the sensor node does not

Recognizing Whether Sensors Are on the Same Body

337

reply (after some timeout period), the sensor node is likely not on the same body, and the mobile node treats it as unauthenticated and unbound.

3 Approach Our goal is to determine whether a sensor node is on the same body as a mobile node receiving the sensor node’s data. That is, we provide a solution for the weak one-body authentication problem. Our solution could be used as the first step in a strong one-body authentication solution by first verifying that all the sensors are on the same body, then using some subset of the sensors to provide strong one-body authentication (i.e., via some biometric one of the sensors could determine) to all the sensors on the body. To maximize the generality of our solution, we require each sensor to have an accompanying accelerometer. Our intuition is that if sensors are on the same body, then (at a coarse level) all of the sensors’ accelerometers experience similar accelerations. If a user is seated, or lying down, then there is not much information we can extract from the accelerometer data to make the determination that a suite of sensors are on the same body. There are a variety of activities that cause bodily acceleration, but we focus on walking. When walking, a human body is largely rigid in the vertical direction. Although our limbs do bend, we hypothesize that the vertical acceleration (i.e., the acceleration relative to gravity) experienced by sensors placed anywhere on a walking body should correlate well. As one foot falls, that side of the body experiences a downward acceleration due to gravity, followed by an abrupt deceleration when the foot contacts the ground. Sensors on one side of the body should experience a similar vertical acceleration, while sensors on the other side of the body will experience the opposite. We should expect positive correlation for one side of the body, and an inverse correlation on the other side. Of course, this observation is complicated by the fact that it is difficult to extract the vertical acceleration component without knowing the orientation of the sensor. Furthermore, although the signal can be very noisy, the accelerations due to walking are likely to dominate the accelerations due to intra-body motion (such as arm swings or head turns) and we should be able to reliably make a determination that the supposed suite of sensors are on the same body. Fortunately, there is already an existing body of work that shows how to do activity recognition given user-annotated data [2], and even on a mobile phone class device [4]; these techniques are particularly good at detecting when a user is walking. Our approach, therefore, is to detect periods when a user is walking by monitoring the accelerometer data periodically; when the data indicates the user is walking, we use Procedure 1 to collect accelerometer data from the sensors. (In Section 8 we discuss users who cannot walk). Lester et al. [11] provide a solution the one-body authentication problem, but only for sensors that are carried in the same location on the body. They also propose using accelerometers attached to each sensor and measure the coherence of the accelerometer data. “Coherence measures the extent to which two signals are linearly related at each frequency, with 1 indicating that two signals are highly correlated at a given frequency and 0 indicating that two signals are uncorrelated at that frequency” [11]. By looking

338

C. Cornelius and D. Kotz

Waist

Right hand

Left hand

Right leg

Left leg

Fig. 1. Five seconds of magnitude data for each position on the body for one user

at the coherence at the 1-10Hz frequencies (the frequency range of human motion), they can experimentally determine a threshold (e.g., coherence > 0.9) at which it is appropriate to deem two sensors as located on the same body. We extend Lester et al. [11] to sensors carried at different locations on the body – wrist, ankle, and waist – by using features often used for activity recognition. We then extract the pairwise coherence of features for the sensors on the same body. Given these coherences, we can train a classifier and use it to determine whether the alleged set of sensors are on the same body. We train our classifier to be as general as possible by using data collected from several individuals; the same model can then be used by all users for all sensor devices. We describe our method in more detail in the following section.

4 Method As stated previously, we assume each sensor node has an accompanying accelerometer; our method uses only the accelerometer data. Specifically, consider a signal s sampled at some frequency such that: s = {(x0 , y0 , z0 ), (x1 , y1 , z1 ), . . .} where xi , yi , and zi are the three axes of the instantaneous acceleration, relative to gravity, at time i. Because sensors might be mounted in different orientations, or might be worn in different orientations each time they are worn, we discount orientation by using the magnitude of the acceleration. Figure 3 shows that the magnitude exposes the overall walking motion well. Thus, we compute the magnitude of all three axes for all samples in s:  mi = x2i + yi2 + zi2 This gives us the rate of change of speed over time for that particular sensor node.

Recognizing Whether Sensors Are on the Same Body

339

4.1 Feature Computation We partition this orientation-ignored signal {m0 , . . . , } into non-overlapping windows of length w. For each window j, comprising {mjw , . . . , mjw+w }, we extract seven common features (mean, standard deviation, variance, mean absolute deviation, interquartile range, power, energy); collectively, these seven values form the feature vector Fj = (fj1 , fj2 , . . . , fj7 ). We chose these features primarily because others [12, 14] have used these features successfully to detect physical activities, and we hypothesize they would similarly be useful for our problem. If they can capture the physical activity of walking and we examine the correlation of these features, we should expect them to correlate if and only if they are attached the same body. 4.2 Coherence Coherence is a measure of how well two signals correlate in the frequency domain. More precisely, it is the cross-spectral density of two signals divided by the autospectral density of each individual signal. Like Lester et al. [11], we approximate coherence by using the magnitude-squared coherence: Cxy (φ) =

|Sxy (φ)|2 Sxx (φ)Syy (φ)

In the above, x and y are the signals, Sxy is the cross-spectral density between signals x and y, Sxx is the auto-spectral density of signal x, and φ is the desired frequency. Cross-spectral density is calculated by the Fourier transform of the cross-correlation function. If x and y are well correlated at some frequency φ, then Cxy (φ) should be close to 1. To get a final measure, we compute the normalized magnitude-squared coherence up to some frequency φmax : N (x, y) =

1 φmax

 0

φmax

Cxy (φ)dφ

We chose φmax = 10 because, as Lester et al. notes, “human motion rests below the 10Hz range” [11]. In addition, to compute the cross-spectral density over different frequencies, it is necessary to window the signals x and y. We choose a Hamming window of length equal to one-half of the size of the signals with no overlap. 4.3 Feature Coherence Given two sets of feature matrices A = (F1 , F2 , . . .) and B = (F1 , F2 , . . .) with entries Fj as described above, we want to determine how well A and B are correlated. Here, A and B represent the feature matrices extracted from the accelerometer data of the mobile node and sensor node respectively.

340

C. Cornelius and D. Kotz

We apply coherence to the feature matrices in the following manner. For some window length c (the feature coherence window), we compute the normalized coherence of A and B as such:   1 2 7 ), N (A2k...k+c , Bk...k+c ), . . . , N (A7k...k+c , Bk...k+c ) NkAB = N (A1k...k+c , Bk...k+c   where A1k...k+c = fn1 ∈ A : k ≤ n < k + c , the window of a specific feature of A. That is, we take each feature (i.e., a column of the matrix) of A and the corresponding feature of B, and compute the normalized coherence using c samples (i.e., the rows of the matrix). At this stage, we are left with a matrix of normalized coherences for each feature and window k. Because we want to capture how the two signals are related over time, the coherence window c should be sufficiently large to capture periodicities in the features. Because the typical walk cycle is on the order of seconds, it is advisable to chose a coherence window on the order of several seconds. 4.4 Supervised Learning and Classification To account for the many positions a sensor node might be placed on the body, we collect data from several locations. In our method, we compare the mobile node’s accelerometer data to each other sensor node’s accelerometer data. That is, the mobile node acts as a reference accelerometer, to which every other sensor node must correlate using the method described above. For a given set of locations and one reference location, we compute the feature coherence of each location (i.e., A in the above) relative to the reference location (i.e., B in the above). In our experiments, we compute the coherence of the right wrist and waist; left wrist and waist; left ankle and waist; and right ankle and waist. When we do this for one user, this yields feature coherences of the sensor on the same body, and we can label them as such. To yield feature coherences of sensors on different bodies, we take pairs of users and mix their locations. For example, at the waist and left hand there are two possible ways to mix up the sensors: Alice’s waist and Fred’s left hand, Fred’s waist and Alice’s left hand. By mixing locations for any pair of users, it is possible to compute an equal number of feature coherences that are and are not on the same body, labeling them as such. Given a set of feature coherences and their respective labels, we can train a classifier to learn a model that is the coherence threshold for each feature. We employ support vector machines (SVMs) for this task since, once trained, they are good at predicting which label a given feature coherence is associated with. An SVM accomplishes this task by finding the hyperplane with the largest separation between the set of training feature coherences that are on the same body, and those that are not on the same body. In our experiments, we trained a support vector machine with a radial basis kernel using LIBSVM [5]. Given a trained SVM, we can use it to classify whether a given feature coherence is on the same body. That is, at the window the feature coherence was computed, the support vector machine can determine if the sensor node is on the same body as the mobile node. The SVM does so by determining on which side of the hyperplane the test feature coherence lies.

Recognizing Whether Sensors Are on the Same Body

341

User Walking Time (minutes:seconds) Magnitude Samples Feature Vectors 1 18:45 288017 9000 2 29:57 460047 14375 3 21:02 322962 10092 4 19:30 299553 9361 5 20:24 313215 9787 6 28:33 438484 13701 7 19:01 291974 9123 Fig. 2. Time spent walking, total acceleration samples, and number of features extracted for each user

4.5 Classification Smoothing The classification method described above makes an instantaneous classification of a feature coherence for that particular coherence window. It is, however, possible to boost the classification rates by examining a window of classifications over time. For example, if over the course of three classifications, two classifications positive and the third classification is negative, we can use a simple voting scheme to smooth over these misclassifications. In the example, because the majority of the classifications are classified as on the same body, we assume the sensor node is on the same body for that classification window. We can empirically determine the best window by varying the window and choosing the one that yields the best classification rates.

5 Dataset We collected a set of accelerometer data, from several test subjects wearing sensors in several locations on their body, to use as training data (for the model) and to use as test data for (for our evaluation). We used WiTilt (version 2.5) accelerometers [15]. We followed the user with a laptop as they walked around a flat, predetermined course. The laptop was used to synchronize the accelerometer readings sent via Bluetooth by the WiTilt nodes. We collected 2.5 hours of acceleration from 5 accelerometers sampled at 255Hz from seven users for a total of 13 hours of acceleration data. The average user walked for 22 minutes while wearing 5 accelerometers (waist, left wrist, right wrist, left ankle, right ankle). We chose the waist (specifically, the right pocket), because it represents a common location for the mobile node (cellphone). Of the likely locations for medical sensors (arms, legs, chest, head) we chose the wrists and ankles for our experiments because (as extremities) we expect they would raise the most difficult challenge for our method. Figure 2 gives more detailed information about how much data was collected for each user.

6 Evaluation We evaluate how well our method performed for each location, at the wrists only, at the ankles only, on the left side of the body, on the right side of the body, and at all locations.

342

C. Cornelius and D. Kotz

For each experiment we used only the data from that location, or type of location, for training and for evaluation; for example, in the “left leg” case we train on (and test on) the accelerometer data from the left ankle in comparison to the data from the waist. In neither the learning process nor in the operation of our system was the data labeled as to which location produced the acceleration data. We varied the coherence window size from 2 to 16 seconds. Using these datasets, we performed two types of cross-validations to evaluate the accuracy of our method. The first cross-validation we performed was a simple 10-fold cross-validation. A k-fold cross-validation partitions the dataset into k partitions, trains the classifier over k − 1 of the partitions (the training set) and classifies the remaining partition (the testing set), repeating this procedure for each partition. This type of crossvalidation will tell us how well our classifier generally performs since it will classify every sample in the dataset. The second cross-validation we performed is a variant of leave-one-out cross-validation we call leave-one-user-out cross-validation. A leaveone-user-out cross-validation leaves an entire user’s data out as the testing set and trains the classifier using the remaining data. We then test the classifier using the left-out user’s data, repeating this procedure for each user. This type of cross-validation will tell us how general our classifier is. Ideally our classifier would not be user-specific, and would perform well in the case of a never-before-seen user. We define a true feature coherence as a feature coherence computed from a sensor node and mobile node on the same body, and a false feature coherence as a feature coherence computed from a sensor node and mobile node not on the same body. A positive classification means the classifier determined that the given feature coherence indicates the sensor node and mobile node were on the same body, while a negative classification means the classifier determined that the given feature coherence indicates the sensor node and mobile node were not be on the same body. It follows, then, that a true positive occurs when a true feature coherence is classified as positive, and a true negative occurs when a false feature coherence is classified as a negative. A false positive occurs when a false feature coherence is classified as positive, and a false negative occurs when a true feature coherence is classified as negative. We present the accuracy, precision and recall for each possible scenario. Accuracy is the sum of true positives and true negatives over the total number of classifications. Accuracy tells us how well our classifier is doing at classifying feature coherences. Precision is the number of true positives over the total number of positive classifications. Precision tells us how well our classifier is able to discriminate between true and false positives. Recall is the number of true positives over the sum of true positives and false negatives. Recall tells us how well our classifier classifies true features coherences. In all of our experiments, we chose a feature window size of 17 acceleration magnitudes with no overlap so that each second may be divided evenly and thus yield 15 features per second. We present results using our dataset for both our method and the method used in Lester et al. [11] for sake of comparison. 6.1 Our Method We ran a 10-fold cross-validation using the data from all users and for each specified location, resulting in Figures 3(a), 3(b), and 3(c). The results show how the choice of

Recognizing Whether Sensors Are on the Same Body

343

coherence window size affects the accuracy, precision and recall. A smaller window is more desirable because the coherence window size is directly proportional to the window of accelerometer data that needs to be transmitted to the mobile node, and wireless communication is typically expensive. However, a smaller window will not capture the periodicity of walking. According to Figure 3(a), a 4–6 second coherence window, or about 60–90 feature values, performed the best and minimized the communication overhead. In such cases our method was about 70–85% accurate. In general, as the coherence window length increases the accuracy briefly climbs then settles down, precision increases steadily, and recall drops significantly. Given a longer coherence window length, this means the classifier is more likely to make negative classifications rather than positive ones. Since a longer coherence window means more walking cycles are taken into account, it also means there is more opportunity for the signals to differ due to accumulated noise and/or a change in walking style in accordance with the environment. These plots show that the method was more accurate for the legs than for the hands, which is not surprising because the legs have more consistent motion behavior during walking than do the hands, particularly across users. The right leg (or left hand) seemed to do better than the left leg (or right hand, respectively), perhaps because the waist accelerometer was always carried in the right pocket, and most people swing their hands in opposition to their legs. When the hands and legs were combined, as in the left-body and right-body cases, this effect was cancelled out and the results of both were fairly similar to the all-body case. In Figure 3(d), we ran a leave-one-user-out cross-validation for each user with a fixed coherence window of 6 seconds. The accuracy, precision, and recall for all users are nearly identical, thus providing some evidence that our trained model is not specific to any user, and can in fact be used to predict a never-before-seen user. 6.2 Lester et al. Method For comparison’s sake, we implemented the method described in Lester et al. [11], after extending it to use a support vector machine for determining the threshold instead of choosing an arbitrary threshold. Figure 4 shows that for any of the given locations, their method has poor classification rates, little better than random guess (0.50). Lester et al. [11] do present results for “devices at other locations on the body, including accelerometers on the wrist, placed in one or both pockets, in a backpack, and in a fanny pack.” These placements, however, are in the same relative location and therefore not comparable. Furthermore, we evaluated the scheme over longer time intervals, and averaged the results for a specified window. 6.3 Classification Smoothing We now return to the leave-one-user-out experiments, as they most closely model how the method would be used in practice. In these experiments, for each user left out (the testing set), we used the model trained on all other users’ data to predict the testing set. Now, instead of instantaneous prediction, we use a simple majority vote to smooth over classifications and plot how well this smoothing performed for a given window size of classifications.

344

C. Cornelius and D. Kotz

10-fold Cross-validation

10-fold Cross-validation Sensor locations Right leg Legs Left leg Left body All Right body Left hand Hands Right hand

Accuracy

0.8 0.7 0.6 0.5 0.4

0.9

Sensor locations Right leg Legs Left leg Left body All Right body Left hand Hands Right hand

0.8 Precision

0.9

0.7 0.6 0.5 0.4

2

4

6

8

10

12

14

16

2

4

6

Coherence window length (sec)

(a) Accuracy

10

12

14

16

(b) Precision

10-fold Cross-validation

Leave-one-user-out Cross-validation

0.9

Sensor locations Right leg Legs Left leg Left body All Right body Left hand Hands Right hand

0.8

Recall

8

Coherence window length (sec)

0.7 0.6 0.5 0.4

0.9

Accuracy Precision Recall

0.8 0.7 0.6 0.5 0.4

2

4

6

8

10

12

Coherence window length (sec)

(c) Recall

14

16

1

2

3

4

5

6

7 Average

User left out

(d) Leave-one-user-out

Fig. 3. Evaluation of our method. Subfigures (a), (b), and (c) were computed from a 10-fold cross-validation of all users at the specified locations and coherence windows. Subfigure (d) was computed from a leave-one-user-out cross-validation for each user with a coherence window of 6 seconds.

Figure 5 shows the average accuracy, precision, and recall over all users for varying classification windows with a fixed coherence window of 6 seconds. Our method, Figure 5(a), benefits slightly from classification smoothing as does Lester et al.’s method, Figure 5(b). This result tells us that our method makes sporadic mis-classifications that can be reduced with smoothing. Like any smoothing scheme, one must strike a balance between the size of a smoothing window and the desired classification rates. For our method, a 42 second smoothing window, or 7 feature coherences, modestly boosts our instantaneous classification rates by 8%.

7 Related Work Mayrhofer et al. [13] provide a solution to exchange a cryptographic key between two devices by manually shaking the two devices together. They use the method described in Lester et al. [11] to determine whether two devices are being shaken together. But, as they notice, coherence “does not lend itself to directly creating cryptographic key material out of its results” [13]. To extract key material they extract quantized FFT coefficients from the accelerometer data to use as entropy for generating a key. Our problem is more difficult because the accelerometers are not being shaken together but are attached to a body and will therefore experience less-correlated accelerations.

Recognizing Whether Sensors Are on the Same Body

10-fold Cross-validation

10-fold Cross-validation Sensor locations Right leg Legs Left leg Left body All Right body Left hand Hands Right hand

Accuracy

0.8 0.7 0.6 0.5

0.9

Sensor locations Right leg Legs Left leg Left body All Right body Left hand Hands Right hand

0.8 0.7

Precision

0.9

0.6 0.5

0.4

0.4 2

4

6

8

10

12

14

16

2

4

6

Coherence window length (sec)

8

10

12

14

16

Coherence window length (sec)

(a) Accuracy

(b) Precision

10-fold Cross-validation

Leave-one-user-out Cross-validation

0.9

Sensor locations Right leg Legs Left leg Left body All Right body Left hand Hands Right hand

0.8

Recall

345

0.7 0.6 0.5 0.4

0.9

Accuracy Precision Recall

0.8 0.7 0.6 0.5 0.4

2

4

6

8

10

12

14

16

1

2

3

Coherence window length (sec)

4

5

6

7 Average

User left out

(c) Recall

(d) Leave-one-user-out

Fig. 4. Evaluation of Lester et al. method. Subfigures (a), (b), and (c) were computed from a 10-fold cross-validation over all users at the specified locations and coherence window lengths. Subfigure (d) was computed from a leave-one-user-out cross-validation for each user with a coherence window of 6 seconds. Smoothed leave-one-user-out cross-validation

Smoothed leave-one-user-out cross-validation

0.9

0.9

0.8

0.8

0.7

0.7

Average Precision Average Accuracy Average Recall

0.6

Average Precision Average Accuracy Average Recall

0.6

0.5

0.5

0.4

0.4 6

18

30

42

54

Classification window (sec)

(a) Our method

66

78

90

6

18

30

42

54

66

78

90

Classification window (sec)

(b) Lester et al. method

Fig. 5. Average accuracy, precision and recall over all users for different classification windows with a fixed coherence window of 6 seconds

Kunze et al. [10] provide a method for determining where on a body a particular sensor is located. They detect when a user is walking regardless of the location of a sensor, and by training a classifiers on a variety of features (RMS, frequency range power, frequency entropy, and the sum of the power of detail signals at different levels) on different positions on the body they can use the classifier to determine where on the body the sensor is located. We seek to provide a method that determines whether a suite of sensors is located on the same body without having to use multiple classifiers for

346

C. Cornelius and D. Kotz

different body locations. It might be the case that knowing the location of a sensor node could boost our classification rates, but we leave that for future work. Kunze et al. [9] provide similar methods to account for sensor displacement on a particular body part. This problem is difficult primarily because “acceleration due to rotation is sensitive to sensor displacement within a single body part” [9]. To alleviate this problem, the authors observe that “combining a gyroscope with an accelerometer and having the accelerometer ignore all signal frames dominated by rotation can remove placement sensitivity while retaining most of the relevant information” [9]. We choose to limit our approach to accelerometers; although the inclusion of a gyroscope might increase accuracy, it would also increase the size, cost, and energy consumption on each sensor device. Sriram et al. [16] provide a method to authenticate patients using electrocardiography and acceleration data for remote health monitoring. While electrocardiography has proven to be useful for authentication, they observe that these methods do not perform well in the real world because physical activity perturbs the electrocardiography data. By employing an accelerometer to differentiate physical activities, they can use electrocardiography data from those physical activities to authenticate patients. We both make the observation that “the monitoring system needs to make sure that the data is coming from the right person before any medical or financial decisions are made based on the data” [16] (emphasis ours). Our work is complementary since it is necessary to establish that accelerometer is on the same body as the sensor used to collect electrocardiography data. Their method extracts 50 features from the electrocardiography and accelerometer data and uses these features to train two types of classifiers, k-Nearest Neighbor and a Bayesian Network, whose output can be used for identification and verification. We follow a similar procedure except that we work exclusively with accelerometer data, again, to reduce the complexity and cost of the solution. We also look at the correlation between sensors, whereas they assume there is a prior profile of the patient’s combined electrocardiography and accelerometer data.

8 Discussion and Future Work There are a variety of existing technologies one could imagine using to solve the weak one-body authentication problem. For example, one could employ a wireless localization technique to ensure the sensors nodes are within some bodily distance. The body, however, might block all or some of the wireless signal thereby limiting localization, nor is it clear how these kind of techniques would provide confidence to a physician that the data is coming from one body. Similarly, one can trivially use a form of body-coupled communication [3], but the security properties these type of communication mediums provide are not well understood. If two users were to hold hands, for example, would they be considered one body? When two people are walking together, it is a common natural phenomenon for two walkers to synchronize their walking patterns. It is unclear whether our method will be fooled by such a situation, mis-classifying Alice’s and Fred’s sensor devices as being on the wrong body. The first dataset we captured to test this method actually employed one user trying to mimic the gait of another user, and our first results showed our algorithm not being fooled by this. This case, however, requires exploration in a larger dataset.

Recognizing Whether Sensors Are on the Same Body

347

Our method relies on the assumption that a user is capable of walking, which may not be true for some users. It remains as future work to determine whether we can extend the method for a person who is confined to a wheelchair, for example. Even for a user who is able to walk, there may be an extended period of time after binding a sensor node and before the user walks. It may be necessary for the mobile node to alert the user that they should walk around so that authentication can be performed. As future work, we may explore other acceleration events; for example, to ask the user for clap their hands, or perform some unique movement. Ideally the algorithm should be tuned to produce more false negatives (i.e., the algorithm determined the sensor nodes to be on different bodies when they really were on the same body) than false positives (i.e., the algorithm determined the sensor nodes to be on the same body when they were actually not) because the consequences of a false positive (recording the wrong person’s data in someone’s health record) are more severe than the consequences of a false negative (losing data). It may be possible to ‘bias’ the SVM toward false negatives by adding a margin to its hyperplane-testing function. Although we do not discuss encryption mechanisms, ensuring data confidentiality is paramount in any health-related scenario. If one were to optimize the authentication phase by simultaneously authenticating all bound sensor nodes, it might be necessary to encrypt the acceleration data to avoid replay attacks (in which the adversary replays one node’s acceleration data in hopes that its rogue sensor node will be authenticated as being on the same body as the victim). Even if such an attack is discounted, the accelerometer data itself might be privacy sensitive because accelerometer data may be used to recognize a victim’s activity. Some activities are clearly privacy sensitive, and some of those sensitive activities might be detected from accelerometer data alone. In a practical system, one must consider energy and computational costs. In our model, the sensor node sends raw acceleration data to the mobile node. If this proves to be too expensive, then the sensor node could compute features from a window of acceleration and communicate those features instead. We leave exploring this delicate balance between extendability (allowing use of other features in the future), computability (due to limited computational capabilities on a sensor node), and energy requirements (with trade-offs specific to the technology in a sensor node) as future work. In terms of the mobile node, we assume the cellphone will be more than capable of computing correlations, but the energy costs of these functions is unknown and require more careful analysis. Should the computation prove to be too expensive or time consuming, then one may need to explore optimizations or approximations or the assistance of a backend server, with due consideration to the trade-off of computational overhead, accuracy, and privacy.

9 Conclusion Mobile health will play a major role in the future of healthcare. Wearable health sensors will enable physicians to monitor their patients remotely, and allow patients better access to information about their health. The method presented in this paper provides the foundation for any mobile-health system, because, in order for the data to be useful, physicians need confidence that the data supposedly collected about a patient actually

348

C. Cornelius and D. Kotz

came from that patient. We provide the first step in that verification process: generically authenticating that all the sensor nodes bound to a mobile node are the same body. We show that our method can achieve an accuracy of 80% when given 18 seconds of accelerometer data from different locations on the body, and our method can be generically applied regardless of the sensor type and without user-specific training data. In summary, we make the following contributions: – We describe a novel problem in the mobile healthcare domain and provide a solution to the weak version of the one-body authentication problem. – We extend Lester et al. [11] to sensors carried at different locations on the body – wrist, ankle, and waist – by extracting used for activity recognition. – We provide empirical results to our solution using a dataset of seven users walking for 22 minutes to show that it is feasible.

Acknowledgements This research results from a research program at the Institute for Security, Technology, and Society at Dartmouth College, supported by the National Science Foundation under Grant Award Number 0910842 and by the Department of Health and Human Services (SHARP program) under award number 90TR0003-01. The views and conclusions contained in this document are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of the sponsors. We also thank the anonymous reviewers, and our colleagues in the Dartmouth TISH group, for their valuable feedback.

References [1] Apex Fitness. BodyBugg (October 2010), http://www.bodybugg.com/ [2] Bao, L., Intille, S.S.: Activity Recognition from User-Annotated Acceleration Data. In: Ferscha, A., Mattern, F. (eds.) PERVASIVE 2004. LNCS, vol. 3001, pp. 1–17. Springer, Heidelberg (2004) [3] Barth, A.T., Hanson, M.A., Harry, J., Powell, C., Unluer, D., Wilson, S.G., Lach, J.: Bodycoupled communication for body sensor networks. In: Proceedings of the ICST 3rd International Conference on Body Area Networks, BodyNets 2008 (2008) [4] Brezmes, T., Gorricho, J.-L., Cotrina, J.: Activity Recognition from Accelerometer Data on a Mobile Phone. In: Omatu, S., Rocha, M.P., Bravo, J., Fern´andez, F., Corchado, E., Bustillo, A., Corchado, J.M. (eds.) IWANN 2009. LNCS, vol. 5518, pp. 796–799. Springer, Heidelberg (2009) [5] Chang, C.-C., Lin, C.-J.: LIBSVM: a library for support vector machines (2001), software http://www.csie.ntu.edu.tw/˜cjlin/libsvm [6] Cornelius, C., Kotz, D.: On Usable Authentication for Wireless Body Area Networks. In: Proceedings of the First USENIX Workshop on Health Security and Privacy (HealthSec) (2010) [7] Fitbit, Inc. Fitbit (October 2010), http://www.fitbit.com/ [8] Freescale Semiconductor. Freescale Xtrinsic accelerometers optimize resolution and battery life in consumer devices (September 2010), press release http://www.media.freescale.com/phoenix.zhtml?c=196520& p=irol-newsArticle&ID=1470583

Recognizing Whether Sensors Are on the Same Body

349

[9] Kunze, K.S., Lukowicz, P.: Dealing with sensor displacement in motion-based onbody activity recognition systems. In: Proceedings of the Tenth International Conference on Ubiquitous Computing (UbiComp), pp. 20–29 (2008) [10] Kunze, K.S., Lukowicz, P., Junker, H., Tr¨oster, G.: Where am I: Recognizing On-body Positions of Wearable Sensors. In: Strang, T., Linnhoff-Popien, C. (eds.) LoCA 2005. LNCS, vol. 3479, pp. 264–275. Springer, Heidelberg (2005) [11] Lester, J., Hannaford, B., Borriello, G.: “Are You with Me?” - Using Accelerometers to Determine If Two Devices Are Carried by the Same Person. In: Ferscha, A., Mattern, F. (eds.) PERVASIVE 2004. LNCS, vol. 3001, pp. 33–50. Springer, Heidelberg (2004) [12] Maurer, U., Smailagic, A., Siewiorek, D.P., Deisher, M.: Activity Recognition and Monitoring Using Multiple Sensors on Different Body Positions. In: Proceedings of the International Workshop on Wearable and Implantable Body Sensor Networks (BSN), pp. 113–116 (2006) [13] Mayrhofer, R., Gellersen, H.: Shake Well Before Use: Intuitive and Secure Pairing of Mobile Devices. IEEE Transactions on Mobile Computing 8(6), 792–806 (2009) [14] Ravi, N., Dandekar, N., Mysore, P., Littman, M.L.: Activity Recognition from Accelerometer Data. In: Proceedings of the Twentieth National Conference on Artificial Intelligence (AAAI), pp. 1541–1546 (2005) [15] SparkFun Electronics. WiTilt v2.5 (October 2010), Data sheet http://www.sparkfun.com/datasheets/Sensors/WiTilt_V2_5.pdf [16] Sriram, J.C., Shin, M., Choudhury, T., Kotz, D.: Activity-aware ECG-based patient authentication for remote health monitoring. In: Proceedings of the Eleventh International Conference on Multimodal Interfaces (ICMI), pp. 297–304 (2009)

Sensing and Classifying Impairments of GPS Reception on Mobile Devices Henrik Blunck, Mikkel Baun Kjærgaard, and Thomas Skjødeberg Toftegaard Department of Computer Science Aarhus University, Denmark {blunck,mikkelbk,tst}@cs.au.dk

Abstract. Positioning using GPS receivers is a primary sensing modality in many areas of pervasive computing. However, previous work has not considered how people’s body impacts the availability and accuracy of GPS positioning and for means to sense such impacts. We present results that the GPS performance degradation on modern smart phones for different hand grip styles and body placements can cause signal strength drops as high as 10-16 dB and double the positioning error. Furthermore, existing phone applications designed to help users identify sources of GPS performance impairment are restricted to show raw signal statistics. To help both users as well as application systems in understanding and mitigating body and environment-induced effects, we propose a method for sensing the current sources of GPS reception impairment in terms of body, urban and indoor conditions. We present results that show that the proposed autonomous method can identify and differentiate such sources, and thus also user environments and phone postures, with reasonable accuracy, while relying solely on GPS receiver data as it is available on most modern smart phones.

1 Introduction Positioning using GPS receivers is a primary sensing modality in many areas of pervasive computing, such as behavior recognition (e.g., health status monitoring [20]), collaborative sensing (map generation [15] and environment impact monitoring [17]) and community applications (e.g., Micro-Blogging [4] and GeoPages [3]). In these application domains, the GPS receivers are assumed to be worn and used by people during their everyday life. However, the mentioned articles do not consider the impact of the user’s body on the positioning performance. Several of the above articles mention that the applications described depend on GPS performance parameters such as availability and accuracy, but link difference and impairment in GPS performance only to the user’s surrounding environments, e.g. urban or indoors. In the first part of our paper, we study and analyze the body impacts on the performance of GPS receivers, focusing on in-phone systems, and intending to inform researchers and developers about these impacts. Our work builds on knowledge from existing studies of the body impact on in-phone GSM communication [1,19], while our methodology as well as our analysis results differ in nature from those described for GSM communication, since, first, the performance parameters of GPS differ from those of communication services, and, second, since a variety of factors, other than K. Lyons, J. Hightower, and E.M. Huang (Eds.): Pervasive 2011, LNCS 6696, pp. 350–367, 2011. c Springer-Verlag Berlin Heidelberg 2011 

Sensing and Classifying Impairments of GPS Reception

351

body effects, impacts GPS positioning more severely than GSM communication (due to the weakness of the GPS signals); such factors include the user’s environment (e.g. urban or indoor) as well as, potentially, other simultaneous phone operations, e.g. GSM or WiFI transmissions or CPU computations [5]. Our study is also motivated by the recent body related issues with modern antenna designs in mobile phones [7] and by a recent short paper by Vaitl et al. [24] who quantify the GPS positioning accuracy for four on-body locations and three phone models in walking experiments. The second part of this paper is motivated by the fact that existing mobile phone applications designed to help users identify sources of GPS performance degradation are restricted to radar views of satellites’ signal strength and accuracy estimates. To help both users as well as application systems in understanding and mitigating body, urban and indoor effects, we want to provide information to the user about which effects are impacting the current performance of the GPS. For the indoor effects we build on the results of our recent study of indoor positioning using GPS presented in Kjærgaard et al. [8]. Consequently, in the second part of this paper we present a concept for how to differentiate these effects utilizing only signal quality data made available by inphone GPS modules, enabling the GPS receiver as a new sensor modality for sensing body placement and environment. Our method calculates a number of features from the signal quality data among others it compares data to an open sky model of how strong signals should have been received given no impairments. The calculated feature values are used as an input to a standard machine learning algorithm that outputs a classification of current positioning impairments. This concept is motivated foremost by the potential of information about GPS impairments and respective sources to improve GPS positioning quality and quality awareness: Both through GPS receiver algorithms, middleware [14] and application systems, utilizing such information, but also via informing the user directly via on-phone applications about current impairments, increasing his understanding of the position quality and help answering questions, such as “What is impacting my GPS positioning accuracy?” and “Can I improve GPS performance by changing my grip style or placement?” One might consider if the need for answering these questions could be removed by switching to other positioning means, such as WiFi or GSM positioning. But while we found body impacts to cause GPS positioning errors in the range between three to thirty meters, the WiFi or GSM positioning exhibits usually even larger errors in rural and urban areas [13]. We make the following contributions in this work: First, we argue that body related issues are significant for GPS performance and present results for different hand grip styles and body placements which show that signal strength drops as high as 10-16 dB can be experienced and double the positioning error. Finally, we propose a method for sensing and classifying GPS reception and positioning impairments in terms of body, urban and indoor conditions using a set of features calculated via a model for open-sky conditions. We present results that show that the method can estimate the correct cause with reasonable accuracies. The remainder of this paper is structured as follows: In Section 2 we give a brief introduction and overview of research on GPS with a focus on in-phone GPS systems. In Section 3 we present our study of body-related impacts on GPS reception. In Section 4

352

H. Blunck, M.B. Kjærgaard, and T.S. Toftegaard

we present the proposed method for sensing present GPS performance impairments and for identifying their sources. In Section 5 we discuss shortcomings and potential improvements and utilizations of the proposed method and provide directions for future work. Finally, Section 6 concludes the paper.

2 GPS: Operation Basics and Sources of Impairment In this section we review basic concepts and recent advances in GPS positioning, as well as, research on how the user’s environment impacts GPS performance and on how the user’s body impacts other phone signal operations, specifically GSM signaling. 2.1 GPS Operation Principles GPS satellites send signals for civilian use at the L1 frequency at 1.575 GHz; these signals are modulated with a Pseudo-Random Noise (PRN) code unique to each satellite. A GPS receiver tries to acquire each GPS satellite’s signal by correlating the signal spectrum it receives at L1 with a local copy of the satellite’s PRN code. An acquisition is successful, once the local copy is in sync with the received signal, which requires shifting the copy appropriately both in time and in frequency. The latter shift is due to the Doppler effect caused by the satellite’s and the user’s relative motion. Once a satellite’s signal has been acquired, the receiver tracks it, that is, the receiver continuously checks the validity of the shift parameters above and updates them if necessary. Each satellite’s signal is modulated not only with its PRN code but additionally with a navigation message, which contains almanac data (for easier acquisition of further satellites) as well as its precise ephemeris data, that is the satellite’s predicted trajectory as a function of time, allowing GPS receivers to estimate the current position of the satellite. Finally, to achieve precise 3D positioning with a standard GPS receiver via trilateration, the positions of and distances to at least 4 satellites have to be known; those distances can be computed from the time shift maintained while tracking the respective satellites. As a general rule, the more satellites can be tracked, and the wider they are spread over the sky as seen by the user, the more precise the positioning –due to the additional distance data and a satellite geometry resulting in less error-prone lateration. A popular enhancement of GPS positioning is given by Assisted GPS (A-GPS) [25], which provides assistance data to GPS receivers via an additional communication channel, which for in-phone GPS operation is usually the cellular network. This assisting data contains ephemerides and often also atmospheric corrections. A-GPS eases satellite acquisition and can therefore drastically reduce the time to first fix and the initial positioning imprecision of a receiver, once the assisting data has been transmitted. Furthermore, A-GPS can improve positioning accuracy by eliminating systemic, e.g. atmospheric, error sources [16, Chapter 13.4]. 2.2 Environment Impacts on GPS Performance GPS performance degrades in terms of both coverage and accuracy when experiencing problematic signal conditions, e.g. in urban canyons and especially in indoor environments. The cause for this is termed signal fading, subsuming two fundamental signal

Sensing and Classifying Impairments of GPS Reception

353

processing obstacles: First, when GPS signals penetrate building materials, they are subjected to attenuation, resulting in lower signal-to-noise ratio (SNR). Furthermore, the signal is subject to multipath phenomena: Reflection and refraction of the signal results in multiple echoes of the line-of-sight (LOS) signal arriving at the receiver. Low signal-to-noise ratios and multipath handicap both acquiring and tracking GPS signals and usually result in less reliable positioning due to less suitable satellite geometry and less accurate individual time shifts measurements. For investigations of GPS positioning in urban and indoor environments and its limitations, see, e.g., [8,22,27]. High-Sensitivity GPS (HSGPS) [12] subsumes advances in GPS receiver technology to alleviate the limitations mentioned above. HSGPS is claimed to allow tracking for received GPS signal strengths down to -190 dBW, corresponding to a nominal SNR value of 14 dB: three orders of magnitude less than the GPS signal strength to be expected in open-sky conditions [16]. These thresholds are constantly being improved using new processing techniques [25, Ch. 6]. 2.3 In-Phone Signal Recption and Antenna Design Considerations Today, most smart-phones allow for reliable and accurate GPS positioning in open-sky conditions. Van Diggelen lists the main technological advances which have led to this achievement, stating furthermore, that “we thought the main benefit of this would be indoor GPS, but perhaps even more importantly it has meant very, very cheap antennas in mobile phones” [26]. It is agreed within the GPS research community, that antenna design, placement, and utilization is key for the further improvement of in-phone GPS positioning [5,6]. Central aspects in this challenge are the cost-effectiveness of the antenna design and the limiting of interference caused by other in-phone components, such as the GSM communication module. Finally, increasing form factor minimization also increases the constraints on antenna size, suggesting cohabitation, i.e., the use of one antenna for multiple services such as GPS reception, and GSM, Bluetooth, or WiFi communication [5]. More recently, a growing focus on in phone GPS technology lies on limiting the power consumption [25] and consequently, most GPS chip manufacturers emphasize and provide details about the improved energy-efficiency of their latest products for in-phone integration. While the in-phone GPS reception is strongly influenced by the kind of environment, e.g. urban or indoor, another source of impairment can be the user himself, more specifically the parts of the user’s body, which are either i) close to or even ii) contacting with the GPS in-phone antenna, or iii) just blocking the line-of-sight between the antenna and specific GPS satellites. All these three phenomena have impacts on GPS reception, the magnitude of which depends also on the design of the smartphone used. Sokova and Forssell give indications, that in difficult positioning conditions, e.g., indoor environments even pedestrians passing by can cause severe impairment of GPS reception [21]. In general, the closer the body is to a receiving antenna and the more it shields it, the more signal power dissipates into the body, impairing the desired resonation of the antenna with the incoming signal. Such body effects have been investigated thoroughly for the sending and receiving of signals of various types, most prominently for cell phone communication signals [1,2]. The above research identified that for the quality of signal sending and reception the following (interrelated) parameters are crucial:

354

H. Blunck, M.B. Kjærgaard, and T.S. Toftegaard

the antenna type, its location within the phone, and the way the user holds the phone, specifically the phone’s orientation and the amount of body shielding and contact the phone’s antenna is subjected to. Furthermore, the results of these studies allow the conclusion that the body impacts on signal reception are complex to model in simulation and that the respective results often differ from the effects as observed in real-world situations. Furthermore, the effects depend in a complex manner on the signal frequency. Consequently, these studies provide an intuition about the body effects on the reception of signals at the GPS signal frequencies –but due to the GPS signals differing from GSM signals not only in frequency, but also in strength, purpose and structure, these studies don’t allow for proper predictions of body effects on GPS reception in real-life use-cases, and even less so for predictions of the resulting impacts on the experienced GPS positioning performance.

3 Quantifying the Body and Phone Impact on GPS Reception In this section we present results from measurements designed to quantify the impact of the user’s presence and handling of the phone in real-world settings. More specifically, we measure impacts of various grip styles selected according to previous work; both in this and the subsequent section we will relate these impacts also to effects originating from the user’s environment. 3.1 Methodology The primary measure we used in our analysis of in-phone GPS performance are the signal strengths as they are experienced on the phones for the GPS satellites tracked by the phone. It has been observed that this set of signal strength values gives a good indication of overall GPS positioning quality including the essential performance parameters availability and accuracy, and we provide evidence for that in Section 3.2. Hence, to evaluate the impairment of GPS performance caused by a form of user interaction, e.g., a certain grip style, we measured the signal strengths in respective setups and compared the observed SNR values to those observed on a reference phone affected neither by body nor user environment impacts. To be able to draw valid conclusions from such signal strength comparisons, it is essential, that the everyday difference in observed signal strengths between two unaffected reference phones is small. To validate this assumption, we have collected sixty hours of measurements with two unaffected Google Nexus One phones placed statically in open-sky conditions, 2 meters apart from each other, with no nearby pedestrians, and running only our measurement collection software. As we measure body and user environment impacts over 10 minutes using two phones and average the GPS signal reception properties over this time span, the two assumptions we depend on are that a) the deviation in signal strength between phones within the ten minutes are small and b) that for the same phone the signal properties measured differ only slightly in consecutive measurements. Mainly to validate our measurement setups, we also investigated whether interference between close by GPS receivers [5] can impair any GPS performance measures: In several experiments in which several GPS enabled phones were placed as close as 5 centimeters apart we observed no visible degradations.

Sensing and Classifying Impairments of GPS Reception

355

Table 1. Absolute signal strength deviations between two unaffected phones in open sky conditions for Google Nexus One phones Scenario

Four Strongest Satellites [dB] All Satellites [dB] μ σ max μ σ max Between Phones 1.10 0.86 4.35 1.20 0.93 4.91 Same Phone (Consecutive) 0.62 0.47 2.29 1.01 0.81 4.04

As shown in Table 1 for the validation of our assumption, we give quantifications of signal strength differences by mean, standard deviation and maximum differences – averaged over i) the four GPS satellite signals which are received strongest on the phone and ii) over all satellite signals tracked by the phone1. The results in Table 1 indicate mean variations around 1 dB with standard deviations below 1 dB and maximum variations below 5 dB. It follows that for our results to deviate from the mean with more than one standard deviation (this deviation is relevant as visual inspection of the distributions supports that they are normally distributed), the signal strength would have to differ at least 1.96 dB in the case of the four strongest satellites and 2.13 for all satellites. 3.2 Measured Impacts of the User Body on GPS Reception As reviewed in Section 2 there are results from studies of, e.g., cellular technologies that confirms that bodies negatively impact signal reception in mobile phones. We want to add to this knowledge by studying the effects on the GPS antenna. In this specific study we focus primarily on the Google Nexus One phone but also present results for the Nokia N97. To select relevant hand and body placements we base our selection on a study by Pelosi et al. [19] who identified common hand grip styles for both data and talk mobile phone usage. Based on their study we have selected three data style grips one with 3 fingers in the bottom third of the device, a five finger style and a double hand style, and a soft and a firm talk style grip with five fingers, as depicted in Figure 1. As GPS usage is also relevant when the user does not have the phone in the hand we have also evaluated an overarm jacket pocket placement, e.g., similar to popular overarm straps for runners, a trouser pocket placement and a placement in the top of a bag carried by a person. To limit the study we did not consider special casings of the phone or special phone configurations, e.g., opening of the keyboard on the Nokia N97. We conducted the experiments outdoors in open sky conditions and collected measurements for 10 minutes with one affected phone held in the evaluated body position and a unaffected phone statically placed 1.5 meter away from the person carrying out the experiment. To compare the data we calculate the drop in signal strength as the mean signal strength difference between the measurements from the affected and the unaffected phone. The results listed in Table 2 from the measurements with two Google Nexus One phones show that signal strength drops depending on the hand grip style and body 1

We chose to give measure - i) additional to measure - ii) because the strongest satellites will also be the most important contributors to the positioning accuracy of the GPS. Therefore, a significant drop of their SNR will have a higher impact than that of the SNR of already weaker satellites that the GPS may weigh less (or not even consider) in the position computation [10].

356

H. Blunck, M.B. Kjærgaard, and T.S. Toftegaard

Fig. 1. Illustrations of different hand grip styles

placement. The most significant drops between 6.9 to 10.6 dB are experienced with Talkstyle, firm grip, 5 fingers and Datastyle, double hand. For comparison, Kjærgaard et al. [8] lists the following values for attenuation of glass 2.43 dB, a wooden wall 4.8 dB, a brick wall 10.38 dB and reinforced concrete 16.7 dB. Generally the drops for the four strongest and all visible satellites correlate only the Running style, overarm jacket pocket scenario deviates where the drop considering all satellites is 3.7 dB. Table 2. Drops in signal strength for the four strongest and all satellites with different hand grip styles and body placements comparing pairs of Google Nexus One and Nokia N97 phones Google Nexus One Nokia N97 Scenario Four Strongest [dB] All [dB] Four Strongest [dB] All [dB] Running style, overarm 0.3 3.7 Datastyle, 3 fingers 0.4 0.4 4.7 2.7 Everyday style, bagpack 1.5 2.0 Talkstyle, soft grip, 5 fingers 2.5 2.0 Talkstyle, firm grip, 5 fingers 6.9 7.3 17.3 14.2 Datastyle, 5 fingers 8.0 3.6 11.8 10.5 Everyday style, trouser back pocket 9.4 6.6 Datastyle, double hand 10.6 8.8 16.1 14.5

To argue that the signal strength drops are not only pertinent to the Google Nexus One phone we collected similar measurements with two Nokia N97 phones for a subset of the scenarios. The results are also listed in Table 2 and indicates even bigger drops in the range of 4.7 dB to 17.3 dB which is five dB higher than for the newer Google Nexus One for similar hand grip styles and body placements. From these measurements we can conclude that the body impact is present and in some cases amounts to the attenuation experienced in indoor environments. To quantify the effect on positioning accuracy during everyday use measurements were collected by a person walking a 4.85 kilometer tour twice through both opensky and urban positioning conditions carrying six phones with different placement. The data set consists of ground truth positions and 1 Hz GPS from the built-in sensors in the Google Nexus One and the Nokia N97. The ground truth was collected at 4Hz with a high accuracy u-blox LEA-5H receiver with an dedicated antenna placed on the top of a backpack carried by the collector. The ground truth measurements were manually inspected to make sure they followed the correct route of the target.

Sensing and Classifying Impairments of GPS Reception

357

100

100

90

90

80

80 Cumulative Probability [%]

Cumulative Probability [%]

Figure 2 shows error plots for four of the Google Nexus One and Nokia N97 traces, respectively. The error is computed as the distance between the ground truth positions reported by the dedicated GPS to the positions reported by the phones. The figures depict cumulative distributions of the individual positioning errors throughout the traces. From the figures one can observe a significant difference in accuracy comparing the nearly body unaffected placements of Upper compartment of bagpack and Datastyle, 3 fingers to the affected trouser placements and the Datastyle, 5 fingers. Considering the median, the increase in error is for the Google Nexus One from five meters to ten meters and for the Nokia N97 from ten meters to twenty meters. One can therefore conclude that the body impact can have a strong impact on the positioning accuracy. In a study, conducted at the same time as ours and for three mobile phones Vaitl et al. [24] also identified the phone placement within trouser pockets as the worst for GPS accuracy.

70 60 50 40 30

70 60 50 40 30

20

20

10

10

0

0 0

10

20

30

40

Error Distance [Meters] Upper compartmet of bagpack Datastyle, 5 fingers

50

0

20

40

60

80

100

Error Distance [Meters] Left jacket pocket Trouser back pocket

Datastyle, 3 fingers Trouser front pocket

Trouser back pocket

Fig. 2. Drops in positioning accuracy with different hand grip styles and body placements for Google Nexus One (left), and Nokia N97 (right)

The body effects also impact the positioning availability in the everyday measurements, but only for the Nokia N97, whereas for the Google Nexus One, which was released in early 2010, 16 months after the N97, there were no major drops. For the N97s the availability dropped from 88% during collection in the Datastyle, 3 fingers case to 51% for the back pocket trouser placement. Availability drops may occur also on the Nexus One, as we observed in the data set collected to evaluate the method proposed in Section 4: In four urban and five indoor data traces a Nexus One placed in a trouser pocket did not produce any fixes at all and in one urban and four indoor data traces, and a Nexus One held with the Datastyle, 5 fingers grip style did produce only very few. In the same traces both a phone held with grip style Datastyle, 3 fingers and a reference phone placed some meters away from the person collecting the data produced continuous fixes throughout the experiments. Therefore, we can conclude that body effects impact GPS availability, however, more significantly for the N97 than for the Nexus One.

358

H. Blunck, M.B. Kjærgaard, and T.S. Toftegaard

4 Sensing and Classifying Impairment Sources To assist both application systems as well as the user in understanding and mitigating body and environment-induced effects, we aim to provide information about which effects are currently impacting the GPS device’s performance and to which extent. In the following, we present an approach for differentiating such effects and respective sources utilizing only GPS signal quality data, in particular SNR measurements paired with directional information about GPS satellites; note, that such data is made available by most popular last generation smart-phones. The information about sources of GPS performance impairment, that we provide, adds to existing user assistance such as radar views of satellites strengths and accuracy estimates, and can be delivered as either visual, audible or tactile feedback, assisting, e.g. in answering questions such as “What is impacting my positioning accuracy?” and “Can I improve GPS performance by changing my grip style or placement?”. 4.1 Classification Concept and Procedure The proposed concept is illustrated in Figure 3. An in-phone GPS module outputs signal quality measurements –even in conditions in which only few and very weak signals can be acquired. Therefore, our method is functional even in cases where the GPS module is not able to produce any position fixes at all. From the signal quality measurements a range of features are computed with the help of an open-sky model that estimates how strong signals would be received for a given satellite and on the device if placed in open-sky conditions and not suffering from body, indoor or urban effects.

Open Sky Model

In-Phone GPS receiver

(prn,elevation, azimuth,snr)

Feature Extraction

(feature1, feature2, ..., featuren)

Classification Model

Impairment Sources

Fig. 3. Process for classifying impairment sources via analyzing in-phone signal measurements

Classification output domain. The classification model outputs information about inferred sources of GPS performance impairment. We have chosen to consider two types of GPS positioning impairments, environment- and body-induced ones. Furthermore, we considered two resulting categorizations: In the first one we distinguish twelve categories, i.e. twelve potential outcomes of the classification, corresponding to the twelve combinations of three environment types (open-sky, urban and indoor) and four phone placement and posture types (datastyle 3 fingers, datastyle 5 fingers, trouser back pocket, and no-body reference). In the coarser second categorization, we consider only six categories, combining the three environment types (open-sky, urban and indoor) with cases of no or weak body effects (datastyle 3 fingers and no-body reference) and cases of significant body effects (datastyle 5 fingers and trouser back pocket).

Sensing and Classifying Impairments of GPS Reception

359

4.2 Feature Extraction The aim in designing suitable features for classification is to numerically capture the occurrence of patterns which –ideally– are specific to a single environment or on-body phone placement or posture, or –more generally– allow us to separate different (combinations of) environment and body placement types. In the following, we first sketch some of these patterns we look for, then we present features designed for detecting them, after introducing a utilized reference model, which holds expected SNR values in open-sky conditions. Some of the patterns we are looking for include: When separating environment types. Naturally, for open-sky environments hardly any SNR drops or deviations w.r.t the model of SNRs in open-sky conditions occur. If deviations exist, they are distributed normally over both azimuths and elevations of the satellites tracked. In urban environments, drops occur for low elevation satellites, while the signals of higher satellites are received stronger. Indoors, satellite signals are received strongest through windows and wall openings, i.e. from satellites at low elevation and within ’horizontal clusters’, i.e. at specific azimuth ranges, corresponding to window areas and wall openings. When separating phone placement types. The blocking effect of the user body shows for different on-body placements in SNR drops of a particular range of azimuth values: E.g., for the trouser back pocket-placement this range is almost hemispherical. In contrast, when the user holds the phone in hand, the attenuation is more evenly distributed with regards to azimuth. Furthermore, different grips styles can often be distinguished by the overall amount of attenuation. An Empirical Model of Open-Sky Conditions. Our identification of GPS reception impairments is based mainly on interpreting signal degradations. An indicator of the latter, which is even more suitable than the absolute SNR values recorded, is given by the drops of SNR w.r.t. ideal conditions, i.e. when not impacted by body- or environmentinduced effects. Therefore, our system is supported by an Open Sky Model which provides estimations of the SNR values to be currently expected on the device at the user’s geographic position. To the best of our knowledge, there does not yet exist an accurate theoretical model for open-sky GPS signal conditions. There are two main reasons for this, firstly, that SNR values depend both on properties of the antenna and the receiver chip, and secondly, that the transmission power of GPS satellites vary depending on their generation and age. Therefore, to characterize open-sky conditions we propose to use a devicespecific and empirical parametrized model. This model holds for each GPS satellite a function which maps for each GPS satellite its evaluation to a Gaussian distribution of the SNR of that satellite at that elevation, as recorded by the device. The resulting function table contains less than 3000 entries, .i.e. a mere 60 kilobytes. The motivation for modeling not only average SNR values, but also error distributions is that deviations are caused by several error sources, such as atmospheric weather, ground multi-path effects and integer rounding imprecision of the elevation data. Note, that since GPS orbits repeat every sidereal day, these differences are observable from the mappings which each use only 24 hours of the recorded data. The daily SNR pattern, as well as

360

H. Blunck, M.B. Kjærgaard, and T.S. Toftegaard

45

40

35

SNR

30

25

20

15

10

5 0

10

20

30

40

50

60

70

80

90

Elevation (degree) 1st Nexus, hours 1-24 1st Nexus, hours 25-48

2nd Nexus, hours 1-24 2nd Nexus, hours 25-48

Fig. 4. SNR measurements for GPS satellite PRN 31, recorded on Google Nexuses in open-sky no-body conditions

differences in this pattern observed over two consecutive days are depicted exemplary for GPS satellite PRN 31 as recorded in Aarhus, Denmark, in Figure 4. As our model’s mappings are gained by empirical data won over 48 hours at one particular location, the accuracy of the model diminishes with the temporal and spatial distance from that recording. We will though later on present evidence that our system performs well also with less accurate or less detailed models. Classification Features Employed. We now give an overview of the 29 features our current classification is based on, as well as discuss their suitability and limits in providing a successful classification of (combinations of) body-induced and environmentinduced impairments. Features Based on Averaged SNR Drop: One of the features used in classification considers the experienced SNR drop w.r.t. the open sky model, averaged over all satellites within the GPS constellation which were trackable according to the open sky model. This feature captures the overall level of signal attenuation experienced. Figure 5(a) illustrates the differentiation of phone postures and placements achievable by this feature: The probability distributions shown represent output of the feature, i.e. average SNR drops. Each distribution subsumes data, described in more detail in Section 4.3, from 12 five-minute measurements for each particular combination of environment and phone-body context. Note, that to achieve better visualization of the characteristica of the distribution, feature output was beforehand mapped to bins. The plot shows, that in open-sky environments almost no SNR drops occur for the grip style using only 3 fingers, while in contrast large drops occur (in any given environment), when the phone is held firmly with 5 fingers, and even significantly larger drops occur when the phone is kept in the back pocket of a trouser. Furthermore, the two distributions in Figure 5(a) showing data gathered in urban settings, show that the environmental

Sensing and Classifying Impairments of GPS Reception 0.1

0.05 opensky+handheld3fingers opensky+handheld5fingers opensky+trouserbackpocket urban+nobody urban+trouserbackpocket

0.09 0.08 0.07

0.045 0.04

opensky+nobody indoor+nobody opensky+body indoor+body

0.035

0.06

probability

probability

361

0.05 0.04 0.03

0.03 0.025 0.02 0.015

0.02

0.01

0.01

0.005

0 0

5

10

15 value

20

25

0 -50

-45

-40

-35

-30

-25

-20

-15

-10

-5

0

value

(a) SNR drop w.r.t. opensky model, averaged (b) Maximal SNR difference between two over all receivable satellites satellites, consecutive in the increasing order of the received satellites by elevation Fig. 5. Probability plots exemplifying two classification features

attenuation visibly ’blurrs’ the separation between different contexts; a fact that represents a general challenge for the body context classification: As a rule of thumb, the more attenuating and impairing the user’s current environment is in itself, the harder it becomes to differentiate between different phone-body contexts. Features Based on Elevation Order: The following feature is designed to distinguish environment types: It captures the maximally negative difference of SNR between two satellites, consecutive in the increasing order of the received satellites by elevation. Figure 5(b) shows output of this feature, originating from the measurements mentioned above. The distributions shown result from grouping measurements, undertaken in open-sky as well as in indoor areas and with various body-phone contexts, by severity of GPS impairment –following the coarser one of the categorizations described in Section 4.1. The plot shows, that for most indoor locations the feature output values are high. This is because satellites are received most strongly at low elevations, e.g. through windows, and because these satellites are ultimately followed in the elevation ordered sequence by a satellite that is highly attenuated by either walls or ceilings. In contrast, the open-sky measurements provide feature values close to zero. This again gives evidence, that if the impairment in one domain –in this case the body-induced one– is severe, it becomes harder for the feature to differentiate impairments of another domain –in this case of the user environment, .i.e. to tell apart indoor locations from open-sky ones. Another feature is obtained as a variant of the one just described: It measures the maximal positive, instead of negative, SNR difference; this allows for identifying the urban environment type, since in urban canyons usually the satellites above the ’skyline’ of surrounding buildings are received significantly stronger. Features Based on SNR Drop Order: Three of the features currently employed are computed on basis of the sequence of the tracked satellites, sorted by the SNR drop experienced for them. E.g., when averaging over the azimuth difference between satellites consecutive in that sequence, the resulting feature captures the entropy for the sequence w.r.t. azimuth; for open-sky no-body conditions this entropy is usually high, while the lowest entropy was obtained in a no-body setup in indoor locations: Satellites similar in

362

H. Blunck, M.B. Kjærgaard, and T.S. Toftegaard

azimuth often reach the user through the same building element and therefore exhibit also similar SNR values. Features Based on Azimuth Order: Alternatively, if one sorts the satellites instead by azimuth, one can capture, e.g. hemispherical, body shielding effects for in-pocket setups: One of our features identifies the azimuth half-space separation, for which the difference between the averaged signal drop in the respective hemispheres is maximized. This feature naturally generalizes for angles other than 180 degree. Further Features: Further features are extracted from already established signal quality indicators, e.g., DOP values as provided by the phone’s GPS system [16]. Feature design techniques, which we consider worth of exploring in future work, we discuss in Section 5. 4.3 Classification Results As depicted in the procedural outline given in Figure 3, the classification model has to infer present GPS positioning impairments, once provided with feature outputs, computed as described in the previous section. To implement the classification model we chose to use the Weka Machine Learning toolkit [28]. Prior to processing the feature outputs, we aggregate them by averaging over a five second window to remove outliers2 . To evaluate the proposed classification concept, we collected a data set at three opensky, three urban, and three indoor exemplary locations, employing different phone-body contexts in order to cover all twelve classification context categories listed in Section 4.1. At each location four consecutive measurements were collected at a fixed position, and for two opposite directions e.g. facing north and south, resp. For each location and orientation, measurements were undertaken by two users of differing stature in order to investigate the influence of body physique on the GPS reception and on our impairment classification process. In each measurement, four Google Nexus One phones were used where one was placed 2 meters away as reference, one was placed in the user’s trouser back pocket and the two remaining ones were held by the user with the Datastyle, 3 fingers and the Datastyle, 5 fingers grip style, resp. In total, 144 measurement traces of five minutes each were collected. Each trace contains GPS position fixes and signal quality measurements sampled at 1 Hz. This experimental setup was designed to collect a balanced data set w.r.t. locations, users, orientations and body placement; however, since at some locations positioning availability was not 100%, some categories naturally have fewer data samples. Thus, we have applied a re-sampling filter to balance the data, so that we are able to judge the performance of the classification model directly from the classification accuracies and confusion matrices. In Table 3 we present classification results for six and twelve categories, resp., and for two different machine learning algorithms, the basic Naive Bayes algorithm and the more accurate J48 decision tree algorithm and for three types of evaluations: firstly, 2

We evaluated different window sizes: For the window size chosen the classification results benefited from the resulting noise removal, whereas for larger windows it suffered too much from the size reduction of the data set.

Sensing and Classifying Impairments of GPS Reception

363

Table 3. Classification accuracy results for classifying GPS positioning impairments into six and twelve categories, resp. 10 Fold Cross-Validation Different Persons Different Orientation 12 Categories Naive Bayes Decision Tree 6 Categories Naive Bayes Decision Tree

59 94

44 50

46 51

73 96

71 75

66 73

ten fold cross-validation on the complete data set, secondly, training with data for one person and testing the resulting classification model with data from the other person and, thirdly, training with data for the respectively first chosen orientation and testing with the data of the respective opposite orientation. The results show that the decision tree algorithm performs better than the naive Bayes algorithm with accuracy rates of 94% and 96% for twelve and six categories, respectively. However, there are indications of overfitting because separating the training and test data, either w.r.t. to person or orientation, lowers the results to 75% and 73%, respectively, for six categories and even more for twelve categories. Similarly, training with data from only half of the investigated environments and subsequent testing with the remaining data, results in a lower classification accuracy –implicating, that for accurate classification in arbitrary environments training data from a broader variety of locations would be essential, as well as further development of the proposed features. To further analyze how the errors are distributed, Table 4 shows the confusion matrix for the results of the decision tree algorithm with six categories and when separating training and test data w.r.t. the collecting person. From the matrix one can see that data from the classes open-sky, no body and urban, no body are classified highly accurate, whereas data from indoor, no body and indoor, body is not; the poor separation performance of the algorithm in this case shows in high confusion values of 28.9% and 30.8% between the two categories. This observation is in agreement with the statement made

Table 4. Confusion matrix for the decision three algorithms for six categories and with separate training and test data depending on the collecting person Classified As open-sky urban indoor open-sky no body no body no body nody open-sky, no body 92.1 0.1 1.2 6.6 urban, no body 3.3 88.2 6.8 0 indoor, no body 7.9 4.7 45.6 9.3 open-sky, body 8.7 0 6.4 70.0 urban, body 0 0 8.5 0 indoor, body 0 0 28.9 0.2

urban body 0 1.5 1.7 1.7 81.5 0.9

indoor body 0 0.3 30.8 13.1 10.0 70.0

364

H. Blunck, M.B. Kjærgaard, and T.S. Toftegaard

above, that in weak signal environments it is harder to differentiate body effects: even telling none body effects situations from those with body effects becomes challenging. The results shown in Table 4 were obtained using an open-sky model that considered each satellite independently which has the drawback that reference data ideally should be collected at many places on the globe to account for local differences in observable signal strengths. Thus, we have also evaluated our method using an alternative, simpler model that combines gathered SNR data across satellites to compute the strength of GPS reception per elevation, averaged over all satellites. When using this model instead of the proposed open-sky model, the accuracy of the classifications, using either the naive Bayes or the decision tree algorithm, did only decrease by 2 to 3 percent.

5 Discussion The work presented shows that classification of GPS impairment sources can be done, relying only on current GPS signal quality data obtainable on most last generation smart phones. In the following, we discuss the classification concept presented, how to utilize it, as well as future research directions for refining and improving it. The feasibility of the classification concept in terms of classification accuracy has been documented in Section 4. We expect further improvement from integrating additional features into our classification procedure. Among these will be the detection of geometric clusters (w.r.t. elevation and azimuth) of similarly strong satellites, to detect environmental features, such as window areas indoors, and street canyons in urban areas. Additionally, the consideration of the recent data history –additional to the most recent GPS signal quality data snapshot, may allow to more reliably detect and keep track of ’static’ features such as windows, walls or buildings, and to tell them apart more easily from body features, which are always ’moving’ with the user. Worth investigating is also the incorporation of indicators for the user’s phone handling as well as his context, e.g., his transportation mode, which are provided through data from sensors other than the GPS. E.g., Vahdatpour et al. [23] propose to detect a device’s on-body placement from accelerometer readings. In terms of output semantics, an integration of further as well as more fine-grained classes of user environments, distinguishing between different building types and transportation vehicles, the user may currently be in, would be desirable, depending on user requirements and application scenarios. W.r.t. resulting classification accuracies, Table 4 illustrates the naturally poorer absolute accuracy when classifying into a higher number of classes; note, though that the majority of false classifications still determine the user environment correctly and only confuse similar phone postures and placements. Furthermore, the issue that the diversity of the physiques of users result in drops in classification performance, as noted in Table 3, has to be addressed: First, the system should be trained through data gathered by subjects of various statures. Secondly, we plan to evaluate the benefits of providing the user with a training procedure, designed to determine the impact of the user’s physique on GPS reception w.r.t., comparing the gathered data with the training data provided by users of various physiques. The potential for application and middleware-specific benefits of the proposed on-device sensing and classifying of GPS impairments in real-time require further

Sensing and Classifying Impairments of GPS Reception

365

investigation, for improving both ad-hoc and general user behaviour, as well as for enhancing the positioning quality and quality awareness of GPS receivers. The latter can be achieved since identifying current impairment sources can inform the receiver which satellites’ ranging data to trust: GPS receivers can ignore ranging information from individual satellites which are believed to be distorted or currently received only indirectly; while such selection has been shown to potentially improve GPS accuracy [11], knowing the current user environment is crucial for picking proper selection criteria: E.g., in open sky environments stronger signals usually provide preciser ranging information, whereas in indoor environments the contrary can hold, when the strongest signals are likely to be signal reflections, reaching the user through windows, but only indirectly and thus yielding large ranging distortations [8]. Furthermore, providing application developers with access to the classifications is an example of benefitial seamfull design for developers [14]: E.g., in a positioning middleware, which is designed following a seamfull design approach to provide translucency w.r.t. the positioning process, the classifications could be used as an input to adapt application logic. To investigate to which extent the proposed classification can benefit ad-hoc and general user behaviour, we are currently considering a phone application which provides as feedback the classification results regarding reception impairments and sensed environments and phone postures. Additionally, we want to explore ways to provide visual or acoustic feedback, which not only assesses and classifies current GPS impairment sources, but which can also guide the user towards a more beneficial phone holding posture or placement, or help him identify more reception-beneficial spots within or close to his current environment, e.g., using information collected by fingerprinting GPS positioning quality [9]. Finally, the computational load and energy consumption induced by different impairment classification schemes should be investigated, to ensure that real-time processing is feasible on common smart phones –also for feature sets, larger than the currently used one. Our preliminary investigations indicate that real-time processing on common smart phones is possible for the presented system.

6 Conclusions We presented a concept for sensing present impairments of GPS reception and positioning performance, and for classifying impairment sources in terms of body, urban and indoor context. Results obtained from a measurement campaign provided reasonable classification accuracy and a proof of concept, that both the type of environment, the user is currently in, as well as the way a user is currently holding or storing his phone can be determined with reasonable accuracy through analysis solely of GPS signal quality data as available on most modern smart phones. Finally, in Section 5 further improvements of the accuracy of the presented classification system were identified, and directions for how to bring benefits of such a classification concept to the users were illustrated. Additionally, to assess user-body effects on GPS reception and to aid and inform existing and future research and application systems, we have empirically evaluated for different hand grip styles and body placements the respective effects on GPS positioning performance of modern GPS enabled smart-phones. The evaluation showed that GPS

366

H. Blunck, M.B. Kjærgaard, and T.S. Toftegaard

reception depends highly on how the phone is kept or held, and that body-effects can cause attenuation of average signal strength of up to 10-16 dB, which is more than that caused by a typical brick wall, and can lead to a doubling of the median positioning error as experienced in open-sky conditions in the absence of body effects.

Acknowledgment We thank Lasse Haugsted Rasmussen for his help in collecting measurements, acknowledge the financial support granted by the Danish National Advanced Technology Foundation for the project ”Galileo: A Platform for Pervasive Positioning” (009-2007-2) and by the Danish National Research Foundation for MADALGO - Center for Massive Data Algorithmics and acknowledge Nokia and Google for hardware grants.

References 1. Al-Mously, S.I., Abousetta, M.M.: Anticipated impact of hand-hold position on the electromagnetic interaction of different antenna types/positions and a human in cellular communications. International Journal of Antennas and Propagation 2008 (2008) 2. Alexiou, A., Kostarakis, P., Christofilakis, V., Zervos, T., Alexandridis, A., Dangakis, K., Soras, C., Petrovi´c, V., Kolundˇzija, B., Dordevi´c, A.: Interaction between gsm handset helical antenna and user’s head: Theoretical analysis and experimental results. The Environmentalist 25(2), 215–221 (2005) 3. Cai, Y., Xu, T.: Design, analysis, and implementation of a large-scale real-time locationbased information sharing system. In: Proceeding of the 6th International Conference on Mobile Systems, Applications, and Services, pp. 106–117. ACM, New York (2008) 4. Gaonkar, S., Li, J., Choudhury, R.R., Cox, L., Schmidt, A.: Micro-blog: sharing and querying content through mobile phones and social participation. In: Proc. of the 6th International Conference on Mobile Systems, Applications, and Services, pp. 174–186. ACM, New York (2008) 5. Haddrell, T., Ricquier, N., Phocas, M.: Mobile-phone gps antennas. GPS World 2 (2010) 6. Hurte, B., Leisten, O.: Getting into pockets and purses. GPS World 11 (2005) 7. A. Inc. Letter from apple regarding iphone 4. Press Release (July 2010) 8. Kjærgaard, M.B., Blunck, H., Godsk, T., Toftkjær, T., Christensen, D.L., Grønbæk, K.: Indoor positioning using GPS revisited. In: Flor´een, P., Kr¨uger, A., Spasojevic, M. (eds.) Pervasive Computing. LNCS, vol. 6030, pp. 38–56. Springer, Heidelberg (2010) 9. Kjærgaard, M.B., Weckemann, K.: PosQ: Unsupervised Fingerprinting and Visualization of GPS Positioning Quality. In: Proceedings of the Second International Conference on Mobile Computing, Applications, and Services (MobiCASE 2010). Springer, Heidelberg (2010) 10. Kuusniemi, H., Wieser, A., Lachapelle, G., Takala, J.: User-level reliability monitoring in urban personal satellite-navigation. IEEE Transactions on Aerospace and Electric Systems 43, 1305–1318 (2007) 11. Kuusniemi, H., Lachapelle, G.G.: GNSS signal reliability testing in urban and indoor environments. In: Proceedings of the NTM Conference (2004) 12. Lachapelle, G., Kuusniemi, H., Dao, D., MacGougan, G., Cannon, M.: HSGPS signal analysis and performance under various indoor conditions. Navigation, Inst. of Navigation 51(1), 29–43 (2004)

Sensing and Classifying Impairments of GPS Reception

367

13. LaMarca, A., Chawathe, Y., Consolvo, S., Hightower, J., Smith, I., Scott, J., Sohn, T., Howard, J., Hughes, J., Potter, F., Tabert, J., Powledge, P., Borriello, G., Schilit, B.: Place Lab: Device Positioning Using Radio Beacons in the Wild. In: Gellersen, H.-W., Want, R., Schmidt, A. (eds.) PERVASIVE 2005. LNCS, vol. 3468, pp. 116–133. Springer, Heidelberg (2005) 14. Langdal, J., Schougaard, K.R., Kjærgaard, M.B., Toftkjær, T.: PerPos: A translucent positioning middleware supporting adaptation of internal positioning processes. In: Gupta, I., Mascolo, C. (eds.) Middleware 2010. LNCS, vol. 6452, pp. 232–251. Springer, Heidelberg (2010) 15. Minamimoto, S., Fujii, S., Yamaguchi, H., Higashino, T.: Local Map Generation using Position and Communication History of Mobile Nodes. In: Proceedings of the 2010 IEEE International Conference on Pervasive Computing and Communications, pp. 2 –10 (2010) 16. Misra, P., Enge, P.: Global Positioning System: Signals, Measurements, and Performance, 2nd edn., Navtech (2006) 17. Mun, M., Reddy, S., Shilton, K., Yau, N., Burke, J., Estrin, D., Hansen, M., Howard, E., West, R., Boda, P.: Peir, the personal environmental impact report, as a platform for participatory sensing systems research. In: Proceedings of the 7th International Conference on Mobile Systems, Applications, and Services, pp. 55–68. ACM, New York (2009) 18. Kivek¨as, T.L.O., Ollikainen, J., Vainikainen, P.: Bandwith, sar, and efficiency of internal mobile phone antennas. IEEE Trans. on Electromagnetic Compatibility 46(1), 71–86 (2004) 19. Pelosi, M., Franek, O., Knudsen, M., Christensen, M., Pedersen, G.: A grip study for talk and data modes in mobile phones. IEEE Transactions on Antennas and Propagation 57(4), 856–865 (2009) 20. Ryder, J., Longstaff, B., Reddy, S., Estrin, D.: Ambulation: A tool for monitoring mobility patterns over time using mobile phones. In: International Conference on Computational Science and Engineering, pp. 927 –931 (2009) 21. Sokolova, N., Forssell, B.: Moderate pedestrian traffic: Indoor hsgps receiver performance. European Journal of Navigation 5(3), 2–7 (2007) 22. Teuber, A., Paonni, M., Kropp, V., Hein, G.: Galileo signal fading in an indoor environment. In: Proc. 21st Intl. Techn. Meeting Satellite Division Inst. of Navigation (ION GNSS) (2008) 23. Vahdatpour, A., Amini, N., Sarrafzadeh, M.: On-body device localization for health and medical monitoring applications. In: Proceedings of the Ninth Annual IEEE International Conference on Pervasive Computing and Communications (2011) 24. Vaitl, C., Kunze, K., Lukowicz, P.: Does on-body location of a gps receiver matter? In: Int. Workshop on Wearable and Implantable Body Sensor Networks, pp. 219–221 (2010) 25. van Diggelen, F.: A-GPS: Assisted GPS, GNSS, and SBAS. Artech House, Boston (2009) 26. van Diggelen, F.: The smartphone revolution. GPS World 12 (2009) 27. Watson, R., Lachapelle, G., Klukas, R., Turunen, S., Pietil, S., Halivaara, I.: Investigating gps signals indoors with extreme high-sensitivity detection techniques. Navigation, Inst. of Navigation 52(4), 199–213 (2006) 28. Witten, I.H., Frank, E.: Data Mining: Practical machine learning tools and techniques, 2nd edn. Morgan Kaufmann, San Francisco (2005)

Author Index

Adams, Matt 232 Akatsuka, Daisuke 250 Alt, Florian 258 Atlas, Les 50 Badger, Eric 206 Bales, Elizabeth 1 Becker, Richard 133 Benford, Steve 232 Bernheim Brush, A.J. 79, 188 Bial, Dominik 258 Blunck, Henrik 350 Bulling, Andreas 314 Buthpitiya, Senaka 97 C´ aceres, Ram´ on 133 Campbell, Andrew T. Campbell, Tim 50 Cha, Hojung 115 Cornelius, Cory 332 Cosker, Darren 294 Derleth, Peter Dey, Anind K. Elhart, Ivan

152

314 19, 97, 170 258

Farr, Ju Row 232 Farrahi, Katayoun 214 Feilner, Manuela 314 Ferreira, Denzil 19 Flintham, Martin 232 Fogarty, James 50 Froehlich, Jon 50 Gatica-Perez, Daniel 214 Greenhalgh, Chris 232 Griss, Martin 97 Harboe, Gunnar 258 Huang, Elaine 258 Isaacman, Sibren Johnson, Peter

133 294

Karlson, Amy K. 188 Kawsar, Fahim 70 Kishino, Yasue 276 kitahara, Soh 250 Kjærgaard, Mikkel Baun 350 Kobourov, Stephen 133 Koo, Jahyoung 115 Kortuem, Gerd 70 Kostakos, Vassilis 19 Kotz, David 332 Krumm, John 79 Langheinrich, Marc 258 Larson, Eric 50 Latora, Vito 152 Lin, Chun-Jung 294 Liu, Jie 188 Lu, Hong 188 Luyten, Kris 70 Madan, Anmol 214 Maekawa, Takuya 276 Marshall, Joe 232 Martonosi, Margaret 133 Mascolo, Cecilia 152 Memarovic, Nemanja 258 Musolesi, Mirco 152 Nakanishi, Yasuto Ohmori, Takuro O’Neill, Eamonn Oppermann, Leif

250 250 294 232

Paek, Tim 206 Patel, Shwetak 50 Pentland, Alex (Sandy) Priyantha, Bodhi 188

214

Reeves, Stuart 232 Robson, Christine 34 Roggen, Daniel 314 Rosenthal, Stephanie 170 Rowland, James 133 Rudchenko, Dmitry 206

370

Author Index

Saba, Elliot 50 Sakurai, Yasushi 276 Scellato, Salvatore 152 Schmidt, Albrecht 258 Scipioni, Marcello P. 258 Sekiguchi, Koji 250 Setlur, Vidya 1 Smith, Kevin 70 Sohn, Timothy 1 Stiefmeier, Thomas 314 Suyama, Takayuki 276

Tandavanitj, Nick 232 Tessendorf, Bernd 314 Toftegaard, Thomas Skjødeberg Tr¨ oster, Gerhard 314 Varshavsky, Alexander Veloso, Manuela 170 Vermeulen, Jo 70 Wright, Michael

133

294

Zhang, Ying 97 Zimmerman, Thomas

34

350