Active Media Technology: 6th International Conference, AMT 2010, Toronto, Canada, August 28-30, 2010, Proceedings (Lecture Notes in Computer Science ... Applications, incl. Internet Web, and HCI)

Lecture Notes in Computer Science Commenced Publication in 1973 Founding and Former Series Editors: Gerhard Goos, Juris ...

Author: Aijun An | Sheila Petty | Runhe Huang

12 downloads 1125 Views 17MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form

DOWNLOAD PDF

Lecture Notes in Computer Science Commenced Publication in 1973 Founding and Former Series Editors: Gerhard Goos, Juris Hartmanis, and Jan van Leeuwen

Editorial Board David Hutchison Lancaster University, UK Takeo Kanade Carnegie Mellon University, Pittsburgh, PA, USA Josef Kittler University of Surrey, Guildford, UK Jon M. Kleinberg Cornell University, Ithaca, NY, USA Alfred Kobsa University of California, Irvine, CA, USA Friedemann Mattern ETH Zurich, Switzerland John C. Mitchell Stanford University, CA, USA Moni Naor Weizmann Institute of Science, Rehovot, Israel Oscar Nierstrasz University of Bern, Switzerland C. Pandu Rangan Indian Institute of Technology, Madras, India Bernhard Steffen TU Dortmund University, Germany Madhu Sudan Microsoft Research, Cambridge, MA, USA Demetri Terzopoulos University of California, Los Angeles, CA, USA Doug Tygar University of California, Berkeley, CA, USA Gerhard Weikum Max Planck Institute for Informatics, Saarbruecken, Germany

6335

Aijun An Pawan Lingras Sheila Petty Runhe Huang (Eds.)

Active Media Technology 6th International Conference, AMT 2010 Toronto, Canada, August 28-30, 2010 Proceedings

13

Volume Editors Aijun An York University Department of Computer Science and Engineering Toronto, ON, M3J 1P3, Canada E-mail: [email protected] Pawan Lingras Saint Mary’s University Department of Mathematics and Computing Science Halifax, NS, B3H 3C3, Canada E-mail: [email protected] Sheila Petty University of Regina Faculty of Fine Arts Regina, SK, S4S 0A2, Canada E-mail: [email protected] Runhe Huang Hosei University Faculty of Computer and Information Sciences Tokyo 184-8584, Japan E-mail: [email protected]

Library of Congress Control Number: 2010933076 CR Subject Classification (1998): H.4, I.2, H.5, C.2, J.1, I.2.11 LNCS Sublibrary: SL 3 – Information Systems and Application, incl. Internet/Web and HCI ISSN ISBN-10 ISBN-13

0302-9743 3-642-15469-7 Springer Berlin Heidelberg New York 978-3-642-15469-0 Springer Berlin Heidelberg New York

This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, re-use of illustrations, recitation, broadcasting, reproduction on microfilms or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable to prosecution under the German Copyright Law. springer.com © Springer-Verlag Berlin Heidelberg 2010 Printed in Germany Typesetting: Camera-ready by author, data conversion by Scientific Publishing Services, Chennai, India Printed on acid-free paper 06/3180

Preface

This volume contains the papers selected for presentation at the 2010 International Conference on Active Media Technology (AMT 2010), jointly held with the 2010 International Conference on Brain Informatics (BI 2010), at York University, Toronto, Canada, during August 28-30, 2010. Organized by the Web Intelligence Consortium (WIC) and IEEE Computational Intelligence Society Task Force on Brain Informatics (IEEE-CIS TF-BI), this conference was the sixth in the AMT series since its debut conference at Hong Kong Baptist University in 2001 (followed by AMT 2004 in Chongqing, China, AMT 2005 in Kagawa, Japan, AMT 2006 in Brisbane, Australia, AMT 2009 in Beijing, China). Active media technology (AMT) is a new area of research and development in intelligent information technology and computer science. It emphasizes the proactive, adaptive and seamless roles of interfaces and systems as well as new media in all aspects of digital life. Over the past few years, we have witnessed rapid developments of AMT technologies and applications ranging from business and communication to entertainment and learning. Examples include Facebook, Twitter, Flickr, YouTube, Moodle, Club Penguin and Google Latitude. Such developments have greatly changed our lives by enhancing the way we communicate and do business. The goal of the AMT conferences is to provide an international forum for exchanging scientific research and technological achievements in building AMTbased systems. AMT 2010 featured a selection of the latest research work and applications from the following areas related to AMT: active computer systems and intelligent interfaces, adaptive Web systems and information foraging agents, AMT for the Semantic Web, data mining, ontology mining and Web reasoning, e-commerce and Web services, entertainment and social applications of active media, evaluation of active media and AMT-based systems, intelligent information retrieval, machine learning and human-centered robotics, multi-agent systems, multi-modal processing, detection, recognition, and expression analysis, semantic computing for active media and AMT-based systems, smart digital media, Web-based social networks, and Web mining and intelligence. All the papers submitted to AMT 2010 were rigorously reviewed by three committee members and external reviewers. The selected papers offered new insights into the research challenges and development of AMT systems. AMT 2010 (together with BI 2010) also featured four keynote talks given by Ben Shneiderman of the University of Maryland, Jianhua Ma of Hosei University, Yingxu Wang of the University of Calgary, and Vinod Goel of York University. They spoke on their recent research in technology-mediated social participation, active smart u-things and cyber individuals, cognitive informatics and denotational mathematical means for brain informatics, and fractionating the rational

VI

Preface

brain, respectively. The abstracts of the first two keynote talks, which were on AMT, are included in this volume. AMT 2010 could not be successful without a team effort. We would like to thank all the authors who contributed to this volume. We also thank the Program Committee members and external reviewers for their dedicated contribution in the paper selection process. Our special thanks go to Tetsuya Yoshida and Yue Xu for organizing a special session on text analysis and utilization, and to Hanmin Jung, Li Chen and Sung-Pil Choi for organizing a special session on technology intelligence. We are grateful to the Chairs and members of the Organizing Committee for their significant contribution to the organization of the conference. In particular, we would like to acknowledge the generous help received from Ning Zhong, Jimmy Huang, Vivian Hu, Jessie Zhao, Jun Miao, Ellis Lau and Heather Bai. Our appreciation also goes to Juzhen Dong for her excellent technical support of the AMT 2010 conference management system and its websites. Last but not the least, we thank Alfred Hofmann and Anna Kramer of Springer for their help in coordinating the publication of this special volume in an emerging and interdisciplinary research area. We appreciate the support and sponsorship from York University and the University of Regina. August 2010

Aijun An Pawan Lingras Sheila Petty Runhe Huang

Conference Organization

Conference General Chairs Sheila Petty Runhe Huang

University of Regina, Canada Hosei University, Japan

Program Chairs Aijun An Pawan Lingras

York University, Canada Saint Mary’s University, Halifax, Canada

Organizing Chair Jimmy Huang

York University, Canada

Publicity Chairs Daniel Tao Jian Yang

Queensland University of Technology, Australia International WIC Institute/BJUT, China

WIC Co-chairs/Directors Ning Zhong Jiming Liu

Maebashi Institute of Technology, Japan Hong Kong Baptist University, Hong Kong

IEEE-CIS TF-BI Chair Ning Zhong

Maebashi Institute of Technology, Japan

WIC Advisory Board Edward A. Feigenbaum Setsuo Ohsuga Benjamin Wah Philip Yu L.A. Zadeh

Stanford University, USA University of Tokyo, Japan University of Illinois, Urbana-Champaign, USA University of Illinois, Chicago, USA University of California, Berkeley, USA

VIII

Organization

WIC Technical Committee Jeffrey Bradshaw Nick Cercone Dieter Fensel Georg Gottlob Lakhmi Jain Jianchang Mao Pierre Morizet-Mahoudeaux Hiroshi Motoda Toyoaki Nishida Andrzej Skowron Jinglong Wu Xindong Wu Yiyu Yao

UWF/Institute for Human and Machine Cognition, USA York University, Canada University of Innsbruck, Austria Oxford University, UK University of South Australia, Australia Yahoo! Inc., USA Compiegne University of Technology, France Osaka University, Japan Kyoto University, Japan Warsaw University, Poland Okayama University, Japan University of Vermont, USA University of Regina, Canada

Program Committee Bill Andreopoulos Pradeep Atrey Virendra Bhavsar Jiannong Cao Sharat Chandran Li Chen Sung-Pil Choi Chin-Wan Chung Tharam Dillon Abdulmotaleb El Saddik Alexander Felfernig William Grosky Daryl Hepting Jiajin Huang Wolfgang Huerst Joemon Jose Hanmin Jung Brigitte Kerhervé Yang Liu Yijuan Lu Brien Maguire Wenji Mao Wee Keong Ng

Technische Universit¨ at Dresden, Germany University of Winnipeg, Canada University of New Brunswick, Canada Hong Kong Polytechnic University, Hong Kong Indian Institute of Technology, Bombay, India Hong Kong Baptist University, Hong Kong Korea Institute of Science and Technology Information, Korea Korea Advanced Institute of Science and Technology (KAIST), Korea Curtin University of Technology, Australia University of Ottawa, Canada Graz University of Technology, Austria University of Michigan, USA University of Regina, Canada Beijing University of Technology, China Utrecht University, The Netherlands University of Glasgow, UK Korea Institute of Science and Technology Information, Korea Université du Québec à Montréal, Canada Shandong University, China Texas State University, USA University of Regina, Canada Institute of Automation, Chinese Academy of Sciences, China Nanyang Technological University, Singapore

Organization

Yoshihiro Okada Eugene Santos Dominik Slezak Xijin Tang

Hiroyuki Tarumi Ruizhi Wang Yue Xu Rong Yan Jian Yang Tetsuya Yoshida Mengjie Zhang Shichao Zhang Zili Zhang William Zhu

Kyushu University, Japan University of Connecticut, USA University of Warsaw and Infobright Inc., Poland Institute of Systems Science, Academy of Mathematics and Systems Science, Chinese Academy of Sciences, China Kagawa University, Japan Tongji University, China Queensland University of Technology, Australia Facebook, USA International WIC Institute, Beijing University of Technology, China Hokkaido University, Japan Victoria University of Wellington, New Zealand University of Technology, Australia Southwest University, China University of Electronic Science and Technology of China, China

External Reviewers Ansheng Ge Mehdi Kargar Mauricio Orozco

IX

Damon Sotoudeh-Hosseini Peng Su Karthikeyan Vaiapury

Table of Contents

Keynote Talks Technology-Mediated Social Participation: Deep Science and Extreme Technology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ben Shneiderman Active Smart u-Things and Cyber Individuals . . . . . . . . . . . . . . . . . . . . . . . Jianhua Ma

1 5

Active Computer Systems and Intelligent Interfaces A Case for Content Distribution in Peer-to-Peer Networks . . . . . . . . . . . . . Morteza Analoui and Mohammad Hossein Rezvani

6

Interactive Visualization System for DES . . . . . . . . . . . . . . . . . . . . . . . . . . . Mohamed S. Asseisah, Hatem M. Bahig, and Sameh S. Daoud

18

Intelligent Implicit Interface for Wearable Items Suggestion . . . . . . . . . . . . Khan Aasim, Aslam Muhammad, and A.M. Martinez-Enriquez

26

Adaptive Web Systems and Information Foraging Agents Folksonomy-Based Ontological User Interest Profile Modeling and Its Application in Personalized Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Xiaogang Han, Zhiqi Shen, Chunyan Miao, and Xudong Luo

34

Visualizing Threaded Conversation Networks: Mining Message Boards and Email Lists for Actionable Insights . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Derek L. Hansen, Ben Shneiderman, and Marc Smith

47

AMT for Semantic Web and Web 2.0 A Spatio-temporal Framework for Related Topic Search in Micro-Blogging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Shuangyong Song, Qiudan Li, and Nan Zheng

63

Exploiting Semantic Hierarchies for Flickr Group . . . . . . . . . . . . . . . . . . . . Dongyuan Lu and Qiudan Li

74

Understanding a Celebrity with His Salient Events . . . . . . . . . . . . . . . . . . . Shuangyong Song, Qiudan Li, and Nan Zheng

86

XII

Table of Contents

User Interests: Definition, Vocabulary, and Utilization in Unifying Search and Reasoning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yi Zeng, Yan Wang, Zhisheng Huang, Danica Damljanovic, Ning Zhong, and Cong Wang Ontology Matching Method for Efficient Metadata Integration . . . . . . . . . Pyung Kim, Dongmin Seo, Mikyoung Lee, Seungwoo Lee, Hanmin Jung, and Won-Kyung Sung

98

108

Data Mining, Ontology Mining and Web Reasoning Multiagent Based Large Data Clustering Scheme for Data Mining Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . T. Ravindra Babu, M. Narasimha Murty, and S.V. Subrahmanya Fractal Based Video Shot Cut/Fade Detection and Classification . . . . . . . Zeinab Zeinalpour-Tabrizi, Amir Farid Aminian-Modarres, Mahmood Fathy, and Mohammad Reza Jahed-Motlagh

116

128

Performance Evaluation of Constraints in Graph-Based Semi-supervised Clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Tetsuya Yoshida

138

Analysis of Research Keys as Tempral Patterns of Technical Term Usages in Bibliographical Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hidenao Abe and Shusaku Tsumoto

150

Natural Language Query Processing for Life Science Knowledge . . . . . . . . Jin-Dong Kim, Yasunori Yamamoto, Atsuko Yamaguchi, Mitsuteru Nakao, Kenta Oouchida, Hong-Woo Chun, and Toshihisa Takagi

158

E-Commerce and Web Services A Semantic Web Services Discovery Algorithm Based on QoS Ontology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Baocai Yin, Huirong Yang, Pengbin Fu, and Xiaobo Chen Implementation of an Intelligent Product Recommender System in an e-Store . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Seyed Ali Bahrainian, Seyed Mohammad Bahrainian, Meytham Salarinasab, and Andreas Dengel Recommendation of Little Known Good Travel Destinations Using Word-of-Mouth Information on the Web . . . . . . . . . . . . . . . . . . . . . . . . . . . . Kouzou Ohara, Yu Fujimoto, and Tomofumi Shiina

166

174

183

Table of Contents

XIII

Entertainment and Social Applications of Active Media The Influence of Ubiquity on Screen-Based Interfaces . . . . . . . . . . . . . . . . . Sheila Petty and Luigi Benedicenti

191

Perception of Parameter Variations in Linear Fractal Images . . . . . . . . . . Daryl H. Hepting and Leila Latifi

200

Music Information Retrieval with Temporal Features and Timbre . . . . . . Angelina A. Tzacheva and Keith J. Bell

212

Evaluation of Active Media and AMT Based Systems Towards Microeconomic Resources Allocation in Overlay Networks . . . . . Morteza Analoui and Mohammad Hossein Rezvani

220

Investigating Perceptions of a Location-Based Annotation System . . . . . . Huynh Nhu Hop Quach, Khasfariyati Razikin, Dion Hoe-Lian Goh, Thi Nhu Quynh Kim, Tan Phat Pham, Yin-Leng Theng, Ee-Peng Lim, Chew Hung Chang, Kalyani Chatterjea, and Aixin Sun

232

Apollon13: A Training System for Emergency Situations in a Piano Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yuki Yokoyama and Kazushi Nishimoto

243

Intelligent Information Retrieval Exploring Social Annotation Tags to Enhance Information Retrieval Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Zheng Ye, Xiangji Jimmy Huang, Song Jin, and Hongfei Lin

255

A Hybrid Chinese Information Retrieval Model . . . . . . . . . . . . . . . . . . . . . . Zhihan Li, Yue Xu, and Shlomo Geva

267

Term Frequency Quantization for Compressing an Inverted Index . . . . . . Lei Zheng and Ingemar J. Cox

277

Chinese Question Retrieval System Using Dependency Information . . . . . Jing Qiu, Le-Jian Liao, and Jun-Kang Hao

288

Machine Learning and Human-Centered Robotics A Novel Automatic Lip Reading Method Based on Polynomial Fitting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Meng Li and Yiu-ming Cheung

296

XIV

Table of Contents

An Approach for the Design of Self-conscious Agent for Robotics . . . . . . . Antonio Chella, Massimo Cossentino, Valeria Seidita, and Calogera Tona K-Means Clustering as a Speciation Mechanism within an Individual-Based Evolving Predator-Prey Ecosystem Simulation . . . . . . . Adam Aspinall and Robin Gras

306

318

Improving Reinforcement Learning Agents Using Genetic Algorithms . . . Akram Beigi, Hamid Parvin, Nasser Mozayani, and Behrouz Minaei

330

Robust and Efficient Change Detection Algorithm . . . . . . . . . . . . . . . . . . . . Fei Yu, Michael Chukwu, and Q.M. Jonathan Wu

338

Multi-Agent Systems Building Users’ Profiles from Clustering Resources in Collaborative Tagging Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Maya Rupert and Salima Hassas

345

Some Optimizations in Maximal Clique Based Distributed Coalition Formation for Collaborative Multi-Agent Systems . . . . . . . . . . . . . . . . . . . . Predrag T. Toˇsić and Naveen K.R. Ginne

353

Multi-Modal Processing, Detection, Recognition, and Expression Analysis Enhanced Intra Coding of H.264/AVC Advanced Video Coding Standard with Adaptive Number of Modes . . . . . . . . . . . . . . . . . . . . . . . . . . Mohammed Golam Sarwer and Q.M. Jonathan Wu Extracting Protein Sub-cellular Localizations from Literature . . . . . . . . . . Hong-Woo Chun, Jin-Dong Kim, Yun-Soo Choi, and Won-Kyung Sung

361 373

Semantic Computing for Active Media and AMT Based Systems Enhancing Content-Based Image Retrieval Using Machine Learning Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Qinmin Vivian Hu, Zheng Ye, and Xiangji Jimmy Huang

383

Modeling User Knowledge from Queries: Introducing a Metric for Knowledge . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Frans van der Sluis and Egon L. van den Broek

395

Computer-Assisted Interviewing with Active Questionnaires . . . . . . . . . . . Seon-Ah Jang, Jae-Gun Yang, and Jae-Hak J. Bae

403

Table of Contents

XV

Smart Digital Media Assessing End-User Programming for a Graphics Development Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Lizao Fang and Daryl H. Hepting

411

Visual Image Browsing and Exploration (Vibe): User Evaluations of Image Search Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Grant Strong, Orland Hoeber, and Minglun Gong

424

Web Based Social Networks Contextual Recommendation of Social Updates, a Tag-Based Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Adrien Joly, Pierre Maret, and Johann Daigremont Semantic Web Portal: A Platform for Better Browsing and Visualizing Semantic Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ying Ding, Yuyin Sun, Bin Chen, Katy Borner, Li Ding, David Wild, Melanie Wu, Dominic DiFranzo, Alvaro Graves Fuenzalida, Daifeng Li, Stasa Milojevic, ShanShan Chen, Madhuvanthi Sankaranarayanan, and Ioan Toma NicoScene: Video Scene Search by Keywords Based on Social Annotation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yasuyuki Tahara, Atsushi Tago, Hiroyuki Nakagawa, and Akihiko Ohsuga

436

448

461

Web Mining, Wisdom Web and Web Intelligence Social Relation Based Search Refinement : Let Your Friends Help You! . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Xu Ren, Yi Zeng, Yulin Qin, Ning Zhong, Zhisheng Huang, Yan Wang, and Cong Wang An Empirical Approach for Opinion Detection Using Significant Sentences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Anil Kumar K.M. and Suresha

475

486

Extracting Concerns and Reports on Crimes in Blogs . . . . . . . . . . . . . . . . . Yusuke Abe, Takehito Utsuro, Yasuhide Kawada, Tomohiro Fukuhara, Noriko Kando, Masaharu Yoshioka, Hiroshi Nakagawa, Yoji Kiyota, and Masatoshi Tsuchiya

498

Automatically Extracting Web Data Records . . . . . . . . . . . . . . . . . . . . . . . . Dheerendranath Mundluru, Vijay V. Raghavan, and Zonghuan Wu

510

XVI

Table of Contents

Web User Browse Behavior Characteristic Analysis Based on a BC Tree . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Dingrong Yuan and Shichao Zhang

522

Clustering Web Users Based on Browsing Behavior . . . . . . . . . . . . . . . . . . . Tingshao Zhu

530

Privacy Preserving in Personalized Mobile Marketing . . . . . . . . . . . . . . . . . Yuqing Sun and Guangjun Ji

538

Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

547

Technology-Mediated Social Participation: Deep Science and Extreme Technology Ben Shneiderman Dept of Computer Science & Human-Computer Interaction Lab, University of Maryland, College Park, MD 20742 USA [email protected]

Abstract. The dramatic success of social media such as Facebook, Twitter, YouTube, Flickr, blogs, and traditional discussion groups empowers individuals to become active in local and global communities. With modest redesign, these technologies can be harnessed to support national priorities such as healthcare/wellness, disaster response, community safety, energy sustainability, etc. This talk describes a research agenda for these topics that develops deep science questions and extreme technology challenges. Keywords: social media, participation, motivation, reader-to-leader framework.

1 Introduction The remarkable expansion of social media use has produced dramatic entrepreneurial successes and high expectations for the future. Beyond these commercial successes, many observers see the potential for social transformations in economic, political, social, educational, medical, and many other domains. Understanding how to increase the motivations for participation is a deep science question that will occupy researchers for many decades. Similarly, building scalable technological foundations that are secure and reliable will challenge software designers and implementers. The goal of these deep science and extreme technologies is to provide billions of users with the capacity to share information, collaborate on ambitious projects, and organize successful governance structures, while coping with malicious attacks, providing high levels of security, and ensuring reliability.

2 Deep Science The enduring questions of raising human motivation have taken on new importance in the age of social media. Wikipedia is a great success story because of its innovative strategies for motivating users to contribute the knowledge and to collaborate with others. But even in this success story, only one in a 1000 readers become registered contributors, and even fewer become regular collaborators who work together over weeks and months. Similarly, while there are billions of viewers of YouTube the numbers of contributors of content is small. A. An et al. (Eds.): AMT 2010, LNCS 6335, pp. 1–4, 2010. © Springer-Verlag Berlin Heidelberg 2010

2

B. Shneiderman

Fig. 1. The Reader-to-Leader Framework suggests that the typical path for social media participation moves from reading online content to making contributions, initially small edits, but growing into more substantive contributions. The user-generated content can be edits to a wiki, comments in a discussion group, ratings of movies, photos, music, animations, or videos. Collaborators work together over periods of weeks or months to make more substantial contributions, and leaders act to set policies, deal with problems, and mentor new users [1].

Motivation or persuasion is an ancient human notion, but the capacity to study it on a global scale is just becoming a reality. The move from controlled laboratory experiments to interventions in working systems is happening because designers and researchers have enabled the capture of usage patterns on a scale never before possible. The Reader-to-Leader Framework [1] (Fig. 1) provides an orderly way of discussing the strategies and conducting research. At each stage innovative entrepreneurs and researchers have developed these strategies such as showing the number of views of a video, enabling ratings of contributions, honoring richer collaborations, and empowering leaders. Many other theories and frameworks are being proposed as commercial, government, and academic researchers rapidly expand their efforts. Traditional social science theories are being adapted to understand, predict, and guide designers who seek to increase trust, empathy, responsibility, and privacy in the online world. Similarly, mathematical theories of network analysis are being enhanced to accommodate the distinctly human dynamics of online social systems. The shift from descriptive and explanatory theories that are based on statistical analyses to predictive and prescriptive theories that provide guidance for community managers is happening rapidly, but much work remains to be done.

3 Extreme Technology The past 40 years of computing technology have produced remarkable progress. Strong credit goes to the chip developers who made the rapid and sustained strides characterized by Moore’s Law – doubling of chip density, speed, capacity every 18 months. Equal credit goes to the user interface designers who opened the doors to billions of users by creating direct manipulation interfaces based on carefully designed menus, effective graphical interfaces, convenient input devices, and comprehensible visual presentations.

Technology-Mediated Social Participation: Deep Science and Extreme Technology

3

The current agenda is rapidly moving to encompass the large-scale social media communities, such as the half billion users of Facebook and the four billion users of cell phones. Newer services such as Twitter have acquired more than 100 million users with billions of exchanges per month, but that is just the beginning. As individuals, organizations, companies, and governments increase their usage, the volume and pace of activity will grow bringing benefits to many users, but so will the impacts of service outages, privacy violations, and malicious attacks. Developers now recognize the primacy of the user interface in determining outcomes, so there is increased research, training, and exploratory design. Simultaneously, there is a growth in tools to track, analyze, and intervene in social media networks to as to promote more positive outcomes. One such effort is the free and open source NodeXL Project (Network Overview for Discovery and Exploration in Excel), which was initially supported by Microsoft Research (www.codeplex.com/nodexl). This tool enables importing of social media networks from Outlook, Twitter, YouTube, Flickr, WWW, etc. into Excel 2007/2010, and then gives users powerful analysis tools, plus rich visualization support [2, 3] (Fig. 2). NodeXL was designed to speed learning by social-media savvy business professionals who already use Excel, as well as by undergraduate and graduate students who

Fig. 2. This NodeXL screenshot shows the U.S. Senate voting patterns during 2007. The 100 Senators are linked to each other by edges whose strength is related to the number of similar votes. By restricting edges to those greater than 65% similarity and using a force directed layout algorithm, the clusters of Democrats (blue nodes on lower right) and Republicans (red nodes on upper left) become visible.

4

B. Shneiderman

are learning social network analysis. By providing easy import of data from important social media tools, NodeXL dramatically expands the community of users who can carry out analyses that lead to actionable business insights and research studies. NodeXL provides a rich set of visualization controls to select color, size, opacity, and other attributes of vertices and edges. The variety of layout algorithms and dynamic query filters allows users to tune the display to their needs. Varied centrality metrics for directed and undirected graphs, as well as a growing number of clustering algorithms, support exploration and discovery. NodeXL is an ongoing project that will be supported through the emerging Social Media Research Foundation (www.smrfoundation.org). Acknowledgments. Thanks to the NodeXL team (www.codeplex.com/nodexl), the community of U.S. National Science Foundation workshop participants (www.tmsp.umd.edu), the University of Maryland Human-Computer Interaction Lab, and Jennifer Preece.

References 1. Preece, J., Shneiderman, B.: The Reader-to-Leader Framework: Motivating technologymediated social participation. AIS Transactions on Human-Computer Interaction 1(1), 13–32 (2009) 2. Smith, M., Shneiderman, B., Milic-Frayling, N., Mendes-Rodrigues, E., Barash, V., Dunne, C., Capone, T., Perer, A., Gleave, E.: Analyzing (social media) networks with NodeXL. In: Proc. Communities & Technologies Conference (2009) 3. Hansen, M., Shneiderman, B., Smith, M.A.: Analyzing Social Media Networks with NodeXL: Insights from a Connected World. Morgan Kaufmann Publishers, San Francisco (2010)

Active Smart u-Things and Cyber Individuals Jianhua Ma Laboratory of Multimedia Ubiquitous Smart Environment, Department of Digital Media, Faculty of Computer and Information Sciences, Hosei University, Tokyo 184-8584, Japan [email protected]

Abstract. Due to the continuing miniaturization of chips and availability of wired/wireless communications, many kinds/forms of devices can be integrated into physical objects and ambient environments. The u-things, as opposed to pure digital e-things existing on computers/Web/Internet, are ordinary physical things with attached, embedded or blended computers, networks, and/or some other devices such as sensors, actors, e-tags and so on. Active smart u-things are ones that can, more or less, sense, compute, communicate, and may take some actions according to their goals, situated contexts, users’ needs, etc. Active smart u-things can be with different levels of intelligence from low to high, and in various intelligent forms, e.g., aware, context-aware, interactive, reactive, proactive, assistive, adaptive, automated, autonomic, sentient, perceptual, organic, life-like, cognitive, thinking, etc. Active smart u-things may cover innumerable types of physical things in the real world. They can be roughly classified into three categories, i.e., smart object, smart space and smart system, according to their appearances and functions. The grand challenge is how to enable these smart u-things to offer desired services to all people in right time, right place and right means with ubisafe guarantee. Furthermore, the essential and existence of human in cyber-physical combined spaces should be re-examined. The Cyber Individual, with a short term ‘Cyber-I’, is a real individual’s counterpart in cyberspace. Cyber-I can be seen as a comprehensive description of a real individual including one’s physical status, physiological states, psychological behaviors, personal features, social relations, history experiences, etc. Such kind of individual description and modeling is fundamental to offer personalized services to different users according to their needs and situations. Keywords: u-thing, sensor, actuator, tag, robot, smart object, space and system, ubiquitous intelligence, cyberspace, cyber individual, user modeling.

A. An et al. (Eds.): AMT 2010, LNCS 6335, p. 5, 2010. © Springer-Verlag Berlin Heidelberg 2010

A Case for Content Distribution in Peer-to-Peer Networks Morteza Analoui and Mohammad Hossein Rezvani Department of Computer Engineering, Iran University of Science and Technology (IUST) 16846-13114, Hengam Street, Resalat Square, Narmak, Tehran, Iran {analoui,rezvani}@iust.ac.ir

Abstract. In large scale peer-to-peer networks, it is impossible to perform a query request by visiting all peers. There are some works that try to find the location of resources probabilistically (i.e. non-deterministically). They all have used inefficient protocols for finding the probable location of peers who manage the resources. This paper presents a more efficient protocol that is proximityaware in the sense that it is able to cache and replicate the popular queries proportional to distance latency. The protocol dictates that the farther the resources are located from the origin of a query, the more should be the probability of their replication in the caches of intermediate peers. We have validated the proposed distributed caching scheme by running it on a simulated peer-to-peer network using the well-known Gnutella system parameters. The simulation results show that the proximity-aware distributed caching can improve the efficiency of peer-to-peer resource location services.

1 Introduction Most of the current P2P systems such as Gnutella, KazaA, and Pastry [1] fall within the category of P2P "content distribution" systems. A typical P2P content distribution system creates a distributed storage medium and allows doing services such as searching and retrieving query messages which are known as "resource location" services. The area of "content distribution systems" has a large overlap with the issue of "resource location services" in the literature. In general, there are two strands of work concerning the proximity-aware methodology. First, there are works on content distribution via constructing the P2P topology [2]. Second, there are works on resource location services [3, 4, 5]. These works assume a given topology setting such as mesh or tree for the P2P system. It has been shown by [6, 7] that finding an optimal-bandwidth topology for the P2P network is a NP-complete problem. So, we shall not try to solve the NP problem of topology construction here. Instead, we will try to optimize the proximity-aware resource locating problem within the given topology setting in the P2P system. In this paper, we are concerned with the design of a resource location service via scalable proximity-aware distributed caching mechanism. We define the resource location service as “given a resource name, find with a proximity probability, the location of peers that manage the resource.” We use Round Trip time (RTT) latency distance as the criterion for the probabilistic caching of each query. Each peer, upon A. An et al. (Eds.): AMT 2010, LNCS 6335, pp. 6–17, 2010. © Springer-Verlag Berlin Heidelberg 2010

A Case for Content Distribution in Peer-to-Peer Networks

7

receiving a query, at first searches its local cache. If the query is found, the peer returns it to the original requesting peer along with the reverse path which is traversed by the query. In this order, the query is cached in the memory of each intermediate node using replication method based on the proposed proximity-aware distributed caching mechanism. The probability of the resource replication and updating of the caches in each intermediate node is proportional to the latency distance between that node and the location where the resource is found. The rest of the paper is organized as follows. We discuss the related researches in Section 2. Section 3 presents our proposed proximity-aware resource location mechanism. Section 4 presents the performance evaluation of the proposed mechanism. Finally, we conclude in Section 5.

2 Related Work A significant research toward proximity-aware resource location services in typical Gnutella-based unstructured P2P system has been done in [4, 8]. Some query search broadcasting policies using Gnutella system has been proposed in [8] and their performance has also been compared with each other. The proximity metric in [4] is Time to Live (TTL) of the query messages. Forwarding the queries is done with a fixed probability. When a query message is reached to a peer, its TTL is decremented. The forwarding of the query messages will be stopped if its TTL is reached to zero. The search technique proposed in [9] is similar to local indices technique which is proposed in [8] with different routing policy for query message. In the other proposed techniques which mentioned in [8] each node maintains “hints” as to which nodes contain data that answer certain queries, and route messages via local decisions based on these hints. This idea itself is similar to the philosophy of hints which is used by Menasce et al. in [4]. Pastry [1] is an example of systems with “strong guarantee” that employ search techniques. These systems can locate an object by its global identifier within a limited number of hops. Zhao et al. in [10] provide a priority-aware and consumption-guided dynamic probabilistic allocation method for a typical cache memory. Utilization of a sample size of a cache memory is measured for each priority level of a computer system. Allocation probabilities for each priority level are updated based on the measured consumption/utilization, i.e. allocation is reduced for priority levels consuming too much of the cache and allocation is increased for priority levels consuming too little of the cache. Another valuable work in the area of proximity-caching in P2P systems is presented by Jung et al. in [11]. They propose a simple caching protocol, which intuitively obtains information about physical network structure. Their caching protocol utilizes the internet address (IP) i.e. first 16 bits of IP address. The metadata used in their caching protocol is exchanged using piggy-back mechanism, and they extract useful IP prefix set by using RTT threshold value. The protocol is deployed into Chord, a well-known distributed hash table-based lookup protocol. Their result show genuine relationship between physical and logical network structure.

3 Proximity-Aware Distributed Caching Each pair of nodes ( s , r ) is associated with a latency distance lat ( s , r ) representing the RTT experienced by communication between them. The latency distance corresponding

8

M. Analoui and M.H. Rezvani

to a specific pair of nodes may be measured either directly through ping messages, or estimated approximately through a virtual coordinate service. Due to space limitations, we do not explain the details of the virtual coordinate service here. Interested readers can refer to [12] for it. Every super-peer in our system has a Local Index Table (LIT) that points to locally managed resources (such as files, Web pages, processes, and devices). Each resource has a location-independent Globally Unique Identifier (GUID) that can be provided by developers of the P2P network using different means. For example, in a distributed online bookstore application, developers could use ISBNs as GUIDs [4]. Each superpeer has a directory cache (DC) that points to the presumed location of resources managed by other super-peers. An entry in the DC is a pair (id, loc) in which id is the GUID of a resource and loc is the network address of a super-peer who might store the resource locally. Each peer s has a local neighborhood N (s ) defined as the set of super-peers who have connected to it. Tables 1 and 2 provide a high-level description of the proposed proximity-aware distributed caching mechanism. The QuerySearch (QS) procedure describes the operations in which a source s is looking for a resource, namely res . The string path

s1 , ..., s m is the sequence of super-peers that

have received this message so far. This sequence is used as a reverse path to the source. The header of each query message contains a TTL field which is used to control the depth of the broadcast tree. For example, Gnutella has been implemented with a TTL parameter equal to 7. The QueryFound (QF) procedure indicates that the resource res being searched by the source s has been found at super-peer v . In this procedure, the max_latency is the latency distance between the super-peer who manages res and the farthest super-peer in the reverse path. Each super-peer, upon receiving the QS message, at first searches within its LIT. If it finds the resource in the LIT, it will return a QF message. The QF message is forwarded to the source following the reverse path which has been used by the QS message. It updates the DCs corresponding to each of the intermediate nodes as well. The contribution of our work emerges at this point where the QF message updates the LIT in each of the intermediate nodes using replication of resources based on the proposed proximityaware distributed caching mechanism. The probability of resource replication and updating the LIT corresponding to each intermediate node is proportional to the latency distance between that node and the location where the resource has been found. To this end, each intermediate node r performs the following actions with a probability that is proportional to the latency distance between itself and the node which has been found as the manager of the resource: 1) establishing a TCP connection with the super-peer who manages the resource, 2) downloading the resource object and saving it in the client c who has enough available space, and 3) updating the LIT via adding the entry ( res , c ) . If the super-peer does not find the resource in its LIT but finds it in the DC, it will send a QS message to the super-peer who is pointed to by that DC. If this super-peer no longer has the resource, the search process will be continued from that point forward. If a super-peer does not find the resource neither in its LIT nor DC, it will forward the request to each super-peer in its neighborhood with a certain probability p which is called the "broadcasting probability." This probability could vary with the length of the path that the request traverses.

A Case for Content Distribution in Peer-to-Peer Networks Table 1. QuerySearch message received by super-peer

9

r

QuerySearch( source , res , (s1, …, sm), TTL) begin If res ∈ LIT then begin max_latency= max{(lat ( r , s1 ), ..., lat ( r , s m )} send QueryFound( source , res , max_latency, (s1, …, sm-1), r ) to sm end else if ( res , loc) ∈ DC then /* send request to presumed location */ Send QuerySearch( source , res , (s1, …, sm, r ), TTL-1) to loc else if (TTL > 0) then for v i = v1 to v m do

/* v i ∈ N (r ) */

begin max_latency= max{(lat ( r , v1 ), ..., lat ( r , v m )} Send QuerySearch( source , res ,(s1, …, sm, r), TTL-1) with probability /* probability p is proportional to

lat ( r , v i )

p

*/

max_ latency

end for end if end if end

Table 2. QueryFound message received by super-peer

r

QueryFound( source , res , max_latency, (s1, …, sm), v ) begin if r ≠ source then begin add ( res , v ) to DC with probability proportional to

lat ( r , v )

do

max_ latency begin Connect to super-peer v to get res from it /* finds a local client with enough available memory */

find local client c add ( res , c ) to LIT end send QueryFound( source , res , max_latency, (s1, …, sm-1), v) end else /* end of query search process */ connect to super-peer v to get res from it. end

to sm

to

vi

10


Fig. 1 illustrates how a QS message would be propagated in the network. In the figure, the maximum number of the nodes to be traversed by a QS message is defined to be equal to 3 hops (apart from the source node itself). Similar to Gnutella, our system uses a Breath-First-Search (BFS) mechanism in which the depth of the broadcast tree is limited by the TTL criterion. The difference is that in Gnutella every query recipient node forwards the message to all of its neighbors, while in our proposal, the propagation is performed probabilistically and is done if the query is not found neither in the LIT nor in the DC of a node. In Fig. 1, the QS message originating from source S1 is probabilistically sent to super-peers S2, S3, and S4 in response to the search for the resource res . The super-peer S3 finds the resource in its LIT, but S2 and S4 do not find such an entry, hence probabilistically forward the message to the peers who have been registered in their DCs. Note that the super-peer S4 does not forward the message to S10 because, for example, in this case the forwarding probability is randomly selected to be zero. Figure 2 illustrates an example of returning QF messages in a reversed path from the location where the

Fig. 1. Forwarding a QS message using maximum hop-count equal to 3


11

Fig. 2. Forwarding the QF message through the reversed path

resource res is found to the node which the query has been originated from. The QF message is routed to the source (node S1) following the reverse path which is used by the QS message. The QF message updates the corresponding DC of each intermediate node based on the proposed proximity-aware distributed caching mechanism. The probability of replication and caching the resource object in the LIT of each intermediate node is proportional to the latency distance between that node and the location where the resource is found. The closer is the intermediate node to the discovered resource; the less will be the probability of caching the resource in the node’s LIT. This probability is shown by a graphical representation with partially boldfaced circles. In the sequence of nodes which consists of S1, S2, S6, and S13, the node S6 caches the address of the resource res with the least probability; whereas the node S1 caches it with the most probability. The probability of caching the resource res by S2 is larger than that of S6 and is smaller than that of S1.

12


4 Experimental Analysis We have performed a large number of experiments to validate the effectiveness of our proximity-aware distributed caching scheme. We have evaluated the performance of the system with a file-sharing application based on several metrics. These metrics include fraction of involving super-peers in the query search, probability of finding an entry in DCs, average number of hops to perform the query requests, and the system load. Among these metrics, the load metric is defined as the amount of work that an entity must do per unit of time. It is measured in terms of two resource types: incoming bandwidth, and outgoing bandwidth. Since the availability of the incoming and the outgoing bandwidths is often asymmetric, we have treated them as separate resources. Also, due to heterogeneity of the system, it is useful to study the aggregate load, i.e., the sum of the loads concerning to all the nodes in the system. All of the results are averaged over 10 runs of experiments and have been come up with 95% confidence intervals. We followed the general routine devised in [13] for the efficient design of the P2P network. So, as the first step, we had to generate an instance topology based on a power-law distribution. We used the PLOD algorithm presented in [14] to generate the power-law topology for the network. The second step was calculating the expected cost of actions. Among three “macro” actions, i.e., query, join, and update, which exist in a cost model [13], we have restricted our attention to the query operations. Each of these actions is composed of smaller “atomic” actions for which the costs are given in [13]. In terms of bandwidth, the cost of an action is the number of bytes being transferred. We used the specified size of the messages for Gnutella protocol as is defined in [13]. For example, the query messages in Gnutella include a 22-byte Gnutella header, a 2 byte field for flags, and a null-terminated query string. The total size of a query message, including Ethernet and TCP/IP headers, is therefore 82 plus the query string length. Some values, such as the size of a metadata record are not specified by the protocol, rather are functions of the type of the data which is being shared. To determine the number of the results which are returned to a super-peer r , we have used the query model developed in [15] which is applicable to super-peer filesharing systems as well. The number of files in the super-peer’s index depends on the particular generated instance topology I . We have used this query model to determine the expected number of the returned results, i.e. E[ N r | I ] . Since the cost of the

query is a linear function of ( N r | I ) and also since the load is a linear function of the cost of the queries, we can use these expected values to calculate the expected load of the system [13]. In the third step, we must calculate the system load using the actions. For a given query originating from the node s and terminating in the node r we can calculate the expected cost, namely C sr . Then, we need to know the rate at which the query action occurs. The default value for the query rate is 9.26×10-3 which is taken from the general statistics provided by [13]. The query requests in our experiments have been generated by a workload generator. The parameters of the workload generator can be set up to produce uniform or non-uniform distributions. Considering the cost and the rate of each query action, we can now calculate the expected load which is incurred by the node r for the given network instance I as follows


E[ M r | I ] =

∑ E [C

s ∈ Network

sr

| I ].E[ Fs ]

13

(1)

Where, Fs is the number of the queries submitted by the node s in the time unit, and E[ Fs ] is simply the query rate per user. Let us define Q as the set of all super-peer nodes. Then, the expected load of all such nodes, namely M Q is defined as follows E[ M Q | I ] =

∑

n∈Q

E[ M n | I ]

(2)

|Q| Also, the aggregate load is defined as follows E[ M | I ] =

∑ E[ M

n ∈ network

n

| I]

(3)

We ran the simulation over several topology instances and averaged E[ M | I ] over these trials to calculate E[ E[ M | I ]] = E[ M ] . We came up with 95% confidence intervals for E[ M | I ] . In our experiments, the network size was fixed at 10000 nodes. As mentioned before, the generated network has a power-law topology with the average out-degree of 3.1 and TTL=7. These parameters reflect Gnutella topology specifications which has been used by many researchers so far. For each pair of the super-peers ( s , r ) , the latency distance lat ( s , r ) was generated using a normal distribution with an average μ = 250 ms and a variance δ = 0.1 [12]. Then, to find the pair-wise latency estimation, namely est ( s , r ) , we ran the virtual coordinate service method over the generated topology. The Least Frequency Used (LFU) is a typical frequency-based caching policy which has been proved to be an efficient policy in the area of distributed systems [16]. In LFU, the decision to replace an object from the cache is proportional to the frequency of the references to that object. All objects in the cache maintain the reference count and the object with the smallest reference count will be replaced. The criterion for replacing an object from the cache is computed as follows Cost Object = frequency Object × recency Object

(4)

Where, frequency Objeect and recency Object denote the "access frequency" and the "elapsed time from recent access", respectively. If the cache has enough room, LFU will store the new object in itself. Otherwise, LFU selects a candidate object which has the lowest Cost Object value among all cached objects. Then, LFU will replace the

candidate object by the new object if the Cost Object of the new object is higher than that of the candidate object. Otherwise, no replacement occurs. Figure 3 shows the experimental results concerning the effect of the resource replication on the fraction of participating super-peers, namely F , and the probability of finding objects, namely Pf , versus different broadcasting probabilities. It can be seen from the figure

14


that Pf attains high values for much smaller values of p . By adjusting the broadcasting probability, one can tune the probability of finding the resource. In the case of using resource replication, Pf achieves larger values in comparison with the case in which the resource replication is not used. On the other hand, when we use resource replication method, F achieves smaller values in comparison with the case in which the resource replication is not used. Thus, the behavior of F is not similar to that of Pf . The reason lies in the fact that in the case of using the resource replication method, some intermediate nodes replicate the queries in their local disks (i.e., they cache the queries into their LIT); leading to a decrease in the LITs miss ratio, thus an increase in the probability of finding the queries. Such nodes do not need to propagate the QuerySearch message to other super-peers anymore.

Fig. 3. The effect of resource replication on the fraction of participating peers and the probability of finding objects for various broadcasting probabilities

Figure 4 shows the average number of the required hops to find the resource, namely H , which is normalized by the total number of super-peers (except the original source). The figure shows the effect of the resource replication method in various broadcasting probabilities. It can be seen in both curves of the Fig. 4 that the average number of hops initially increases until reaches to a maximum point and then begins to decrease. A higher broadcasting probability means that the super-peers who are located further away from the original source are contacted and the resource tends to be found further away from the original source. As p continues to increase, the


15

Fig. 4. The effect of resource replication on hop ratio for various broadcasting probabilities

increased values of hit ratio concerning to intermediate DCs allow the resource to be found in locations where are closer to the original source; hence causes a decrease in the value of H . It is clear from the Fig. 4 that the use of resource replication reduces the number of hops needed to find the resource. For example, in a reasonable practical point of broadcasting probability, such as 0.7, it yields a 31% improvement, where the hop ratio decreases from 0.08 to 0.055 Figure 5 shows the effect of resource replication on the total required bandwidth of the system, i.e. the required incoming and outgoing bandwidth of super-peers for various broadcasting probabilities. By increasing the broadcasting probability, some additional costs are imposed to the system. The most important costs include the cost of sending queries to each super-peer, a startup cost for each super-peer as they process the query, and the overhead of additional packet headers for individual query responses. Some of these factors are mentioned in the literature by prior researchers. Interested readers can find useful hints in [13]. The upper curve in Fig. 5 shows the required bandwidth in the absence of the resource replication. In this case, as the broadcasting probability p increases, the required bandwidth of the super-peers increases and reaches to 7.7 × 108 bps for a value of p equal to 0.8. From this point forward, the growing of bandwidth occurs more slightly until reaches to 7.9 × 108bps at the value of p equal to 1. The lower curve in Fig. 5 shows an improvement in the required bandwidth in the presence of the resource replication. In this case, the required bandwidth decreases to 6.6 × 108bps for a value of p equal to 0.8, resulting in a 14% improvement in comparison with the same point in the upper curve.

16


Fig. 5. The effect of resource replication on total bandwidth for various broadcasting probabilities

5 Conclusions In this paper we have targeted the proximity-aware location service for peer-to-peer systems. The proposed protocol provides a scalable distributed caching mechanism to find the peers who manage a given resource and achieves an enhancement by replicating the objects based on the latency distance factor, resulting in less aggregate load over the system. The simulation results showed that using the probabilistic resource discovery service in peer-to-peer systems combined with latency-aware probabilistic resource replication, improves the overall performance of the system in terms of aggregated load, throughput, and the number of the peers who are involved in the search process.

References 1. Rowstron, A., Druschel, P.: Pastry: Scalable, Distributed, Object Location and Routing for Large-Scale Peer-to-Peer Systems. In: Guerraoui, R. (ed.) Middleware 2001. LNCS, vol. 2218, pp. 329–350. Springer, Heidelberg (2001) 2. Dai, L., Cao, Y., Cui, Y., Xue, Y.: On Scalability of Proximity-Aware Peer-to-Peer Streaming. Computer Communications 32(1), 144–153 (2009) 3. Menascé, D.A., Kanchanapalli, L.: Probabilistic Scalable P2P Resource Location Services. ACM Sigmetrics Performance Evaluation Rev. 30(2), 48–58 (2002) 4. Menascé, D.: Scalable P2P Search. IEEE Internet Computing 7(2) (2003) 5. Zhu, Y., Hu, Y.: Efficient, Proximity-Aware Load Balancing for DHT-Based P2P Systems. IEEE Transactions on Parallel and Distributed Systems 16(1), 349–361 (2005)


17

6. Zhu, Y., Li, B., Pu, K.Q.: Dynamic Multicast in Overlay Networks with Linear Capacity Constraints. IEEE Transactions on Parallel and Distributed Systems 20(7), 925–939 (2009) 7. Zhu, Y., Li., B.: Overlay Networks with Linear Capacity Constraints. IEEE Transactions on Parallel and Distributed Systems 19(2), 159–173 (2008) 8. Yang, B., Garcia-Molina, H.: Improving Search in Peer-to-Peer Networks. In: The 22nd International Conference on Distributed Computing Systems (ICDCS 2002), Vienna, Austria (2002) 9. Adamic, L., Lukose, R., Puniyani, A., Huberman, B.: Search in Power-Law Networks (2001), http://www.parc.xerox.com/istl/groups/iea/papers/plsearch/ 10. Zhao, L., Newell, D., Iyer, R., Milekal, R.: Priority Aware Selective Cache Allocation. Patent (2009) 11. Jung, H., Yeom, H.Y.: Efficient Lookup Using Proximity Caching for P2P Networks. In: Proceeding of International Conference on Grid and Cooperative Computing (GCC), Wuhan, China, pp. 567–574 (2004) 12. Jesi, G.P., Montresor, A., Babaoglu, O.: Proximity-Aware Superpeer Overlay Topologies. IEEE Transactions on Network and Service Management (2007) 13. Yang, B., Garcia-Molina, H.: Designing a Super-Peer Network. In: Proc. Int’l Conf. Data Eng. (ICDE), pp. 49–63 (2003) 14. Palmer, C., Steffan, J.: Generating network topologies that obey power laws. In: The GLOBECOM (2000) 15. Yang, B., Garcia-Molina, H.: Comparing Hybrid Peer-to-Peer Systems. In: Proc. 27th Int. Conf. on Very Large Data Bases, Rome (2001) 16. Song, J.W., Park, K.S., Yang, S.B.: An Effective Cooperative Cache Replacement Policy for Mobile P2P Environments. In: Proceeding of IEEE International Conference on Hybrid Information Technology (ICHIT 2006), Korea, vol. 2, pp. 24–30 (2006)

Interactive Visualization System for DES Mohamed S. Asseisah, Hatem M. Bahig, and Sameh S. Daoud Computer Science Division, Department of Mathematics, Faculty of Science, Ain Shams University Cairo, Egypt [email protected]

Abstract. The Data Encryption Standard (DES) is a secret key encryption scheme adopted as standard in the USA in 1977. Most cryptographic courses and textbooks include DES, and its variations. Interaction and visualization are key factors supporting the learning process. We present a dynamic interactive educational system that visualizes DES, and its variations. The aim of the system is to facilitate teaching and learning DES, and its variants for undergraduate and postgraduate students. The system has been used in Ain Shams University – Faculty of Science, in the course “Cryptography”. The analysis of the data emerging from the evaluation study of our system has shown that students found the system attractive and easy to use. On the whole, student interactions within the system helped them to become aware of DES, to conceptualize it, to overcome learning difficulties, and to correct themselves.

1 Introduction The exponential growth of information that characterizes modern age makes the need for learning more important than ever. But the sheer volume of what we have to learn and the speed at which we must learn it can be daunting. Meeting this challenge require new thinking about how we acquire knowledge and skills, and how we deploy learning resources that can keep up with the knowledge growth. Interaction and visualization are key factors supporting the learning process. They support learning while doing tasks [7]. Interaction is a vital part of the learning process and the level of interaction has an impact on the quality of the learning experience. Instructional designers should make learners active participants, not passive spectators in the process. Interaction shifts the instructional focus from the facilitator and materials to the learner, who must actively engage with peers, materials, and the instructor. A review of the literature reveals other reasons for using interactions. It has been shown that higher levels of interaction are associated with improved achievement and positive learning. E-learning [1] is learning or training that is delivered by electronic technology. Visualization is the most cutting-edge e-learning technique. It offers a radical departure from the highly criticized page-turns, drill-and-practice programs, and workbooks online. Visualization software promise to engage learners by making them active participants in real-world problem solving and allowing them to engage in role plays, providing a safe environment for exploration. These promises have captured the attention of the A. An et al. (Eds.): AMT 2010, LNCS 6335, pp. 18–25, 2010. © Springer-Verlag Berlin Heidelberg 2010

Interactive Visualization System for DES

19

instructional designers and their clients. Visualization must be evaluated in the context of the problem-based design. Using technology-based visualization, learners have the opportunity to experiment and to try a variety of strategies in ways that are often not practical or financially feasible in traditional classroom-based simulations. Most educational systems have been designed to simplify understanding the ideas of some main problems or in general overall course materials. There are few visualizing systems or applets [2, 3, 4, 5, 6, 9, 10, 11, 12, 14] to help students to understand cryptographic protocols. To the best of our knowledge, the systems [2, 5, 9, 10, 14] do not include both DES and AES. GRACE [4] includes a simple visualization of how to use DES, one variant of DES, and AES. It does not include any details of DES and AES. The applets in [11] gave a simple visualization of DES and its triply-encrypted variant. Cryptool [6] visualized DES and AES with a fixed message, and key. It does not support the learner to input the message and the key. Also, it does not include any variation of DES. This paper introduces a highly interactive educational system for DES, one of the most two important symmetric-key encryptions. The main features of the system are: 1. Dynamic: the user has ability to input plaintext/ciphertext and the key in different formats. 2. Step-by-step: there are controls for stepping the process back, forth, and restart the whole process at a time. 3. Tracing: it allows learner to see in detail what is happening in encryption and decryption processes. 4. Animation: it has interesting animations and graphics. 5. Standalone descriptive text and voice. Each step has a text and voice to help learner to understand each step of encryption/decryption. 6. Easy to use: the system is easy to use. The programming language used to develop the system is C# 2.0. In this paper we present a visualization of DES. The other part, variants of DES is similar. The paper is organized as follows. Section 2 contains a brief description of DES. Section 3 presents the visualization of DES in some details. Section 4 shows the outcomes when the system has been tested by the students. Finally, conclusions and future work are presented in Section 5.

2 DES The Data Encryption Standard [8, 13] operates on blocks of 64 bits using a secret key that is 56 bits long. The original proposal used a secret key that was 64 bits long. It is widely believed that the removal of these 8 bits from the key was done to make it possible for U.S. government agencies to secretly crack messages. Encryption of a block of the message takes place in 16 stages or rounds. From the input key, sixteen 48 bit keys are generated, one for each round. The block of the message is divided into two halves. The right half is expanded from 32 to 48 bits using fixed table. The result is combined with the subkey for that round using the XOR operation. Using the S-boxes, the 48 resulting bits are then transformed again to 32 bits, which are subsequently permutated again using yet another fixed table. This

20

M.S. Asseisah, H.M. Bahig, and S.S. Daoud

by now, thoroughly shuffled right half is now combined with the left half using the XOR operation. In the next round, this combination is used as the new left half. In real-life applications, the message to be enciphered is of variable size and normally much larger than 64 bits. There are five modes of operation that have been devised to encipher message of any size. These modes are: electronic codebook, cipher block chaining, cipher feedback, output feedback, counter [8].

3 Visualizing DES We have two models: an encryption model and a decryption model. In the encryption model, the system takes plaintext and a key as inputs and returns the ciphertext, After the system launch, the user is presented with three textboxes. The first textbox (on the left) is for inputting the plaintext. The second textbox (in the middle) is for inputting the key. The third textbox (on the right) is for displaying the output of the cipher. There are also some disabled buttons such as an encryption button, see Fig. 1. In order to enable these buttons, the learner should input plaintext and a key into their respective textboxes. The key can be entered in two formats. The first one is in a binary representation with length equals to 64 bits. The other format is text with a length of at least eight characters.

Fig. 1. DES interface before the encryption in details

After this preparation, the learner can do the encryption by one of two ways: 1. 2.

Encrypt the whole plaintext altogether: This is the usual way. By clicking on "Encrypt" button, the learner gets the resulting ciphertext, see Fig. 1. Encrypt the plaintext step-by-step: In this way, the learner can move through plaintext’s blocks (each block contains 8 characters) and stops on the desired block, which should be highlighted by yellow color. By clicking on the “Begin” button, the learner can step through the encryption process in full details, i.e., tracing, of the DES algorithm, see Fig. 1.


21

The main feature of our visualizing system is to trace the encryption/decryption process in a dynamic, interactive and step-by-step way. DES Cipher consists of sixteen rounds. The first fifteen rounds consist of three main functions f, XOR, and Swap, while the last round only consists of f, and XOR. In addition, there are two permutations, one before the first round and another after the last round. The learner can use the “MainPanel”, see Fig. 2, to navigate through these functions and learn in detail about each step in the encryption. In MainPanel, we use gray color to indicate the click-ability of a label. Every button in MainPanel interface takes an input and then passes it inside the animated region. In order to move to the next round or to the previous round, the user can click on the forward or backward button respectively. These buttons are beneath MainPanel, see Fig. 2. Now, we will explain each function in detail.

Fig. 2. MainPanel interface

3.1 f Label This label triggers the most important interface. When a learner clicks on it, a new panel (named f method panel) expands along the left side of the window (beside MainPanel). The f method panel consists of two subpanels. The first one contains four clickable labels: Expansion P-Box method, XOR method, S-Boxes methods, and Straight P-Box methods. And the second subpanel to exhibit the simulation of the current method. • Expansion P-Box Label: it expands the binary input by using a permutation called P-Box permutation, see Fig. 3. • XOR Label: the output of the previous label is added to the round’s key using XOR operation as we can see in Fig. 4. • S-Boxes Label: S-Boxes do an important function in the encryption. So the learner must grasp it very well, for that we have designed a spectacular interface to simulate the process. The interface comprises of eight S-Boxes as we can see in Fig. 5. The output of XOR label (64 bits) is divided into 8 binary strings of length 6. Therefore, each S-Box gets 6 bits as input. Each S-Box’s simulation runs as follows.

22


Fig. 3. f method interface

Fig. 4. XOR interface

i. Get the equivalent decimal value, say r, of the two bits on the sides of the 6-bit string. ii. According to the value of r, a blue arrow is pointed to the row number r in the S-Box, see Fig. 5. iii. Get the equivalent decimal value, say v, of the four bits in the middle of the 6bit string. iv. According to the value of v, a blue arrow is pointed to the column number v in the S-Box, see Fig. 5. v. The two arrows together will intersect in the resulting value, which is highlighted by blue color. vi. Then the resulting value is converted to 4-bit format, which is the final output of the S-Box. Finally, we get 32 bits string as the result of the all S-Boxes. • Straight P-Box Label: A simple permutation is performed on the input, i.e., output of S-Boxes. This permutation is called Straight P-Box permutation, see Fig. 6.


23

Fig. 5. S-Boxes interface

Fig. 6. Straight P-Box interface

3.2 Main XOR Label The main XOR operation is applied on the output of the function f and the half-left of the current round’s input. The visualization of this operation is similar to Fig. 4. 3.3 Swap Region It swaps the output of the previous Main XOR label with the half-right of the round’s input. Thus, the left becomes the right and vice versa. After that, the two halves are passed to the next round as its input, see Fig. 2. 3.4 Initial and Final Permutation Labels Each one of the two permutations takes 64 bits and permutes it according to a predefined table. The initial permutation is executed before the first round, whereas, the final permutation is executed after the last round. The visualization of the initial and final permutation is similar to Fig. 6.

24


3.5 The Key Generation Process The system also visualizes how to generate a key (called a sub-key) of length 56 bits for each round of encryption using binary string input of length 64 bits (called the master key). Each sub-key is used in its corresponding round in the MainPanel, specifically in XOR method which is included in f method. As we can see in Fig. 7, the main operations in each round of the key generation are: split operation, left Shift Operation, and compression P-Box operation. Before the first round, the input should face the Parity Drop permutation, which shrinks the input string (which is presented in binary format) from 64 bits to 56 bits. By clicking on Parity Drop label we can see the visualizing interface of the Parity Drop operation.

Fig. 7. Key Generation interface

4 Evaluation of the System The system has been used for teaching cryptography to undergraduate (forth-year) and pre-master students in Computer Science Division – Department of Mathematics at Faculty of Science, Ain shams University. They were 25 students. The evaluation of the system has been conducted using two steps: the first one was a questionnaire, and the second was an oral exam. The main function of the questionnaire was to show the students’ impression of the system. The results of each question, in terms of percentage response, are shown in Table 1. The oral exam is considered as credible evidence to prove the results concluded from the questionnaire. In fact, the second author makes a face-to-face interview with each student. During this interview, the student is asked some questions. From the answers of the students, we made sure that the answers in the questionnaire are credible.


25

Table 1. Percentages of students response to the system Question • Enhances student’s learning • Increases effectiveness of the course • Has interesting animations and graphics • Has a suitable concept of help guides • Has an easy navigation system

Strongly agree 80

Agree

Neutral

Disagree

15

5

-

Strongly disagree -

87

13

-

-

-

13

69

17

1

-

11

32

50

7

-

20

76

4

-

-

5 Conclusions and Future Work In this paper, an interactive step-by-step visualization system has been presented to support understanding DES. Visualizing variants of DES is similar to DES. This system can be extended to the Advanced Encryption Standard (AES). AES is the winner of the contest, held in 1997 by the US Government, after the DES was found too weak because of its small key size and the technological advancements in processor power. We also intend to extend the system to support cryptanalysis of DES and AES.

References 1. Armstrong, M.: A handbook of human resource management practice (2003) 2. Asseisah, M., Bahig, H.: Visual Exploration of Classical Encryption on the Web. In: The Ninth IASTED International Conference on Web_based Education, March 15-17 (2010) 3. Bishop, D.: Introduction to Cryptography with Java Applets. Jones and Bartlett Publishers, USA (2003) 4. Cattaneo, G., De Santis, A., Ferraro Petrillo, U.: Visualization of cryptographic protocols with GRACE. Journal of Visual Languages and Computing 19, 258–290 (2008) 5. Crytpography demos, http://nsfsecurity.pr.erau.edu/crypto/index.html 6. Cryptool, http://www.cryptool.org 7. Hsi, S., Soloway, E.: Learner-Centered Design: Addressing, Finally, the Inique Needs of Learners. Proceedings of Computer Human Interaction 98, 211–212 (1998) 8. Menezes, A., Van Oorschot, P., Vanstone, S.: Handbook of Applied Cryptography, 2nd edn. CRC Press, Boca Raton (2001) 9. Protoviz, a simple protocol visualization, http://www.cs.chalmers.se/_elm/courses/security 10. RSA demo applet, http://cisnet.baruch.cuny.edu/holowczak/classes/9444/rsademo 11. Schweitzer, D., Baird, L.: The design and use of interactive visualization applets for teaching ciphers. In: Proceedings of the 7th IEEE Workshop on Information Assurance. US Military Academy, West Point (2006) 12. Spillman, R.: A software tool for teaching classical cryptology. In: Proceedings of the 6th National Colloquium on Information System Security Education, Redmond, Washington, USA, 13. Stinson, D.: Crytpography theary and practice. CRC Press, Boca Raton (2004) 14. Zaitseva, J.: TECP Tutorial Environment for Cryptographic Protocols. Master’s thesis, Institute of Computer Science, University of Tartu (2003)

Intelligent Implicit Interface for Wearable Items Suggestion Khan Aasim1, Aslam Muhammad1, and A.M. Martinez-Enriquez2 1 Department of CS & E., U.E.T., Lahore, Pakistan [email protected], [email protected] 2 Department of Computer Science, CINVESTAV-IPN, Mexico [email protected]

Abstract. In daily routine life, people frequently perform computer aided physical activities explicitly shifting from real to virtual world and conversely. In this shift and in order to get some recommendations, people are inquired about personal information. Normally, people do not know how to organize ideas to answer to an automatic inquiry or sometimes, they are reluctant to disclose their own particulars. These issues slow down computer usage to get assistance concerning suggestions about wearable items. We tackle the problem by developing an intelligent interface for helping customers to choose entities like dresses, shoes, and hair style. The system is based on implicit Human Computer Interaction concept and Artificial Intelligence heuristics particularly on knowledge based systems. The developed system gathers customer information like height, weight, waist, skin color, in order to facilitate the selection of daily life commodities. The performance of the system is very encouraging applied for getting suggestions of dress selection for a business man, shoes for professional players, and party hair style for men. Keywords: Implicit HCI, Knowledge based systems, Ubiquitous computing.

1 Introduction In order to escape from stress or unpleasant situations, humans perform certain physical activities, like getting aid from computer resources to perform better and comfortable lives. Nevertheless, there is a gap between the real world and the computer environment. Main reason behind is that to perform automatic assistance for activity recommender system, people have to shift explicitly from visible to virtual computer environment. This shifting not only slows down human’s performance but also reasonably decreases the real usage in performing physical activities. Nowadays, computer applications are provided by limited Human Computer Interface (HCI) which is unaware of physical real environment. Normally, to get some computer assistance, it is required that users provide information by explicit input. The huge gap between physical and virtual environment increases the work load on people in performing computer assisted activities. These issues cover different research areas like context aware computing [3], tangible interaction [4], multi-modal interaction [2]. Moreover, in our society, not many people are aware of using A. An et al. (Eds.): AMT 2010, LNCS 6335, pp. 26–33, 2010. © Springer-Verlag Berlin Heidelberg 2010

Intelligent Implicit Interface for Wearable Items Suggestion

27

computers, they might hesitate to give confidential information or to be marked by tedious repetition. In order to address this problem, we propose an implicit HCI interface that gathers customer’s information seamlessly and then suggests or advises him regarding his getup. The rest of the paper is organized as follows: related work is presented in Section 2. Section 3 describes proposed system in detail. A case study is described in Section 4. In Section 5, conclusions and some future perspectives are given.

2 Related Work When we read an article on British Broadcasting Company (http://www.bbc.com) regarding adjustable undergarments, we had the impression that normally, people feel uncomfortable when their undergarments are not well fitted to them. To overcome it, an “adjustable undergarments” concept is presented, according to which a microcontroller can be used to adjust wearable item to attain customer comfort. Meng, et al [1] introduced a shoe-integrated system for human gait detection: normal gait, toe in, toe out, over supination, and heel walking gait. The inertial measurement unit (IMU) concept is introduced which consists of three-dimensional gyroscopes. To measure angular velocity and acceleration of foot, an accelerometer is used. During the research work on wearable computer and sensor interfaces Yangsheng et al [9] argue that the major benefit provided by wearable intelligent devices is the close proximity that maintain with users. Therefore, the development of an intelligent shoes system consisting of a micro controller, a suite of sensors for acquiring physiological, motion, and force information, as well as a wireless transmitterreceiver set are proposed. The data gathered from this intelligent shoe-integrated platform is used to further analyze real-time health and gait monitoring, real-time motion (activity) identification and real-time user localization. Development about new sensing cloths [7] allowing the simultaneous record of physiological signals, it is used the concept in textile industry by means of sensors integrated into fabrics. The sensing devices gathers user information used for health monitoring providing a direct feedback to user acting on awareness level and allowing better control for user conditions. Availability of sensor technology, ranging from temperature sensor to complex acceleration sensor, for instance Hendrik,, HolgeR, et al [5] present the concept of wearable comfortable clothes, to continually worn supporting users during the work. Touch interaction on mobile phone is a natural HCI. But, occlusion problem is presented, i.e. a “big finger” may lose high percentage of information presented on a small screen [6]. Up to the best of our knowledge, no such system exists which facilitates customer to select wearable items.

3 A Wearable Items Recommender System The developed infrastructure is composed of hardware like screen immiscible environment and pointing devices, functionalities for satisfying requirements like friendly interface, control mechanisms for providing information request from the interface,

28

K. Aasim, A. Muhammad, and A.M. Martinez-Enriquez

relationships of several tools and work process. Our system consists of five subsystems (see Fig. 1): 3.1 Information Gathering (IG) This system gets user information for the first time, before passing through the magical room. There are different ways to accomplish this step: 1) An operator inquires and records user's preference in order to promote implicit HCI, 2) Dedicated functionality for information gathering where user answers questions related with his preferences, interacting with speaker and mike, 3) Multi-mode information (manual and plus HCI). The automatic mode system is especially for technical users who have IT knowledge and who can access Internet. User information is retrieved from Internet rather than inquired him, for each available website source, when user has a web site or may have a Facebook account / any public profile application. This information can be gathered from based on tags.

Fig. 1. System Information Flow

IG saves users attributes like gender, age, address, facebook name, height, and some other. Website or Facebook username can be used to implicitly capture user’s data and some fields like favorite celebrity and profession. Some of these attributes are explicitly asked and others are implicitly captured by the system like user’s skin color, height. Implicit captured information come from camera and is updated by the “Information Refinement & Profile Update” component. 3.2 Event Category Selection (ECS) Users select the event category for which getup is chosen. The category can be business meeting, wedding dresses, sportswear, casual, etc. as shown in Figure 2. For each type of category a button appears aligned horizontally. User selects event category so that system can show him information according to his occasional match when he reaches the Magical Selection. In order to go to the magical room user has to pass through this door button. By default door is closed and is opened when user presses on. After pressing, the door is opened like automatic doors, i.e. from inside to outside horizontally. The category that user has selected is shown as a default category while he is in magical room.


29

Fig. 2. Category Selection

3.3 Information Refinement and Profile Update (IR&PU) This is one of implicit process of the system, because user information is gathered without involving him physically to feed the data to update the user’s profile. When user walks toward the category selection door, a revolving camera takes users information. The camera is mounted on a reeling to move back and forth. The camera is mounted via circular surface which helps the camera to revolve 360 degrees. When user is walking towards the door this camera revolves around user and gathers facial's user characteristics. This information is used for getup selection purposes, in magical room. This step is very important as have to process on user’s latest information. The system uses Trace transform [8] which is a generalization of the Radon transform. The technique helps to recognize objects under transformations. 3.4 Magical Selection System (MS) MS is composed by the following modules: a) Gait detection The walking styles are deduced by our walking device since different walking styles exist due to kind of work performed, place, status, a particular used freestyle walking, etc. While walking people, they place their feet on different angles. In addition, they have different physical characteristics like large or flat footed. The walking device contains pressure sensors which are laid on a metallic surface. Customer walks on this device bare footed, giving the force of different part of his feet to sensor that detect and get user gait information. The walking devise is provided by different lights beneath: green color when nobody is walking on the device, blue when someone is walking on it and the system is getting information without any error, and red color when someone is walking and the system is not getting the correctly input. Red color conveys the message to person to again walk on the device. A voice message is also sent through speakers saying “Repeat again”. b) Camera system Although, we have gathered user’s facial information during Information Refinement step. So this is an optional system. The camera is placed in a magical room that can rotate 360 degrees. Camera is used when user is not satisfied with quality of earlier taken pictures or when he wants to view this getup in a new style. We place this system again in the magical room/cabin, such that user does not need to go back and new photographs can be captured for information refinement.

30

c)

K. Aasim, A. Muhammad, and A.M. Martinez-Enriquez

Screen

Other used hardware is a computer screen on which information is being displayed, whose main advantage is a touch screen which enables users to have a more natural interaction with the device. But sometimes occlusion of information may be presented, becoming a disadvantage. In order to provide usable interaction with the touch systems, we introduce a new concept of “Grip able Needle”, similar as a pen but with a feature that it can grip user’s finger. When user puts his finger inside this needle and the finger reaches a certain depth and touches the inner side of the needle, the two grips move and hold users finger. Thus, “Grip able Needle” sticks together user’s finger and acts as unique. Now user does not need to use three fingers to hold the needle. User can now easily interact with touch systems. More over different sizes needles are placed at the side of touch screen, so user can choose a needle according to his finger thickness. Other interesting part of “Computer Screen” is that user appearance is viewed on. Thus, he has the possibility to provide feedback and all controls are also placed. In addition, a big screen can be used, on which these filters are done, showing users view. In this way, users view themselves live during the selection of particular things. Thus, he does not need to go to a mirror or someone who makes comments whether he is all right. User can view himself on the big screen and decides what follow is on. d) Dedicated Functionality The dedicate functionality works on the top of knowledge base which consists of rules written in first order logic. The user information constitutes the premise part of a rule and system for suggestion/recommendation forms the action part. 3.5 Order Placement System (OP) When user presses “Place Order” button either from the touch screen or from keyboard, this process is launched. First the system gathers all the items that the user has added to the cart during this trip. When all selected items are available, the bill is generated. When an item is not available, a searching is triggered within the store or in other outlets. When an item is available at later time, the system generates the bill, marking the booked time and fixes the user’s address to be delivered.

4 Case Study Let’s consider as a case study, the business man, who comes to getup for a meeting. A customer has gone all previous steps and now he is going to use our currently software process. We consider that dress is the most important article. Thus, dresses are shown at first by default. This sequence can also be changed. Note that filters are already populated with specific values. Gender, age and height are populated from businessman’s profile information. The category drop down is populated from businessman information selected during event category selection step. When businessman’s mind changes, he can also select items from different category, providing flexibility since, at same time, users can shop items from multiple categories. Items, stuff, and brand information is populated based on user’s history.


31

Fig. 3. Dress Selection

For instance, if user has selected suits most of times and cotton stuff, then these values are automatically populated. Anyhow, user can change this information to have different items and stuffs, at any time. The screen also contains a navigation pane which contains several types of links. Businessman has the possibility to organize views by clicking on the appropriate link: - Newly arrived stuff, - Particular style like plain, check, and lining, - Most ordered, - Most viewed, -Preferred color. When user has selected it, the system only displays items contained selected color. Let’s consider that the businessman has selected particular brand cotton as stuff and newly arrived as selected category (see Figure 3) to the businessman. Customer can try different suits and view himself on the big screen. Now after viewing different suits, he can select a particular color; now suits with that particular color are shown to user. He can add the selected suit to the compare cart by clicking “Compare” button. Now, let’s suppose that user has changed the color option and want to view suits in different color in same variety. After viewing a couple of suits he has finalized a particular suit. Now, he wants to compare it with the previously selected suit. He can click the “Comparison Mode” option. All the selected suits are shown to user; user can select all or some of them for comparison. He can select one and leave them on comparison cart or can delete them from comparison cart to minimize the selection options. If user has selected a particular suit he can click buy button to add it in the final cart. When user wants a recommendation from system, he clicks on “suggest suit” link. Our system uses a knowledge based system to recommend customers suitable clothing according with a selected social client's event. An example of rule is as follows: StartRule "Winter Dress Suit Suggestion" If nature (x) = customer /* ‘x’ is a customer */ color (x) = "fair" /* customer color is fair */ favoriteColor(x) = “black” /* customer favorite is black */ newFashionColor(selected_item) = “brown” Then Suggest(x) 0 , the aggregate demand for commodity k exceeds the aggregate endowment of commodity k , and so there is excess demand for commodity k . When z k ( P (t )) < 0 , there is excess supply of the commodity k . A Walrasian equilibrium * * price vector P (t ) >> 0 clears all markets. That is Z ( P (t )) = 0 . Since the utility functions and production possibility sets satisfy the conditions of "Existence Theorem", the existence of the Walrasian equilibrium price will be guaranteed. Interested readers can refer to pages 210-211 of [14] to see the proof of this theory.

5 Experimental Analysis In order to control the competitive overlay economy we consider a dedicated server which is responsible for calculating the existing equilibrium price and informing all nodes about it. We name this server Overlay Market Control Server (OMCS). In fact, the OMCS bears the characteristics of "rendezvous point" in protocols such as [6, 7, 15]. In such protocols upon joining the overlay network, the new node sends a request to the rendezvous point and then acquires from the rendezvous point the IP address of its assigned parent in the multicast tree. Upon joining or leaving of the nodes, the OMCS solves N equations of the Eq. (18) with N unknown equilibrium prices to find the new equilibrium price, P (t ) , and broadcasts it to the all nodes i ∈ V by proper messages. Due to space limitations, we omit the messaging details of the project here. *

Towards Microeconomic Resources Allocation in Overlay Networks

227

The minimum and the maximum allowed bandwidths of the service are defined to be 500Kbps and 4Mbps, respectively. Each user i , on joining the network is endowed with an initial income ei , which has Poisson distribution with mean 400Kbps. The time between the two consecutive "joining" events is uniformly distributed in the interval [0, 200 msec]. Similarly, the inter-arrival time of the "leaving" events has uniform distribution in the interval [0, 2 sec]. We have used BRITE topology generator [16] to create the backbone of the underlying physical network which consists of 512 routers and 1024 physical links. The bandwidth of each physical link has Heavytailed distribution in the interval [10Mbps, 100Mbps] and the propagation delay has uniform distribution in the interval [1ms, 2ms]. Each overlay node is connected to a backbone router through an access link, whose capacity is exponentially distributed with an average of 15 Mbps. We have assumed that the downlink capacity of each access link is two times bigger than its uplink capacity. Also, the maximum tolerable loss rate and the maximum tolerable delay of each overlay flow are %5 and 1 second, respectively. The number of services in the competitive overlay economy is 5. So, there exist 5 origin servers in the economy. In order to evaluate the resilience of the proposed algorithm against the network dynamics, we ran our simulations with dynamic networks in which the nodes join and leave according to the given distributions. Fig. 2 shows the average throughput of each node resulted from the microeconomics-based resource allocation mechanism compared to the case in which no priced-based mechanism is used. In the non-price mechanism, the algorithms of the joining and the leaving are designed similar to those of the price-based case, but they do not contemplate the calculation of the equilibrium price at all. For the sake of completeness, a comparison with the average upper bound throughput is provided in Fig. 2 as well. By "average upper bound throughput", we mean the average uploading capacity of non-leaf nodes in all multicast trees. Clearly,

Fig. 2. Average throughput per node (average social welfare)

228


Fig. 3. Percentage of improvements by the microeconomic-based mechanism

the aggregate receiving rate of the overlay nodes in an overlay tree cannot exceed the sum of the upload capacities of the non-leaf nodes. So, we can gain further insights into the microeconomics-based mechanism by evaluating it with "the average upper bound throughput" as a best-case metric. Clearly, the metrics related to the upper bound throughput vary with topological form of the multicast trees at any given moment. From the "First Welfare Theorem" it is guaranteed that every WEA is Pareto optimal in the sense that the average social welfare of the economy is maximized. Interested readers can refer to pages 218-219 of [14] to see the proof of this theory. It is evident from Fig. 2 that the resultant average social welfare of the proposed mechanism is better than the case in which no price-based mechanism is used. Fig. 3 illustrates the percentage of the nodes who their utility have been improved using the microeconomics-based mechanism during the network dynamics. To this end, for each population we have logged acquired utility of each user in both cases of using the microeconomics-based and the non-priced mechanisms and then have compared these two values with each other. Next, we have normalized the number of improved users by the total number of the users and have stated the result in the form of the percentage of improvements. It is clear from the figure that using the proposed mechanism enhances the perceived quality of the services in the multicast groups. Fig. 4 shows the price of each service during network dynamics. In order to allocate the demanded services to the new users, the proposed algorithms at first seek the upper levels of each multicast tree. Thus, the resultant multicast trees typically have near-balanced shapes. For the sake of illustration, let us consider the case in which each multicast tree has the structure of a balanced tree. Let K denote the maximum n

number of allowed children for each user node, and Vleaf denote the number of the leaf nodes in the n -th multicast tree. Then, we have [17] V ( K − 1) + 1 n Vleaf = K

(19)


229

Since we have assumed K = 4 in the experimental setup, it follows from Eq. (19) 3V + 1 n . In other words, when each user is allowed to have four children, that Vleaf = 4 approximately 75 percent of the users will be leaf nodes. Clearly, the leaf users only consume the services and do not supply any services at all. Therefore, since the total number of the consumers in the overlay economy is potentially greater than the total number of the producers, we expect the prices to increase during the time.

Fig. 4. The price of each service during network dynamics

From the Fig. 4, we can see that the prices of the last rounds increase much slower than those of the early rounds. The economic intuition underlying it is clear. As the time passes, the nodes who are located in the upper levels of the tree earn more profit by selling the services to their children. The earned profit can be added to the current budget of each node and improve its ability for buying the services from its parents. So, the upper level nodes will reach to a state in which they will have enough budgets to buy all of their demanded services from their parents. In other words, as the time passes the upper level nodes no longer have budget constraint and can demand more service compared to the early rounds. This in turn allows the parents to share their uploading capacity more than before and increase their supply. With respect to (17), when the gap between the supply and the demand decreases, Z ( P ) increases with smaller slope than before. This acts as a barrier against increasing the prices and causes the price vector not to rise rapidly in the last rounds. Another factor that significantly affects the price vector is the amount of the initial endowments in the society. The term

∑ ei

k

in the right-hand side of Eq. (17) represents the aggregated

i∈I

amount of the initial endowments for a given service k . This value increases proportional to the ever increasing number of the users; leading to slow increase of Z ( P ) ,

230


and therefore the price vector. Another reason that causes the price vector to rise slowly in the ultimate rounds of the system lies in the nature of the perfectly competitive market system. As the number of the consumers and the producers become sufficiently large, no single one of them, alone, has the power to significantly affect the market price. As it is evident from the Fig. 4 the prices of the services 1 and 2 rise with larger slope compared to that of the services 3, 4, and 5. The reason lies in the fact that in the experimental setup we have assigned the most relative importance to the services 1 and 2 ( β 1 = 0.45, β 2 = 0.40) . So, as is shown in Fig. 5, the consumers will try to buy the services 1 and 2 in higher quantities compared to the other services. This results in the more excess demand and the more displacement of the prices for these two services.

Fig. 5. The rate of each service during network dynamics

6 Conclusions In this paper we have viewed the overlay multicast network as a competitive exchange economy. The proposed algorithms manage the provisioning of multiple services and allocate the bandwidth to the users in multi-rate fashion, with a goal of maximizing the aggregate utility of the system. To this end, the algorithms try to regulate the price of each service in such a way that the demand of each service becomes equal to its supply. Our experiments have proved the efficiency of the system and have shown that it works near-optimal and allocates the resources in such a way that the Pareto optimality is met.


231

References 1. Zhu, Y., Li, B.K., Pu, Q.: Dynamic Multicast in Overlay Networks with Linear Capacity Constraints. IEEE Transactions on Parallel and Distributed Systems (2009) (to appear) 2. Zhu, Y., Li, B.: Overlay Networks with Linear Capacity Constraints. IEEE Transactions on Parallel and Distributed Systems 19(2), 159–173 (2008) 3. Cui, Y., Xue, Y., Nahrstedt, K.: Optimal Resource Allocation in Overlay Multicast. IEEE Transactions on Parallel and Distributed Systems 17(8), 808–823 (2006) 4. Tran, D.A., Hua, K.A., Do, T.: ZIGZAG: An Efficient Peer-To-Peer Scheme for Media Streaming. In: Proc. of IEEE INFOCOM 2003, San Franciso, CA, USA (2003) 5. Chu, Y.H., Rao, S., Seshan, G.S., Zhang, H.: A case for end system multicast. IEEE J. on Selected Areas in Communications 20(8) (2002) 6. Banerjee, S., Bhattacharjee, B., Kommareddy, C.: Scalable Application Layer Multicast. In: Proc. of ACM SIGCOMM 2002, Pittsburgh, PA, USA (2002) 7. Castro, M., Druschel, P., Kermarrec, A.-M., Rowstron, A.: SCRIBE: a large-scale and decentralized application-level multicast infrastructure. IEEE J. on Selected Areas in Communications 20(8) (2002) 8. Roughgarden, T., Tardos, É.: How bad is selfish routing? J. ACM 49(2), 236–259 (2002) 9. Wu, C., Li, B.: Strategies of Conflict in Coexisting Streaming Overlays. In: INFOCOM 2007, pp. 481–489 (2007) 10. Levin, D., LaCurts, K., Spring, N., Bhattacharjee, B.: Bittorrent is an auction: analyzing and improving bittorrent’s incentives. In: SIGCOMM, pp. 243–254 (2008) 11. Wu, C., Li, B., Li, Z.: Dynamic Bandwidth Auctions in Multioverlay P2P Streaming with Network Coding. IEEE Trans. Parallel Distrib. Syst. 19(6), 806–820 (2008) 12. Wang, W., Li, B.: Market-Based Self-Optimization for Autonomic Service Overlay Networks. IEEE J. on Selected Areas in Communications 23(12), 2320–2332 (2005) 13. Bertsekas, D.: Nonlinear Programming, 2nd edn. Athena Scientific, Belmont (1999) 14. Jehle, G.A., Reny, P.J.: Advanced Microeconomic Theory. Addison-Wesley, Reading (2001) 15. Pendarakis, D., Shi, S.Y., Verma, D., Waldvogel, M.: ALMI: an application layer multicast. In: 3rd USENIX Symp. on Internet Technologies and Systems (2001) 16. Medina, A., Lakhina, A., Matta, I., Byers, J.: BRITE: An Approach to Universal Topology Generation. In: Proc. IEEE Int’l Symp. Modeling, Analysis and Simulation of Computer and Telecomm. Systems, MASCOTS (2001) 17. Horowitz, E., Sahni, S., Mehta, D.: Fundamentals of Data Structures in C++. W.H. Freeman Press, New York (1995)

Investigating Perceptions of a Location-Based Annotation System* Huynh Nhu Hop Quach1, Khasfariyati Razikin1, Dion Hoe-Lian Goh1, Thi Nhu Quynh Kim1, Tan Phat Pham1, Yin-Leng Theng1, Ee-Peng Lim2, Chew Hung Chang3, Kalyani Chatterjea3, and Aixin Sun4 1

Wee Kim Wee School of Communication & Information, Nanyang Technological University {hnhquach,khasfariyati,ashlgoh,ktnq,tppham,tyltheng}@ntu.edu.sg 2 School of Information Systems, Singapore Management University [email protected] 3 National Institute of Education, Nanyang Technological University {chewhung.chang,kalyani.c}@nie.edu.sg 4 School of Computer Engineering, Nanyang Technological University [email protected]

Abstract. We introduce MobiTOP, a Web-based system for organizing and retrieving hierarchical location-based annotations. Each annotation contains multimedia content (such as text, images, video) associated with a location, and users are able to annotate existing annotations to an arbitrary depth, in effect creating a hierarchy. An evaluation was conducted on a group of potential users to ascertain their perceptions of the usability of the application. The results were generally positive and the majority of the participants saw MobiTOP as a useful platform to share location-based information. We conclude with implications of our work and opportunities for future research.

1 Introduction In recent years, various location-based annotation systems [2, 5, 7, 8] have popularized the use of maps for people to create and share geospatial content. Put differently, a location-based annotation system allows users to create and share multimedia content that are typically associated with latitude-longitude coordinates using a map-based visualisation. As an information sharing platform, location-based annotation systems could facilitate the users' needs in information discovery by the availability of searching and browsing features [20]. Also, in the spirit of social computing, such systems could also allow users' to create annotation as well annotating existing content [1]. Threads of discussion or topics that are organised hierarchically are then induced from the collaborative effort. Despite the growing amount of research in this area, to the best of our knowledge, there are few studies done to investigate the usability of these applications. We argue that this is critical in the understanding how users perceive these applications and *

This work is partly funded by A*STAR grant 062 130 0057.

A. An et al. (Eds.): AMT 2010, LNCS 6335, pp. 232–242, 2010. © Springer-Verlag Berlin Heidelberg 2010

Investigating Perceptions of a Location-Based Annotation System

233

their constituent features. This will help in the design and implementation of locationbased annotation systems. In this paper, we investigate the usability of MobiTOP (Mobile Tagging of Objects and People). As its name suggests, the application supports location-based tagging or annotating. MobiTOP offers a Web-based platform where users are able to freely create, contribute, and comment on location-based content. The application also enables users to explore, search and browse annotations using a variety of techniques. In previous work, we have conducted a small-scale pilot evaluation of MobiTOP [9]. While useful in guiding the development of further iterations of the system, the results were not generalizable due to the small number of participants involved. Here, we complement the previous study by involving a larger number of participants. The remainder of this paper is as follows. Section 2 provides an overview of the related research while Section 3 introduces MobiTOP, a location-based annotation system that we have implemented. Section 4 presents evaluation of the system. This paper closes with Section 5 that discusses the implications of our work and opportunities for future research.

2 Related Work Here, we review literature related to location-based annotation systems. One such system is the World Explorer [4] where the users are used to explore and browse large-scale georeferenced photo collections. Using spatial, textual and photographic data mined from Flickr, the system visualizes the most representative tags of a geographical area. This visualization improves users’ exploring and browsing experiences. However, World Explorer does not provide any search function that allows users to look for specific tags. Moreover, users of World Explorer are unable to share or discuss their contents directly on the system. GeoAnnotator [3], on the other hand, facilitates location-based discussion threads by connecting annotations to geographic references and other annotations. However, users are limited to sharing only textual content and this functionality is not extended to other types of content such as multimedia content. Urban Tapestries [6], is another system that allows users to share their location-based multimedia contents. Moreover, this system also allows users to follow a discussion thread as a hierarchical content. However, there is no usability study done on the system’s map interface and the annotation visualization. There are limited usability studies related to location-based annotation systems. Komarkova et al [19] proposed a set of 138 heuristics for usability evaluation of location-based applications. In that study, 14 GeoWeb applications were used to test this framework. The usability of such systems has been evaluated and criticized by a group of expert users. Despite the fact that major online web-mapping systems such as Google Maps or Microsoft Live Search have been significantly improved regarding their usability, there are up to now limited usability evaluations by the end-users of such systems. Studies [19, 21, 22, 23] have found evaluating the usability of applications directly by the end-users to be more promising.

3 Introducing the Web-Based MobiTOP System MobiTOP has been introduced in our previous work [9], which described carefully the architecture of the whole system as well as the concept of multimedia hierarchical

234

H.N.H. Quach et al.

annotation. The latest version of Web-based MobiTOP provides more functions such as identification, organization, searching and visualization of location-based content. Moreover, using Google Maps™ API for representing the MobiTOP’s user interface, these functions has been organised consistently in the web application. In this section, we describe the web user interface of MobiTOP as well as explore its functionality. The MobiTOP Web client offers an AJAX-based user interface to facilitate its widespread use without the need to installing additional software. We have adopted a map-based visualization to access the location-based annotations (Figure 1). An important component of MobiTOP is its support for hierarchical multimedia annotations that allow users to annotate existing annotations, essentially creating a thread of discussion. Here, annotations consist of locations, images and other multimedia, as well as textual details augmented by tags, titles and descriptions. The content of an annotation is displayed across two columns in MobiTOP (Figure 1). One column displays the hierarchical view of the selected annotation while the other column displays the annotation’s content. The content itself is divided among various tabs and consists of the annotation’s details, tag cloud, and media attachments. MobiTOP’s functionality may be divided into seven main components: • • •

Registration: Before a user can start to contribute and browse the annotations in MobiTOP, an account needs to be registered. A registered user would be able to view the main interface of MobiTOP (Figure 1) after logging in. Map navigation: MobiTOP provides standard features for map navigation such as zooming and panning. Users are also able to reposition the map to a specific area by entering the address in the search bar. Browsing annotations: Users are able to access annotations through various ways. One of these is by using the View menu at the top left corner of the screen (Figure 1). This menu encapsulates the different annotation access features such as viewing all the annotations in the system, the user’s contributed annotations, recently contributed annotations, and the tag cloud generated from all Search bar

View menu functions

Filtering feature

Tabs to various content

List of annotations

Hierarchy of related annotations

$QQRWDWLRQ¶V detailed content

Map

Markers

Fig. 1. User interface of the MobiTOP Web client


•

• •

235

annotations. These functions enable the users to make serendipitous information discovery. Another way for the user to browse the annotations is by navigating the tree view that is displayed in the individual annotation’s details. Searching annotations: Users are able to search desired annotations by entering relevant keywords in the search bar. However, retrieved annotations could clutter the map and impede the searching process if too many results are returned [14, 17]. We overcome this problem by clustering the results. Here, the annotations in the search results are grouped based on their locations (Figure 2). The clustering algorithm is an adaptation of DBScan [12] that groups the annotations by density. The novelty of this approach is that the clustering results vary between different zoom levels depending on the distance between the annotations. The numbers on each marker on the map in Figure 2 shows the numbers of annotations in the cluster. In addition, a tag cloud of each cluster is also shown to the user. Users are thus able to explore individual annotations in each cluster by clicking on the tag cloud (Figure 3). Further, users are able to search without clustering the resulting annotations. Filtering annotations: In addition to clustering, filtering the annotations to narrow search results is also supported. Here, options are available to narrow the results by distance, user ratings and by time. (Figure 1). Creating annotations: When creating a new annotation, a user enters title, tags, description and attaches relevant multimedia files (Figure 4). We attempt to alleviate the problem of noisy tags [18] as well as to save users’ time and effort in keying in tags [10] by providing tag recommendations (Figure 4). The tags are recommended based on the location of the annotation [11], its parents’ tags and the owners’ contributed tags thus far. Given the current location of the user, the

Summary of individual clusters

Clusters on map

Fig. 2. Clustered search results list are displayed on left panel and on the map

236

H.N.H. Quach et al.

Current cluster tag cloud

List of annotations in current cluster

Individual annotations in current cluster

Fig. 3. Interface of the clustered the search results showing the annotations of a cluster

List of recommended tags

Fig. 4. Creating an annotation and list of recommended tags


237

Fig. 5. Editing an annotation textual content as well as its attachments

•

algorithm first aggregates the tags that had been used in the surrounding location. Each of the tags that had been used in the surrounding location is given a score based on the frequency of its usage. We go further to make distinctions between the number of times the tag is used by the current user and other annotation creators. This is to maintain the preference of the current user over other owners. The tag’s score is also determined by how recently the tag was used. Again, we made distinction between the current user and other owners. Finally, the top ten tags with the highest score are recommended to the user. Editing/deleting annotations: Users are only able to edit or delete the annotations that they had created. The edit form (Figure 5) provides functions that are similar with that to create annotations. Users are also able to edit the textual content, add or delete multimedia files.

4 Usability Evaluation A study of the MobiTOP’s Web user interface was conducted to determine its usability. A total of 106 participants took part in the evaluation. There were 57 male and 49 female participants, and they were students and working adults. Their ages ranged from 18 to 37, with an average age of 23. Further, 38% of the participants had a computer science background while the rest had other academic backgrounds. Participants were familiar with the use of social computing applications such as blogs, wikis, photo/video sharing, social tagging and social networking. Here, 65% of the participants reported to view such content at least once a week, while 55% of the participants reported to contribute such content at least once a month.

238

H.N.H. Quach et al.

4.1 Methodology During the one-hour evaluation session, participants were first briefed on the concept of annotations and were introduced to seven components of Web-based MobiTOP as describe in Section 3. After that, the short demonstration was provided to the participants to show them how to perform some basic tasks directly on the web application. Right after the introduction and demonstration, the travel planning scenario together with fifteen tasks were assigned to each of the participants in order to evaluate the user interface. The tasks focused on using the seven components of MobiTOP as described in Section 3 in order to plan and share travelling trips through the system. Research assistants were on hand to clarify doubts that the participants had while doing the tasks. After completion of their tasks, participants were required to complete a questionnaire with demographic questions and those related to their perceptions of the usability of the MobiTOP system. Each question was given in the form of the affirmative statement followed by a scale of 1 (Strongly Disagree) to 5 (Strongly Agree). The usability section of the questionnaire was further divided into two parts. The first sought to determine MobiTOP’s overall usability via four indicators [13]: • • • •

Learnability: measures how easily the users learn to navigate the system and complete a task. Efficiency: determines the users’ ability to complete a task within a reasonable amount of time. Error Handling: verifies the users understanding of the error encountered and the ability to recover from the errors. Satisfaction: validates the users’ sense of satisfaction after completing the tasks and intention to adopt the system.

The second part of the questionnaire focused on the usability of each component, and questions were asked about the ease of use of the respective features. Participants were also encouraged to elaborate in their evaluation by answering three subjective questions about which components they liked or disliked, as well as suggestions on useful features that could be included in future versions of MobiTOP. 4.2 Results Table 1 shows the mean and standard deviation (SD) of MobiTOP’s overall usability with respect to the four indicators. Results indicate that MobiTOP was perceived to be relatively usable in general. For instance, during the study, most participants were observed to be able to complete the tasks after the short briefing, suggesting the learnability of the system (“It is easy to learn using the application”- Participant 2). In addition, the efficiency indicator suggested that participants took a reasonable amount of time to complete their tasks. It was observed that all of the participants were able to complete the tasks within the specified amount of time. Further, participants generally knew the meaning of error messages encountered and were able recover from the errors without seeking help from the research assistants. Finally, they appeared to have enjoyed creating and sharing annotations with others, and most of them felt satisfied after completing the tasks (“I can get a lot of information if users upload their experiences to MobiTOP… It is an interesting system”- Participant 37).


239

Table 1. Overall usability results (1 = strongly disagree; 5 = strongly agree) Usability Indicators Learnability Efficiency Error Handling Satisfaction

Mean 4.08 3.72 3.60 3.84

S. D. 0.36 0.54 0.59 0.51

In addition to overall usability, Table 2 shows the mean and standard deviation of the usability of each of MobiTOP’s seven major features. The results indicate that participants found the individual components to be usable as well: •

•

•

•

User Registration. Overall, all participants knew how to register for a MobiTOP account without any trouble. They found the registration form to be intuitive and this could be due to their familiarity with other Web applications’ registration components. In addition, all participants knew how to handle registration errors. Annotation Navigation. All the features of the View menu were appreciated by the participants. For instance, Participant 4 commented that “viewing my annotations helps me to conveniently keep track of all my uploaded annotation”. Participant 29 found viewing annotations of a particular user to be useful as it “allows me to search for their friends’ activities easily”. Similarly, Participant 7 found tag cloud function to be convenient: “I don't have to think about the word I want to search, it’s great”. Finally, Participant 45 liked the viewing of recent annotations as it “allows me to quickly update the information in the system”. On the other hand, the “View All Annotations” feature received less positive responses compared to the others. One likely reason was that most users could not access their preferred annotations are among the large number of annotations in the system. In summary, the participants were able to learn how to use the features in the View menu without any difficulty and most of them found that this component greatly helped the way they accessed the annotations. Map Navigation. Most of the participants felt comfortable in browsing the map. Additionally, they felt that the map-based interface was quite intuitive. The reason could be their familiarity with Google Maps as 70% of the participants used Web-based mapping applications at least once a week. The way of representing annotations on the map was also well received by the participants (“It’s quite easy to navigate the map and explore annotations through pop-up windows”-Participant 98). Creating Annotations. The participants found that the annotations were easy to create because of the simplicity and responsiveness of the interface. As Participant 46 remarked: “The speed of uploading the annotations amazes me. It doesn't take more than a minute to add a new annotation. It's simple to attach a picture too”. Although the concept of hierarchical annotations was new to most of the participants, they were able to create sub-annotations easily. Perhaps the tree view visualization provided them with the proper mental model to understand the concept of hierarchical annotations. The participants also realized the advantages in organizing the annotations hierarchically. This sentiment was echoed by Participant 6 who felt that creating sub-annotations was easy as “we

240

•

•

•

H.N.H. Quach et al.

usually share our experiences in the some similar topics with others. The tree structure let us to conveniently organize the information”. Editing/Deleting Annotations. Participants found editing an existing annotation to be easy and useful as they were able to provide updated information to their contributions. They also found deleting their annotations to be easy (“The … delete annotations (was) made simple and easy” – Participant 63). Searching Annotations. Participants found that the search without clustering feature was easy to use as the results were ordered by relevance and organized across pages. Participant 44 found that the “searching feature is easy to use and it helps me to find the information I need”. On the other hand, presenting the search results in clusters was a new concept to some users. However, most participants managed to complete the tasks related to this function. A common sentiment shared by the participants was that clustering helped to reduce information overload. Participant 10 sums this nicely: “Clusters are neatly organized and the tag cloud of each cluster helps in the searching process”. However, there were comments on the unresponsiveness of searching with clustering. This was because of the processing time required by the clustering algorithm. Filtering Annotations. Most of the participants agreed that being able to filter the annotations by different attributes was helpful in discovering information and at the same time reduces information overload. Participant 106 commented that “it is a handy tool to narrow down the information from the large results list”. Table 2. Components’ usability results (1 = strongly disagree; 5 = strongly agree) Components Usability Registration Annotation Navigation Map Navigation Creating Annotation Editing/Deleting Annotation Searching Annotation Filtering Annotation

Mean 4.22 4.19 3.99 4.11 4.13 3.82 4.17

S. D. 0.51 0.48 0.61 0.52 0.63 0.56 0.55

5 Discussion and Conclusion In this paper, a usability evaluation was conducted with the goal of ascertaining the usability of MobiTOP, a location-based annotation system. The overall usability was found to be above-average by the 106 participants. This is despite the fact that there were new concepts being utilized to support information access such as hierarchical annotations and clustering of search results. Moreover, observations during the evaluation showed that participants needed very little training to be able to use the system. Arising from our results, the following are some implications for the design of location-based annotation systems: • Using familiar visualizations to represent new concepts helps users to orientate themselves more quickly to the system. We have adopted the tree view to represent the hierarchical aspect of annotations and the map-based visualization to represent the annotations. As these visualizations provide the relevant mental


•

•

•

241

model for users to map the respective concept with the visualization, users are more likely to find the application easy to use, as demonstrated by our results. As with any information system, searching is an essential component. Additionally for a location-based system, searching is often tied to a specific location and is visualized on a map. However, presenting results as individual annotations on the map may overwhelm the user especially when there are large numbers of annotations returned. As such, clustering results on the map should be considered to alleviate the information overload. Provide filtering functions that are based on the different attributes of the data model. As our annotations are contributed by the users, having a mechanism that distinguishes the more useful annotations from the less useful ones would benefit the users. In terms of geo-spatial attributes, narrowing the radii of search focuses the users to the relevant area of interest. Finally, being able to filter the annotations by time attributes narrows the annotations to the relevant time period. Finally, eliminate the need for the user to manually input the data by providing recommendations. For instance, in MobiTOP, relevant tags are suggested to the user when while creating an annotation. The users of course have the freedom to make their selections. This reduces the mental effort needed to create annotations, thus improving users’ perceptions of the usability of the application.

There are limitations in our study that could be addressed in future work. First, our clustering algorithm is limited to geo-spatial locations. Perhaps clustering the annotations semantically [15] in addition to location would further help users obtain relevant content. Next, our evaluation was cross-sectional in nature and confined to the use of MobiTOP in a single session. Further work could track the usability and usefulness of the system over a longer period of time. Finally, memorability was not considered as a usability indicator. It would be interesting for future work to investigate this aspect of MobiTOP [16].

References [1] Kim, T.N.Q., Razikin, K., Goh, D.H.-L., Theng, Y.L., Nguyen, Q.M., Lim, E.P., Sun, A., Chang, C.H., Chatterjea, K.: Exploring hierarchically organized georeferenced multimedia annotations in the MobiTOP system. In: Proceedings of the 6th International Conference on Information Technology: New Generations, pp. 1355–1360 (2009) [2] Leclerc, Y.G., Reddy, M., Iverson, L., Eriksen, M.: The GeoWeb—A New Paradigm for Finding Data on the Web. In: Proceedings of the International Cartographic Conference, Beijing, China (2001) [3] Yu, B., Cai, G.: Facilitate Participatory Decision-Making in Local Communities through Map-Based Online Discussion. In: Proceedings of the Fourth International Conference on Communities and Technologies, pp. 215–224 (2009) [4] Ahern, S., Naaman, M., Nair, R., Yang, J.: World Explorer: Visualizing Aggregate Data from Unstructured Text in Geo-Referenced Collections. In: Proceedings of the JCDL Conference, pp. 1–10 (2007) [5] Girardin, F., Blat, J., Nova, N.: Tracing the Visitor’s Eye: Using Explicitly Disclosed Location Information for Urban Analysis. IEEE Pervasive Computing 6(3), 55 (2007) [6] Lane, G.: Urban Tapestries: Wireless Networking, Public Authoring and Social Knowledge. Personal Ubiquitous Computing 7(3-4), 169–175 (2003)

242

H.N.H. Quach et al.

[7] Doyle, S., Dodge, M., Smith, A.: The potential of Web-based mapping and virtual reality technologies for modeling urban environments. Computer, Environment and Urban System 22, 137–155 (1998) [8] Friedl, M.A., McGwire, K.C., Star, J.L.: MAPWD: An interactive mapping tool for accessing geo-referenced data set. Computers and Geosciences 15, 1203–1219 (1989) [9] Razikin, K., Goh, D.H.-L., Theng, Y.L., Nguyen, Q.M., Kim, T.N.Q., Lim, E.-P., Chang, C.H., Chatterjea, K., Sun, A.: Sharing mobile multimedia annotations to support inquirybased learning using MobiTOP. In: Liu, J., Wu, J., Yao, Y., Nishida, T. (eds.) AMT 2009. LNCS, vol. 5820, pp. 171–182. Springer, Heidelberg (2009) [10] Naaman, M., Nair, R.: ZoneTag’s Collaborative Tag Suggestions: What is This Person Doing in My Phone. IEEE Multimedia 15(3), 34–40 (2009) [11] Moxley, E., Kleban, J., Manjunath, B.S.: SpiritTagger: A Geo-Aware Tag Suggestion Tool Mined from Flickr. In: Proceeding of the 1st ACM International Conference on Multimedia Information Retrieval, pp. 24–30 (2008) [12] Ester, M., Kriegel, H.-P., Sander, J., Xu, X.: A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise. In: Proceedings of the KDD 1996, pp. 226–231. AAAI, Menlo Park (1996) [13] Nielson, J.: Usability Engineering. Morgan Kaufmann, San Diego (1993) [14] Jaffe, A., Naaman, M., Tassa, T., Davis, M.: Generating Summaries and Visualization for Large Collections of Geo-Referenced Photographs. In: Proceedings of the Multimedia Information Retrieval, pp. 89–98 (2006) [15] Cutting, D.R., Karger, D.R., Pederson, J.O., Tukey, J.W.: Scatter/Gather: a cluster-based approach to browsing large document collections. In: Proceedings of the 15th Annual International ACM SIGIR Conference, pp. 318–329 (1992) [16] Hornbæk, K.: Current practice in measuring usability: Challenges to usability studies and research. International Journal of Human-Computer Studies 64, 79–102 (2006) [17] Nguyen, Q.M., Kim, T.N.Q., Goh, D.H.-L., Theng, Y.L., Lim, E.P., Sun, A., Chang, C.H., Chatterjea, K.: TagNSearch: Searching and Navigating Geo-referenced Collections of Photographs. In: Christensen-Dalsgaard, B., Castelli, D., Ammitzbøll Jurik, B., Lippincott, J. (eds.) ECDL 2008. LNCS, vol. 5173, pp. 50–61. Springer, Heidelberg (2008) [18] Goh, D.H.-L., Lee, C.S., Chua, A.Y.K., Razikin, K.: Resource Discovery through Social Tagging: A Classification and Content Analytic Approach. Online Information Review 33(3), 568–583 (2009) [19] Komarkova, J., Novak, M., Bilkova, R., Visek, O., Valenta, Z.: Heuristic Evaluation of Usability of GeoWeb Sites. In: Abramowicz, W. (ed.) BIS 2007. LNCS, vol. 4439, pp. 411–423. Springer, Heidelberg (2007) [20] Lim, E.-P., Liu, Z., Yin, M., Goh, D.H.-L., Theng, Y.-L., Ng, W.K.: On organizing and accessing geospatial and georeferenced Web resources using the G-Portal system. Information Process & Management 41(5), 1277–1297 (2005) [21] Nielsen, J.: Useit.com: Technology Transfer of Heuristic Evaluation and Usability Inspection (2005), http://www.useit.com/papers/heuristic/learning_inspection.ht ml [cit 2010-01-22] [22] Haklay, M., Tobon, C.: Usability evaluation and PPGIS: toward a user-centered design approach. International Journal of Geographical Information Science 17, 577–592 (2003) [23] Haklay, M., Zafiri, A.: Usability Engineering for GIS: Learning from a screenshot. The Cartographic Journal 45(2), 87–97 (2008)

Apollon13: A Training System for Emergency Situations in a Piano Performance Yuki Yokoyama and Kazushi Nishimoto Japan Advanced Institute of Science and Technology 1-1, Asahidai, Nomi, Ishikawa, 923-1292, Japan {y_yoko,knishi}@jaist.ac.jp

Abstract. During a piano performance, there is always the possibility that the musician will cease playing on account of an unexpected mistake. In a concert, such a situation amounts to an emergency state in the piano performance. Therefore, we propose a system named “Apollon13” that simulates emergency states by replacing particular notes with different ones, in the manner of mistouches, by referring to the performer’s degree of proficiency as determined by a performance estimation algorithm. From the results of user studies, we confirmed that Apollon13 is basically effective as a training system for handling emergency states. However, the estimation algorithm could not precisely identify the note-replacement points where the subjects become upset. Accordingly, we evaluated the estimation algorithm by comparing it with the player’s subjective assessment based on the data of an experiment. As a result, we found a clear relationship between the subjective assessment and the points, obtained by experiment, at which players become upset. This result suggests that an algorithm could gain the ability to detect the “upset points” by approximating a human’s subjective assessment. Keywords: emergency training, performance estimation, piano performance, note-replacement.

1 Introduction This paper proposes a novel piano-performance training system named “Apollon13.” This system aims to foster the ability to avoid performance cessation caused by unexpected mistakes such as mis-touches. Performance cessation, where the performer “freezes up,” is a “fatal situation” in a piano concert. Therefore, the performer must avoid such a situation by any means, and the performance must go on despite whether mistakes occur. However, no countermeasures to this situation have been taught in conventional piano lessons, and there is no active training methodology for avoiding performance cessation. A piano lesson usually consists of several steps. The first step is basic training. In basic training, an educand learns the method of reading scores and trains in fingerings using some etude (e.g., HANON). The second step is building a repertoire. This step is further divided into two sub-steps. The first is partial exercise and the second is full A. An et al. (Eds.): AMT 2010, LNCS 6335, pp. 243–254, 2010. © Springer-Verlag Berlin Heidelberg 2010

244

Y. Yokoyama and K. Nishimoto

exercise. In this step, the educand learns musical performance and musical expression. Although the educand can build a repertoire in these steps, the educand is not yet able to train for a piano concert. Generally, the way to train one’s skills toward a piano concert is simply to repeat a full performance again and again after memorizing the score and fingering. However, this way of training cannot develop the educand’s ability to cope with an unexpected accident: The only way to accomplish this is to actually perform in concerts. Obviously, it is impossible for typical educands to use concerts for training. Various piano-performance training systems have been developed [1][2][3]. However, these systems have only supported the users in becoming able to perform a musical piece accurately in accordance with its score. The problem of performance cessation during a concert has been completely out of the scope of such systems. Consequently, there has been no active ways to train performers in avoiding performance cessation. In the aerospace field, astronauts and pilots spend much time in training. Of course they learn how to control aircraft and spaceships under normal conditions. However, to accomplish a mission safely, it is much more important to know how to deal with abnormal emergency situations quickly and effectively. For this purpose, in this field, they conduct training for emergency situations using simulators. We introduce such a situational training concept to piano-performance training. Apollon13 simulates unexpected mistakes as emergency situations. Using Apollon13 in the final stage of exercises before a concert, it is expected that the educand can acquire the ability to avoid the worst result, i.e. performance cessation. There has been no training method or training system against performance cessation up to now. Therefore, we believe that our attempt has very high levels of novelty and utility.

2 How to Simulate Emergency Situations How to simulate emergency states was important in designing Apollon13. While there are many causes of emergency states, we focused on mis-touches in performance. A mis-touch results in an unexpected sound, which makes the player upset and, in the worst case, leads to performance cessation. To induce a similar situation, Apollon13 replaces a few of the performed notes with different notes. By trying to keep playing even when the output notes are different from his/her intended notes, the player would be able to learn how to recover from mis-touches without falling into performance cessation. It’s important to understand that the note replacement function should be used only in the final stage where the player is repeating the full exercise, in contrast conventional piano-lesson support systems are used in the initial stage. Musicians use various feedbacks in playing musical instruments. In particular, they are alert to auditory feedback. The proposed system’s note replacement intentionally breaks our auditory sense. In the initial stage of a piano lesson, however, auditory feedback is a fundamental element. Therefore, the note-replacement function must not be used in the initial stage of a piano lesson.

Apollon13: A Training System for Emergency Situations in a Piano Performance

245

Previous literature [4] demonstrated that note replacement has the effect of disorienting piano performance. However, although a keyboard with the notereplacement function is used in this research, the objective of this research is to formulate a kind of stuttering model. Therefore, the way of note replacement in the earlier work is factitious, since such mis-touches never happen in real piano performances. To adopt note replacement in piano practice, a note-replacement method that simulates realistic mis-touches is required. To simulate such realistic mistouches, there are two factors that should be considered: which performed note should be replaced, and by which note. In section 3, we describe the employed simulation method.

3 System Setup 3.1 Overview Apollon13 is a MIDI (Musical Instrument Digital Interface) based system that consists of a MIDI-keyboard, a personal computer, and a MIDI sound module. Apollon13 has two operation modes: a practice-monitoring mode and a rehearsal mode (Table 1). In the practice-monitoring mode, the system tracks and records the user’s full piano performances. In using this mode, the user repeats the full performance of a musical piece many times. A score-tracking function (described in 3.2) compares each performance with the score and records how accurately it is performed. When the practice-monitoring mode is finished, the system decides which notes should be replaced. Too many replacements would become an excess burden for the user. Therefore, the system finds only a few notes where the user would surely become upset by note replacement based on the performance estimation results using the recorded tracking data (described in 3.3). We call such a selected note a “replacing-point” hereafter. In the rehearsal mode, the system tracks the user’s performance again. When the user performs the replacing-point, the system replaces this note with another note neighboring the correct note. This is done because an actual mis-touch in piano performances follows such a pattern. Table 1. Operation mode of proposed system Practice monitoring mode System

Score tracking Recording performance

User

Repeat of full performance

→ Decision of notereplacement part

Rehearsal mode Score tracking Note replacement Continue performing even if mis-touches are simulated

3.2 Score Tracking A score-tracking technology is necessary to obtain performance data for performance estimation. Apollon13 utilizes the score-tracking function of “Family Ensemble” (FE)

246


[5]. FE is a piano-duo support system for a novice child and his/her parent who is an amateur at piano. Since FE’s score-tracking function is robust, it is applicable to tracking performances that include mistakes. We modified FE’s score-tracking function in two points. First, FE’s original score-tracking function tracks only the highest notes at each place. We modified it to polyphony compatible by simply summing all note numbers of the notes in the chord and regarding the sum as the note number of the chord. Second, FE outputs three tracking data: performed position in the score, whether the performed note is correct or incorrect, and timestamp of each performed note. We further added velocity data to represent the loudness of each note for the performance estimation. 3.3 Performance Estimation The aim of performance estimation is to find where the user would surely become upset by note replacement. The performance-estimation algorithm classifies each score-event (i.e., (a) note-on event(s) at the same instant in FE’s score data) into four categories. The criterion of estimation is performance stability. If the performance of a score-event is highly stable throughout all performances in the practice-monitoring mode, the score-event is estimated as “skillful.” If the performance of a score-event is not so stable, it is estimated as “poor.” If the performance of a score-event becomes stable, it is estimated as “improved.” The other score-events are estimated as “other.” 3.3.1 Factors Used for Performance Estimation Previous related studies [2][6] used three factors for the performance estimation, i.e., IOI (Inter Onset Interval), duration, and velocity. On the other hand, we use three factors obtained from the score-tracking function, i.e., IOI (calculated by the received timestamps of the performed score-events), velocity of each score-event, and data on whether performed score-event is correct or erroneous (CE-data, hereafter). Mukai et al. used deviation of IOI for estimating the performance: If the deviation value of the same fingering pattern is large, this pattern is estimated as poorly stable [7]. We also use deviation of IOI as well as that of velocity. However, we calculate the deviations of each score-event in all of the full performances, while Mukai et al. calculated those of the same fingering patterns. Fluctuation in the overall tempo of each performance influences the deviation of tempo at each score-event. To cancel this effect, we calculated normalized local tempo at each score-event. First, average tempo of each entire performance is calculated. Then, the normalized local tempo is calculated by dividing each local tempo at each score-event by the average tempo of the performance. Here, the note value of each score-event is necessary to calculate the normalized local tempo; therefore, we added the note value data to the score data of FE. 3.3.2 Classification of Each Score-Event The performance estimation requires at least three sessions, each of which should include at least ten full performances. This algorithm classifies each score-event based on the deviations (stability) with progress of the sessions as follows:


1.

Calculating “coarse score” A)

B)

C)

D)

Calculating “tempo score”: First, the deviation of normalized local tempo at each score-event for all performances in all practice sessions is calculated. Then, all of the score-events are sorted based on their deviation value. Finally, the 30% of score-events with the smallest deviation values score 2 points, the 30% of score-events with the largest deviation values score 0 point, and the remaining score-events with moderate deviation values score 1 point. Calculating “velocity score”: First, the deviation of velocity at each scoreevent for all performances in all practice sessions is calculated. Then, all score-events are sorted based on their deviation value. Finally, the 30% of score-events with the smallest deviation values score 2 points, the 30% of score-events with the largest deviation values score 0 point, and the remaining score-events with moderate deviation values score 1 point. Calculating “accuracy score”: First, the accuracy rate of each score-event for each practice session is calculated based on CE-data. Then, the transition of accuracy rate for each score-event through all practice sessions is obtained from the regression line of the accuracy rates. Finally, one-third of the score-events, having the highest gradient values of the regression lines, score 2 points, one-third of the score-events, having the lowest gradient values, score 0 point, and the remaining one-third of score-events with moderate gradient values score 1 point. Coarse score is calculated by the following equation:

Coarse score = tempo score * 5 + velocity score * 3 + accuracy score *2 2.

247

(1)

Calculating “adjustment score” A)

B)

C)

Calculating “fine tempo score”: First, the deviation of normalized local tempo at each score-event for all performances in each practice session is calculated. Then, the transition of deviation for each score-event through all practice sessions is obtained from the regression line of the tempo deviations. Finally, one-third of the score-events with the lowest gradient values of the regression lines score 1 point, one-third of the score-events with the highest gradient values score -1 point, and the remaining onethird of score-events with moderate gradient values score 0 point. Calculating “fine velocity score”: First, the deviation of velocity at each score-event for all performances in each practice session is calculated. Then, the transition of deviation for each score-event through all practice sessions is obtained from the regression line of the velocity deviations. Finally, one-third of the score-events with the lowest gradient values of the regression lines score 1 point, one-third of the score-events with the highest gradient values score -1 point, and the remaining one-third of score-events with moderate gradient values score 0 point. Adjustment score is calculated by the following equation:

248


Adjustment score = fine tempo score + fine velocity score 3.

(2)

Classifying each note into one of four categories (skillful, improved, poor and other) based on the coarse score and the adjustment score. Table 2 shows the classifying rules. Table 2. Classifying rule of performance estimation value Skillful part Improved part

Poor part Other part

coarse score >= 15 coarse score < 15 and adjustment score > 0 or coarse score < 5 and adjustment score = 2 coarse score < 5 or coarse score < 15 and adjustment score < 0 coarse score < 15 and adjustment score = 0

4 Experiments After users continuously use Apollon13, if they eventually lose the tendency to become upset when certain notes are suddenly replaced by incorrect ones, we can say that it is an effective training system for emergency situations in piano performance. In this experiment, we investigate the effects of training with Apollon13 by analyzing the users’ subjective assessments and their performance data. 4.1 Experimental Settings and Procedures We conducted experiments with three subjects, who were 23–24-year-old males. They have 18–20 years experience of playing the piano. We prepared a compulsory musical piece “Merry Christmas Mr. Lawrence” composed by Ryuichi Sakamoto, which has only a two-page score. It takes about two minutes to perform it. We selected this piece since it is not too difficult but not so easy to perform by one hand. The subjects received the score one week before the experiments. We asked them to practice one week freely to finish the partial exercise stage. We confirmed that the subjects could play through it before the experiments. Table 3 shows the equipment used in the experiments. The experimental period was five days, and two sessions were held each day: ten sessions in total for each subject. A session takes about thirty minutes. In a session, each subject was required to perform the compulsory piece ten times or more. The interval between the sessions was at least five hours. The first five sessions were assigned as “practice sessions.” In these sessions, Apollon13 works in practicemonitoring mode. The remaining five sessions were assigned as “rehearsal sessions.” In these sessions, Apollon13 works in rehearsal mode. In one rehearsal session, we enabled Apollon13’s note-replacement function in about five randomly selected performances.


249

Table 3. Equipment used in experiments MIDI-keyboard MIDI sound source MIDI-IO PC

YAMAHA grand piano C5L + silent ensemble professional model YAMAHA MU128 Midiman MIDISPORT2×2 Notebook type, CPU: Core2Duo T7250 2.00 GHz, memory: 1.0 GB

The number of replacing-points in one performance was four, and these were selected according to the results of the performance estimation obtained in the practice sessions. At present, although the performance estimation algorithm works, it cannot decide in which category the users definitely become upset. Therefore, in this experiment, the system chooses one replacing-point for each category classified by the performance-estimation algorithm (four points in total) to collect data for validating the performance-estimation algorithm. At the end of each practice session, we asked the subjects to indicate, note by note, where they could skillfully perform and where they could not. At the end of each rehearsal session, we asked the subjects where they became upset. 4.2 Results Figure 1 shows the transition of the ratio of the number of replacing-points where each subject became upset to the number of all replacing-points in each session. The horizontal axis indicates the rehearsal sessions and the vertical axis indicates the ratio. Thus, as the sessions progressed, the subjects gradually came to avoid getting upset by the note replacement. To investigate the effect of note replacement in detail, we analyzed fluctuations in the performances before and after the replacing-points. For this analysis, we first prepared the target performances (with note replacement) and the baseline performances (without note replacement). We employed the performances in the 4th and 5th practice sessions as the baseline performances: average IOI and velocity of each score-event of the baseline performances were calculated as the baseline data. On the other hand, we prepared two target performances: “R1-3” target performances consist of the performances where the note-replacing function was activated in the 1st to 3rd rehearsal sessions, and “R3-5” target performances consist of those in the 3rd to 5th rehearsal sessions. We also calculated average IOI and velocity of each scoreevent of R1-3 and R3-5. Finally, the difference values of seven score-events before and after each replacing-point were calculated (namely, the difference values at the 15 points in total, including the replacing-point, were obtained). Figure 2 shows an example of the obtained average difference values between subject A’s R1-3 performances and his baseline performances at a certain replacingpoint. The horizontal axis indicates the score events. The 8th event corresponds to the replacing-point. The vertical axis indicates the average difference. If the performance becomes disordered by the replaced note at the 8th event, the graph becomes undulant after there. Therefore, we compared deviation of the performance data before and after the 8th score-event by F-test. As a result, we found a significant difference in IOI

250


Fig. 1. Transition of percentage of upset points

Fig. 2. Difference in performance around a replacing-point

of subject A’s R1-3 (p> 1 + T1 >> 3)

(11)

The variance σ 2 and mean μ2 are defined as, 12

12

i =5

i =5

σ 2 = ∑ Pi − μ2 and μ2 = ( ∑ Pi ) >> 3 where μ2 is the mean value of block boundary pixels of top and top-right blocks. Table 3. Binary representation of modes of case 2 Mode Vertical Horizontal Diagonal-down-right Horizontal-up

Binary representation 00 01 10 11

(12)

368

M.G. Sarwer and Q.M.J. Wu

The flow diagram of the proposed ANM method is presented in Fig. 3. The variance σ1 and threshold T1 are calculated at the start of the mode decision process and if

the variance is less than the threshold ( σ1 < T1 ) only DC prediction mode is used. In this case computational expensive RDO process is skipped and a lot of computations are saved. In addition, no bit is necessary to represent intra prediction mode because only one mode is used. On the other hand, if σ1 < T1 is not satisfied, encoder calcu-

lates the variance σ 2 and threshold T2 . If σ 2 < T2 , vertical, horizontal, diagonaldown-right and horizontal-up modes are used as candidate modes in RDO process. A substantial saving in computations is achieved using 4 prediction modes instead of 9 modes of the original RDO process. In order to represent the best mode, 2 bits are sent to the decoder and Table 3 shows the four prediction modes with corresponding binary representations. As shown in Table 3, if the diagonal-down-right mode is selected as the best mode, the encoder sends “10” to the decoder. In this category, only 2 bits are used to represent the intra prediction mode whereas 3 bits are used in the original encoder. Consequently a large number of intra prediction mode bits are saved. If σ 2 < T2 is not satisfied, nine prediction modes are used as the candidate mode and one of them is selected through the RDO process, as in H.264/AVC. The new prediction mode numbers are recorded and compared against H.264/AVC in Table 4.

Fig. 3. Flow diagram of proposed ANM method

Since 8x8 intra prediction also uses 9 prediction modes, the proposed ANM method is also applied to 8x8 intra prediction mode. Assume Pi is the i-th reconstructed pixel of Fig. 1 (c). Variances and thresholds of a 8x8 block are defied as

Enhanced Intra Coding of H.264/AVC Advanced Video Coding Standard 8

16

24

i =1

i =9

i =17

μ1′8 x8 = ( ∑ Pi + ( ∑ Pi ) > 5 24

24

i =9

i =9

24

and σ18 x 8 = ∑ Pi − μ18 x 8 ′ i =1

369

(13)

μ28 x8 = ( ∑ Pi ) >> 4 and σ 28 x8 = ∑ Pi − μ28 x8

(14)

⎧ 2QP + 24 if QP ≤ 24 T18 x 8 = ⎨ ⎩10QP − 180 Otherwise

(15)

and T28 x 8 = (T18 x8 >> 1 + T18 x8 >> 3)

Table 4. Prediction modes recording of the proposed method Mode

Mode number H.264/AVC

Mode number Proposed

Diagonal-down-left Vertical-right Horizontal-down Vertical-left Vertical Horizontal DC Diagonal-down-right Horizontal-up

3 5 6 7 0 1 2 4 8

0 1 2 3 4 5 6 7 8

4 Simulation Results To evaluate the performance of the proposed method, JM 12.4 [7] reference software is used in simulation. Simulation conditions are (a) QPs are 28, 36, 40, 44 (b) entropy coding: CABAC (c) RDO on (d) frame rate: 30 fps and (e) number of frames: 100. The comparison results are produced and tabulated based on the average difference in the total encoding ( ΔT1 %) and decoding ( ΔT2 %) time, the average PSNR differences ( ΔP ), and the average bit rate difference ( ΔR% ). PSNR and bit rate differences are calculated according to the numerical averages between RD curves [8]. The encoding ( ΔT1 %) and decoding ( ΔT2 %) complexity is measured as follows

ΔT1 = ΔT2 =

T penc − Toenc Toenc T pdec − Todec Todec

× 100%

(16)

× 100%

(17)

where, Toenc and Todec are the total encoding and decoding time of the JM 12.4 encoder, respectively. Tpenc and Tpdec are the total encoding and decoding time of the proposed method, respectively.

370

M.G. Sarwer and Q.M.J. Wu Table 5(a). RD performances of proposed methods ( only 4x4 modes, All I frames) DWP[4]

Grand Mother (QCIF) Sales man (QCIF) Stefan (QCIF) Carphone (QCIF) Silent (CIF) Hall (CIF) Mobile Calendar (HD-1280x720) Average

SMP [6]

ΔP 0.04

ΔR % -1.4

ΔP 0.37

ΔR % -15.4

Proposed IDCP only ΔP ΔR % 0.11 -4.6

Prop ANM only ΔP ΔR % 0.41 -16.4

ANM + IDCP ΔP 0.47

ΔR % -18.0

0.02

-0.2

0.32

-12.9

0.12

-6.6

0.39

-13.5

0.42

-18.9

0.01 0.04

-0.2 -1.0

0.10 0.66

-2.7 -18.4

0.09 0.07

-1.3 -2.1

0.19 0.79

-5.8 -22.3

0.21 0.84

-5.6 -20.7

0.02 0.02 0.03

-1.0 -0.5 -2.4

0.35 0.32 0.19

-15.4 -8.6 -6.8

0.07 0.10 0.06

-2.4 -2.6 -2.7

0.40 0.37 0.25

-17.3 -9.8 -9.3

0.45 0.46 0.26

-19.1 -11.2 -10.1

0.03

-0.96

0.33

-11.5

0.09

-3.19

0.40

-13.5

0.44

-14.8

4.1 Experiments with 4x4 Intra Modes Only

In this experiment all frames are intra coded and only 4x4 mode is enabled. The performance comparisons are presented in Table 5. In these tables, a positive value indicates increment and a negative value represents decrement. As shown in Table 5(a), the proposed IDCP improves 0.09 dB PSNR and reduces bit rate by 3.19% whereas DWP improves PSNR by only 0.03 dB and reduces bit rate by only 0.96% of the original encoder. In terms of computation, the proposed IDCP increases the encoding and decoding time by 4.01% and 2.51%, respectively. In case of SMP method, the average PSNR improvement is about 0.33 dB and average bit rate reduction is about 11.5%. Whereas in our proposed ANM method, the average PSNR improvement is about 0.40 dB and average bit rate reduction is about 13.5%. The proposed ANM only method also reduces the computation of the original encoder by 41.5%. Although this method introduces some extra computations of the decoder side, the simulation results Table 5(b). Complexity comparisons of proposed methods (only 4x4 modes, All I frames) DWP[4]


SMP [6]

Proposed IDCP only

Prop ANM only

ANM+IDCP

ΔT1

ΔT2

ΔT1

ΔT2

ΔT1

ΔT2

ΔT1

ΔT2

ΔT1

ΔT2

% 2.22

% 2.25

% -39.7

% 2.09

% 4.35

% 4.39

% -52.1

% 1.99

% -44.9

% 4.7

1.17

0.04

-31.2

1.19

3.70

1.12

-37.9

0.91

-34.7

2.3

1.01 1.29

0.39 1.18

-17.9 -33.8

0.39 0.42

3.46 4.27

0.54 2.65

-25.4 -46.0

0.33 0.39

-21.0 -43.5

1.1 3.2

1.34 1.35 2.62

0.99 0.35 1.28

-35.8 -38.8 -27.6

1.74 1.45 1.55

4.77 3.16 4.39

2.81 1.71 1.84

-45.8 -48.5 -34.7

1.53 1.18 1.59

-41.6 -46.1 -32.9

3.1 3.2 2.8

1.57

0.93

-32.2

1.27

4.01

2.51

-41.5

1.13

-37.8

2.91

Enhanced Intra Coding of H.264/AVC Advanced Video Coding Standard

371

of Table 5(b) confirm that, the computational overhead of the decoder is very low (about 1.13%). It is shown that if we combine our both methods together, about 14.8% bit rate reduction is achieved along with a 0.44 dB improvement in PSNR in the expense of 2.91% increment of decoding time. The proposed method reduces 37.8% computation of encoder. 4.2 Experiments with All Intra Modes

In this experiment all frames are encoded by intra coding and all intra modes (4x4, 8x8, and 16x16) are enabled. The results are tabulated in Table 6. Here proposed IDCP method is applied in 4x4 block and ANM method is implemented in 4x4 and 8x8 blocks. Since only small amount of MBs are encoded with 16x16 modes, the proposed methods are not implemented in 16x16 mode for computational difficulties. We have seen that the average gain is in the range of 0.37 dB PSNR and 12.2% bit rate saving, with a maximum for sequence Carphone with 0.79 dB and 17.7%. We have seen that the proposed method reduces 31.2% computation of original encoder. The computation increment of decoder side is very low and that is 2.3% on average. Table 6. Experimental results of proposed methods (All I frames, all Intra modes) Sequence

Δ P in dB

ΔR %

ΔT1 %

ΔT2 %


0.33 0.40 0.18 0.79 0.36 0.31 0.20 0.37

-14.2 -13.1 -5.8 -17.7 -12.3 -9.1 -13.1 -12.2

-36.8 -29.0 -17.9 -33.8 -37.1 -34.7 -29.2 -31.2

4.0 1.4 1.1 3.8 3.1 1.3 2.0 2.3

5 Conclusions In this paper, we propose two methods to improve the RD performance of H.264/AVC intra encoder. At first, a distance based improved DC prediction is utilized to better representation of smooth region of sequences. Then a bit rate reduction scheme for representing the intra prediction mode is described. The proposed methods not only improve the RD performance but also reduce the computational complexity of H.264/AVC intra coder.

References 1. ISO/IEC 14496-10, Information Technology-Coding of audio-visual objects-Part:10: Advanced Video Coding. ISO/IEC JTC1/SC29/WG11 (2004) 2. Shiodera, T., Tanizawa, A., Chujoh, T.: Block based extra/inter-polating prediction for intra coding. In: Proc. IEEE ICIP 2007, pp. VI-445–VI-448 (2007)

372

M.G. Sarwer and Q.M.J. Wu

3. Ye, Y., Karczewicz, M.: Improved H.264 Intra coding based on bi-directional intra prediction, directional transform, and adaptive coefficient scanning. In: Proc. IEEE ICIP 2008, pp. 2116–2119 (2008) 4. Yu, S., Gao, Y., Chen, J., Zhou, J.: Distance based weighted prediction for H.264 Intra Coding. In: Proc. ICALIP 2008, pp. 1477–1480 (2008) 5. Wang, L., Po, L.M., Uddin, Y.M.S., Wong, K.M., Li, S.: A novel weighted cross prediction for H.264 intra coding. In: Proc. IEEE ICME 2009, pp. 165–168 (2009) 6. Kim, D.Y., Han, K.H., Lee, Y.L.: Adaptive Single-Multiple Prediction for H.264/AVC Intra Coding. IEEE Trans. on Circuit and System for Video Tech. 20(4), 610–615 (2010) 7. JM reference software 12.4, http://iphome.hhi.de/suehring/tml/download/old_jm/jm12.4.zip 8. Bjontegaard, G.: Calculation of average PSNR differences between RD-curves. Presented at the 13th VCEG-M33 Meeting, Austin, TX (April 2001) 9. Sarwer, M.G., Po, L.M., Wu, J.: Fast Sum of Absolute Transformed Difference based 4x4 Intra Mode Decision of H.264/AVC Video Coding Standard. Journal of Signal Processing: Image Commun. 23(8), 571–580 (2008) 10. Sarwer, M.G., Po, L.M., Wu, J.: Complexity Reduced Mode Selection of H.264/AVC Intra Coding. In: Proceeding on International Conference on Audio, Language and Image Processing (ICALIP 2008), China, pp. 1492–1496 (2008) 11. Sarwer, M.G., Wu, Q.M.J.: Adaptive Variable Block-Size Early motion estimation termination algorithm for H.264/AVC Video Coding Standard. IEEE Trans. Circuit and System for Video Technol. 19(8), 1196–1201 (2009) 12. La, B., Jeong, J., Choe, Y.: Most probable mode-based fast 4 × 4 intra-prediction in H.264/AVC. In: International Conference on Signal Processing and Communication Systems, ICSPCS 2008, pp. 1–4 (2008) 13. Elyousfi, A., Tamtaoui, A., Bouyakhf, E.: A New Fast Intra Prediction Mode Decision Algorithm for H.264/AVC Encoders. Journal of World Academy of Science, Engineering and Technology 27, 1–7 (2007) 14. La, B., Jeong, J., Choe, Y.: Fast 4×4 intra-prediction based on the most probable mode in H.264/AVC. IEICE Electron. Express 5(19), 782–788 (2008) 15. Krishnan, N., Selva Kumar, R.K., Vijayalakshmi, P., Arulmozhi, K.: Adaptive Single Pixel Based Lossless Intra Coding for H.264 / MPEG-4 AVC. In: International Conference on Computational Intelligence and Multimedia Applications (ICCIMA 2007), vol. 3, pp. 63– 67 (2007)

Extracting Protein Sub-cellular Localizations from Literature Hong-Woo Chun1 , Jin-Dong Kim2 , Yun-Soo Choi1 , and Won-Kyung Sung1 1 2

Korea Institute of Science and Technology Information, 335 Gwahangno, Yuseong-gu, Daejeon, 305-806, Republic of Korea Database Center for Life Science, Research Organization of Information and System, Japan {hw.chun,armian,wksung}@kisti.re.kr, [email protected]

Abstract. Protein Sub-cellular Localization (PSL) prediction is an important task for predicting protein functions. Because the sequence-based approach used in the most previous work has focused on prediction of locations for given proteins, it failed to provide useful information for the cases in which single proteins are localized, depending on their states in progress, in several different sub-cellular locations. While it is difficult for the sequence-based approach, it can be tackled by the text-based approach. The proposed approach extracts PSL from literature using Natural Language Processing techniques. We conducted experiments to see how our system performs in identification of evidence sentences and what linguistic features from sentences significantly contribute to the task. This article presents a text-based novel approach to extract PSL relations with their evidence sentences. Evidence sentences will provide indispensable pieces of information that the sequence-based approach cannot supply.

1

Introduction

Analysis of where a protein resides in a cell is an important component to predict genome, transcriptome, and proteome annotation, and many research have tackled to predict the place where a protein is located in. Protein Sub-cellular Localization (PSL) prediction is the name of the task. While most of programs available are applied Machine Learning (ML) techniques using features from the amino acid sequences, Text Mining (TM) and Natural Language Processing (NLP) can also contribute to the task. First, the number of pairs of proteins and their localization reported in papers increases very rapidly. Since experimentally recognizing the PSL is a time consuming and expensive task, it would be of great use if one could identify sentences of localization in published literature which are not known in databases. Second, the sequence-based approach can only predict localization and biologists would like to have further confirmation from relevant publications. Bibliographic links will be indispensable. Third, since we have constructed a corpus with respect to the PSL, it may provide useful features for the sequence-based PSL prediction programs. A. An et al. (Eds.): AMT 2010, LNCS 6335, pp. 373–382, 2010. c Springer-Verlag Berlin Heidelberg 2010

374

H.-W. Chun et al.

There have been several researches for PSL prediction. Most approaches to PSL prediction have employed ML techniques to combine various features. For example, WoLF PSORT [1] employed the k-Nearest Neighbors (kNN) algorithmbased ML to deal with features from amino acid sequences. As an attempt of TM approach to the task, Stapley et al. (2002) and Brady and Shatkay (2008) used text-based features to train Support Vector Machine (SVM)) [2] [3]. Although TM techniques have contributed to the PSL prediction, they took a role to provide useful features to the sequence-based approach in the most previous work. In addition, they have not provided further confirmation from literature that may be a most important role of TM techniques. Such outputs are of intrinsically limited value, because a protein can be located in more than one sub-cellular location. It might be more useful if output contains not only pairs of proteins and sub-cellular locations but also additional information. To analyze such additional conditions, we have to use NLP techniques. Our aim is to develop an extractor that predicts protein sub-cellular localizations for proteins using the state of the arts NLP techniques. The proposed approach provides not only protein sub-cellular localizations but also their evidential sentences that may contain the additional information. In the relation extraction point of view, we have regarded the task as a relation extract between proteins and sub-cellular locations.

2

Methodology

This section describes a novel relation extraction approach using NLP techniques. To begin with the explanation of the relation extraction method, we will describe construction of gold standard data for the PSL, and explain features that used in an ML techniques. 2.1

Construction of Gold Standard

A gold standard corpus has been constructed using GENIA corpus [4] as an initial data, because the term annotation task of GENIA corpus has already done by human experts. The term annotation of GENIA corpus contains annotation of Protein and Cellular component that is a candidate of sub-cellular location. Table 1 describes statistical analysis of GENIA corpus version 3.02 with the viewpoint of the PSL. There are three points that should be mentioned to describe the annotation task. The first is that the only sentence-level co-occurrences have been processed in the approach. Thus, the co-occurrence indicates a sentence that contains at least one pair of protein and sub-cellular location names. The second is that the term sub-cellular location is used instead of cellular component in the GENIA corpus as the viewpoint of the PSL, although these cellular components contain not only sub-cellular locations but also other components. The third is that the proteins in the annotation task are the terms annotated by Protein. The category of protein in GENIA ontology contains the following sub-categories: Protein complex, Protein domain or region, Protein family or group, Protein

Extracting Protein Sub-cellular Localizations from Literature

375

Table 1. Statistical analysis of GENIA corpus GENIA corpus Abstracts Sentences Proteins Sub-cellular locations Co-occurrences Unique pairs

Frequency 2,000 18,554 34,265 (unique:8,369) 743 (unique: 225) 863 557

molecule, Protein substructure. In addition, we only considered surface forms annotated as cellular component. We have defined two types of annotation for the PSL. One is classification of sub-cellular location names into the fine-grained sub-cellular locations, and the other is categorization of relations between proteins and sub-cellular locations into the pre-defined three categories. For the task of classification of sub-cellular location, the following 11 fine-grained sub-cellular locations were selected based on Gene Ontology by biologists: Cytoplasm, Cytoskeleton, Endoplasmic reticulum, Extracellular region, Golgi apparatus, Granule, Lysosome, Mitochondria, Nucleus, Peroxisome, Plasma membrane. Among the selected sub-cellular locations, “Extracellular region” is actually not included in the sub-cellular location. Because it is also regarded as an important location we should deal with, the extracellular region was considered as one of the sub-cellular locations in the approach. Table 2 describes the statistical analysis of the classification of subcellular locations. Three human experts have annotated classification of sub-cellular locations with 1.0 of Fleiss’ kappa score that is a statistical measure for assessing the reliability of inter-annotator agreement between a fixed number of annotators. Kappa score 1.0 means all annotation results among annotators are the perfectly same. To calculate the inter-annotator agreement of three annotators, all annotators have annotated the same 31 co-occurrences from 54 MEDLINE abstracts. The kappa score, κ, can be defined as, κ=

P0 − Pc 1 − Pc

(1)

where Po is the proportion of observed agreements and Pc is the proportion of agreements expected by chance [5]. For the task of categorization of relations between proteins and sub-cellular locations, we have decided the following three categories. The notation in examples are followed that of the above example. – Positive assertion: The true relation of a protein and a sub-cellular location names in a co-occurrence that indicated the positive existence of the protein in a certain sub-cellular location. e.g.) We conclude that [p56]P exists in [cytosol]L in a higher order complex

376

H.-W. Chun et al.

containing hsp70 and hsp90, both of which in turn have been found to be associated with untransformed steroid receptors. (PMID: 2378870) – Negative assertion: Another true relation of a protein and a sub-cellular location names in a co-occurrence that indicated the negative existence of the protein in a certain sub-cellular location. e.g.) There is no detectable [NF-AT]P protein expression in the [nuclei]L of resting eosinophils. (PMID: 10384094) – Neutral: Neither positive nor negative relation was indicated, although a protein and a sub-cellular location names occurred together in a sentence (a co-occurrence). In the following example, a relation of the protein Rex and the sub-cellular location cytoplasm describes a neutral relation, and another relation of the protein gag/pol and the sub-cellular location cytoplasm describes a positive relation. e.g.) Monocyte adhesion and receptor cross-linking induced [stress fiber]L assembly, and inhibitors of [myosin light chain kinase]P prevented this response but did not affect receptor clustering. (PMID: 10366600) Table 2. Statistical analysis of newly annotated corpus: classification of cellular components Gold standard Total sub-cellular locations Cytoplasm Cytoskeleton Endoplasmic reticulum Extracellular region Golgi apparatus Granule Lysosome Mitochondria Nucleus Peroxisome Plasma membrane

Frequency 743 184 12 14 1 1 77 3 6 346 5 94

Table 3. Statistical analysis of newly annotated corpus: categorization of relations

Sub-cellular locations Total sub-cellular locations Nucleus Cytoplasm Plasma membrane Granule Lysosome Cytoskeleton

# Relevant Co-occurrences # Irrelevant Co-occurrences (Positive + Negative) Neutral 301 (286 + 15) 562 173 (159 + 14) 233 94 (94 + 0) 189 23 (23 + 0) 77 9 (8 + 1) 47 1 (1 + 0) 5 1 (1 + 0) 11


377

Table 3 describes the numbers of relevant and irrelevant relations between proteins and sub-cellular locations in the relation-categorized corpus. The relevant relations contain positive and negative relations, and the irrelevant relations contain neutral relations. Three human experts have annotated categorization of relations with 0.688 of Fleiss’ kappa score. The kappa score in this annotation can be argued as a substantial agreement by the criterion of Landis and Koch [6]. To calculate the inter-annotator agreement of three annotators, all annotators have annotated the same 203 co-occurrences from 54 MEDLINE abstracts. 2.2

Extraction of Protein Sub-cellular Localizations Using Natural Language Processing Techniques

Maximum Entropy (ME)-based ML technique is used to build the relation extraction system. The ME technique can combine various types of features with maximally unbiased, and it is a general technique to estimate probability distributions of data. Thus, the ME has been widely used to incorporate various features for classification problems in NLP [7]. The ME-based relation extraction method has used various features that are from contexts including co-occurrence information, syntactic analysis. The features considered in the proposed method are as follows: – Frequency of protein and sub-cellular location pairs that appear in GENIA corpus. (1∼2) (1) In the GENIA corpus (2,000 abstracts). (2) In MEDLINE 2007∼8: We checked the frequency of protein and subcellular location pairs in MEDLINE abstracts that were published between 2007(19,051,558 abstracts) to 2008(∼May 16, 2008: 18,623,706 abstracts). Among 557 unique pairs of a protein and a sub-cellular location in GENIA corpus, 122 pairs do not exist in MEDLINE 2007∼8. It may be due to the fact that the GENIA corpus only contains rather old papers (from 1990 to 1999). (3) Protein and sub-cellular location names annotated by human experts. – Adjacent words of protein and sub-cellular location names. (4∼5) (4) Adjacent one words. (5) Adjacent one and two words. (6) Bag of words: All contextual terms in a co-occurrence. (7) Order of candidate names: ProLocRE determined whether or not a protein name appeared before a sub-cellular location name in a co-occurrence. (8) Distance between protein and sub-cellular location names: The number of words between protein and sub-cellular location names. – Features from the syntactic analysis (9∼11) (9) Syntactic category of protein and sub-cellular location names: ProLocRE conducted to parse all co-occurrences using the deep-syntactic parser ENJU ver. 2.1 [8]. ENJU parser provides Part-of-Speech tags of all words and syntactic categories of phrase structures. Syntactic categories

378

H.-W. Chun et al.

We conclude that p56 exists in cytosol in a higher order complex containing N(CD) V(VBZ) P(IN) N(NP)

hsp70 and hsp90, both of which in turn have been found to be associated with untransformed steroid receptors.

Fig. 1. An example for explanation of features from the full parsing results. (Protein: p56, Sub-cellular localization: cytosol).

of protein and sub-cellular location names were used as features. In Figure 1, p56 is a protein name and cytosol is a sub-cellular location name, and their categories are equally Noun phrase (N). (10) Predicates of protein and sub-cellular location names: ENJU parser can analyze deep syntactic and semantic structure of an English sentence and provide predicate-argument relations among words. Predicates of protein and sub-cellular localization names were used as features. In Figure 1, “exists” and “in” are predicates of protein (p56) and sub-cellular location (cytosol) names, respectively. (11) Part-of-Speech of predicates: In Figure 1, “VBZ” and “IN” are Partof-Speech of predicates, exists and in, respectively. VBZ indicates verb that has the 3rd person singular present as its subject, and IN indicates preposition.

3

Experimental Results

In the experiments, we evaluated how well the proposed approach extracts relevant relations. To evaluate the system, we performed a 10-fold cross validation and measured the precision, recall, and F-score of the system for all experiments. All co-occurrences in GENIA corpus are used, and they contain 301 relevant relations and 562 irrelevant relations (Table 3). We conducted two sets of experiments: one set of experiments is to extract relevant relations between proteins and sub-cellular locations without classification of sub-cellular locations, and the other set of experiments is to extract relevant relations between proteins and sub-cellular locations with classification of sub-cellular locations. Both sets of experiments have an assumption in baseline experiments: a pair of protein and sub-cellular location indicates a relevant relation if they occur together in a co-occurrence. 3.1

Extraction of Relevant Relations between Proteins and Sub-cellular Locations without Classification of Sub-cellular Locations

We conducted experiments to extract relevant relations between proteins and sub-cellular locations without using the classification information of sub-cellular


379

Table 4. Performances of protein sub-cellular localization prediction for all cellular components. (# of relations: 301). Features 1 2 3 4 5 6 7 Baseline Experiment Effectiveness of √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √

each √ √ √ √ √ √ √ √ √ √ √

Best experiment √ √ √ √

8

9

feature √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √

√ √ √ √ √ √ √ √

√ √

√ √

√ √ √ √ √ √ √ √ √

√ √ √ √ √

10

√ √ √ √ √ √ √ √ √ √ √ √

Performance 11 Precision Recall F-score

√ √ √ √ √ √ √ √ √ √ √

0.349

1.0

0.517

0.836 0.828 0.830 0.830 0.837 0.825 0.820 0.819 0.808 0.821 0.836 0.831

0.872 0.875 0.875 0.875 0.877 0.865 0.874 0.845 0.863 0.874 0.870 0.872

0.854 0.851 0.852 0.852 0.857 0.844 0.846 0.832 0.835 0.847 0.853 0.851

√

0.837 0.877 0.857 Effectiveness of syntactic analysis (w/o features from syntactic analysis) √ √ √ √ √ √ √ 0.826 0.868 0.846 N ote) TEXT: Text-based approach; SEQUENCE: Sequence-based approach; CO-OCCURRENCE: Co-occurrence-based features; Features from texts: 1∼11, Features from sequences: 12∼13: 1. Frequency of co-occurrence in GENIA corpus; 2. Frequency of co-occurrence in MEDLINE abstracts 2007∼8; 3. Protein and cellular component names; 4. Adjacent one words of names; 5. Adjacent two words of names; 6. Bag of Words; 7. Order; 8. Distance; 9. Syntactic category of protein and sub-cellular localization names; 10. Predicates of protein and sub-cellular localization names; 11. POS of predicates.

locations. Table 4 describes experimental results, and it consists of five types of experiments. The first type of experiments is baseline experiment. The second type of experiments describes the effectiveness of an individual feature for the text-based approach. This type of experiments is to show ablation study that is to describe a set of experiments where leaving out individual features in order test their contribution. All features are used in the first experiment in the second type of experiments. Through the experiments, we found that the feature Order and Distance are the effective features to extract the relevant relations. The third type of experiment describes the best performing experiment, and it used all features except one feature that is adjacent one words of protein and sub-cellular location names. To show the effectiveness of syntactic features, experiment without features from the syntactic analysis was conducted in the fourth type of experiment.

380

H.-W. Chun et al.

Table 5. Performance of protein sub-cellular localization prediction for each finegrained sub-cellular location Sub-cellular # Relevant location relations Nucleus 173 Cytoplasm 94 Plasma membrane 23 Granule 9

3.2

Performance : F-score (Precision, Recall) Baseline w/o Syntactic analysis Best combination 0.334 (0.200, 1.0)) 0.767 (0.742, 0.794) 0.776 (0.748, 0.807) 0.196 (0.109, 1.0) 0.842 (0.812, 0.874) 0.850 (0.821, 0.880) 0.052 (0.027, 1.0) 0.873 (0.821, 0.932) 0.911 (0.857, 0.973) 0.017 (0.009, 1.0) 0.915 (0.878, 0.956) 0.926 (0.880, 0.978)

Extraction of Relevant Relations between Proteins and Sub-cellular Locations with Classification of Sub-cellular Locations

We conducted experiments to extract relevant relations between proteins and sub-cellular locations using the classification information of sub-cellular locations. Therefore, this set of experiments focus on the more fine-grained relations compared with the first set of experiments. Three types of experiments have been compared based on four fine-grained sub-cellular locations in Table 5. The three types of experiments are baseline, experiments without using features from the syntactic analysis, experiment using best combination in Table 4. In the baseline experiments, we assumed that all pairs in 863 co-occurrences indicate relevant relations for the corresponding fine-grained sub-cellular locations. The next two types of experiments used the same combination of each type of experiment in the Table 4. The four finegrained sub-cellular locations have been selected when the number of relevant relations is more than one. These experiments showed that the features from the syntactic analysis contributed to improve performance for all fine-grained sub-cellular locations. , and frequency-based features also played an important role in improving performance in three of four fine-grained sub-cellular locations.

4 4.1

Discussion and Conclusion Discussion

There are some issues that should be discussed in the proposed approach. Some of them would be dealt with in the future work. (1) Table 6 describes examples of predicted protein names for the corresponding fine-grained sub-cellular locations. We found that a protein was located in not only one fine-grained sub-cellular location. To categorize proteins into the fine-grained sub-cellular locations, the proposed approach provides the evidential sentences for relations. The following two sentences are the evidential sentences. The first sentence describes the pair of NF-kappaB and nucleus, and the second sentence describes the pair of NF-kappaB and cytoplasm.


381

Table 6. Examples of predicted proteins for the corresponding intracellular compartments Sub-cellular locations Nucleus Cytoplasm Plasma membrane

Protein names NF-AT, NF-kappaB, Radiolabeled VDR, ... FKBP51, NF-kappaB, hGR, protein kinase C, ... monoclonal antibody, CD4, protein kinase C, ...

– IkappaB further reduces the translocation of [NF-kappaB]P into the [nucleus]L thus preventing the expression of proinflammatory genes. (PMID: 10487715) – Associated with its inhibitor, I kappaB, [NF-kappaB]P resides as an inactive form in the [cytoplasm]L. (PMID: 9032271) Although the proposed approach provides the evidential sentences, it needs to analyze the additional information such as the corresponding progress, conditions. (2) At the first time, 11 fine-grained sub-cellular locations selected to deal with all sub-cellular locations, but GENIA corpus contains relevant relations with respect to only six fine-grained sub-cellular locations (See Table 3.). Two of six fine-grained sub-cellular locations have only one relevant relation. We expect that the remaining fine-grained sub-cellular locations might be dealt with by building the relevant corpora to them. (3) With the second issue, MEDLINE abstracts in GENIA corpus are related with only human. We expect that the annotated corpus will become more valuable if it consists of MEDLINE abstracts that are related with not only human but also other kinds of animals or plants. (4) The proposed approach has considered only sentence-level co-occurrences. We expect that context extension to a paragraph or a section might provide much more information than that from the sentence-level co-occurrences. 4.2

Conclusion

This article describes a novel text-based relation extraction approach for PSL. There are three contributions in the proposed approach. The first contribution is improvement of the extraction performance for the PSL by using the various state of the arts NLP techniques. Features from the syntactic analysis also played an important role in extracting PSLs. The second contribution of the proposed approach is that the proposed method extracted not only relations but also their evidential sentences. The evidential sentences are very important to support the relation extraction method and categorize proteins into the fine-grained sub-cellular locations. Moreover, the evidential sentences would be good start materials to analyze the additional information for the relations. The

382

H.-W. Chun et al.

third contribution is construction of the gold standard that contains classification of sub-cellular locations into 11 fine-grained sub-cellular locations, and categorization of relations between proteins and sub-cellular locations into three categorizes.

References 1. Horton, P., Park, K.J., Obayashi, T., Nakai, K.: Protein Subcellular Localization Prediction with WoLF PSORT. In: Asia Pacific Bioinformatics Conference (APBC), pp. 39–48 (2006) 2. Stapley, B.J., Kelley, L., Sternberg, M.: Predicting the subcellular location of proteins from text using support vector machines. In: Pacic Symposium on Biocomputing, PSB (2002) 3. Brady, S., Shatkay, H.: EPILOC: A (Working) Text-Based System for Predicting Protein Subcellular Location. In: Pacific Symposium on Biocomputing, PSB (2008) 4. Kim, J.D., Ohta, T., Tsujii, J.: Corpus annotation for mining biomedical events from literature. BMC Bioinformatics 9(10) (2008) 5. Sim, J., Wright, C.C.: The Kappa Statistic in Reliability Studies: Use, Interpretation, and Sample Size Requirements. Physical Therapy 85(3), 206–282 (2005) 6. Landis, J.R., Koch, G.G.: The measurement of observer agreement for categorical data. Biometrics 33, 159–174 (1977) 7. Berger, A.L., Della Pietra, S.A., Della Pietra, V.J.: A maximum entropy approach to natural language processing. Computational Linguistics 22(1), 39–71 (1996) 8. Tsujii Laboratory: ENJU Deep Syntactic Full Parser ver. 2.1., http://www-tsujii.is.s.u-tokyo.ac.jp/enju/index.html/ 9. Tsujii Laboratory: GENIA Project, http://www-tsujii.is.s.u-tokyo.ac.jp/GENIA/

Enhancing Content-Based Image Retrieval Using Machine Learning Techniques Qinmin Vivian Hu1 , Zheng Ye1,2 , and Xiangji Jimmy Huang1 1

2

Information Retrieval and Knowledge Management Research Lab, York University, Toronto, Canada Information Retrieval Lab, Dalian University of Technology, Dalian, China [email protected], {jhuang,yezheng}@yorku.ca

Abstract. In this paper, we propose a term selection model to help select terms in the documents that describe the images to improve the content-based image retrieval performance. First, we introduce a general feature selection model. Second, we present a painless way for training document collections, followed by selecting and ranking the terms using the Kullback-Leibler Divergence. After that, we learn the terms by the classification method, and test it on the content-based image retrieval result. Finally, we setup a series of experiments to confirm that the model is promising. Furthermore, we suggest the optimal values for the number maxK and the tuning combination parameter α in the experiments.

1

Introduction

For centuries, a wealth of technologies has been done on text in different languages for efficient retrieval. However, when it comes to pictures, machines do not perform images as well as text. One reason which causes this distinction is that text is human being’s creation, while typical images are a mere replica of what human being has seen, concrete descriptions of which are relatively elusive. Naturally, the interpretation of what we see is hard to characterize, and even harder to teach a machine. Yet, over the past decade, ambitious attempts have been made to make machines learn to process, index and search images with great progress [4]. Image retrieval is to retrieve “similar” images to the query which is provided by a user. In general, the query terms are such as keyword, image file/link, or click on some image. The similarity used for retrieval criteria could be meta tags, colour distribution in images, region/shape attributes, etc. Therefore, there are two classical ways for image retrieval as image meta search and content-based image retrieval (CBIR). Image meta search is a search of images based on associated metadata such as captioning, keywords, or text describing a image, etc. CBIR is a technology that in principle helps retrieve images based on their visual content. This characterization of CBIR places it at a unique juncture within the scientific community. While continually developing new techniques for image retrieval, researchers in the field have leveraged mature methodologies developed A. An et al. (Eds.): AMT 2010, LNCS 6335, pp. 383–394, 2010. Springer-Verlag Berlin Heidelberg 2010

384

Q.V. Hu, Z. Ye, and X.J. Huang

in related fields including databases, information retrieval, signal and image processing, graphics, vision, document imaging, design research, human-computer interaction, machine learning, statistical modelling, data mining, artificial intelligence, social sciences, and so on [16]. However, there are shortcomings of CBIR as a real-world technology. For example, one problem is the reliance on visual similarity for the semantic gap between low-level content and higher-level concepts [14]. ImageCLEF is a continuing image retrieval track, run as part of the Cross Language Evaluation Forum (CLEF) campaign. This track evaluates retrieval of images described by text captions based on queries in a language. The goal of its retrieval track is to explore and study the relationship between images and their captions during the retrieval process. It is likely that this track will appeal to members of more than one research community, including those from image retrieval, cross language retrieval and user interaction [11]. In this paper, we focus on the automatic retrieval task of the medical image retrieval in English. We propose a combined term selection model to improve the performance of CBIR based on the text retrieval technology. First, we briefly introduce a general feature selection method. Then, in order to overcome the shortcomings of general feature selection, we propose a painless way to collect training document effectively. We employ the Kullback-Leibler Divergence to give the terms weights, and measures how a terms distribution in the feedback documents diverges from its distribution in the whole training documents. Furthermore, we apply the classification method on the CBIR results. Last but not at least, we conduct a series of experiments to show that our model is promising. The remainder of this paper is organized as follows. First, we propose a term selection model in Section 2. Then, in Section 3, we set up our experimental environment. After that, we present an empirical study to present the experimental results, discuss and analyze the influences of our work in Section 4. Furthermore, we describe the related work in Section 5. Finally, we briefly draw the contributions and conclusions of this paper in Section 6.

2

Term Selection Model

We propose a term selection model for selecting and ranking terms from the learning documents to improve the CBIR performance in this section. First, we introduce a general feature selection method in Section 2.1. Then we describe how to collect the training documents in Section 2.2. After that, we present a method for selecting and ranking features in Section 2.3, followed by introducing a classification method in Section 2.4. Finally, we provide a combination for the weights from the baseline and the classification method respectively in Section 2.5. In addition, we present the pseudo codes for the algorithm in Section 2.6. 2.1

Feature Selection

In statistics and machine learning, feature selection, also known as subset selection, is a process where a subset of features from the data are selected for

Enhancing Content-Based Image Retrieval

385

building learning models. The best subset contains the least number of feature dimensions that most contribute to accuracy. The remaining feature dimensions are discarded as unimportant. Therefore, feature selection helps improve the performance of learning models by: 1) alleviating the effect of the curse of dimensionality; 2) enhancing generalization capability; 3) speeding up learning process; and 4) improving model interpretability [6]. Feature selection algorithms typically fall into two categories: feature ranking and subset selection. Feature ranking ranks the features by a metric and eliminates all features that do not achieve an adequate score. Subset selection searches the set of possible features for the optimal subset. However, simple feature selection algorithms are ad hoc, even though there are also more methodical approaches. From a theoretical perspective, it can be shown that optimal feature selection for supervised learning problems requires an exhaustive search of all possible subsets of features of the chosen cardinality. If large numbers of features are available, this is impractical. For practical supervised learning algorithms, the search is for a satisfactory set of features instead of an optimal set. Therefore, we propose our term selection model to provide narrowed feature candidates for feature ranking and best subset selection. First, we extract features from the text retrieval results as terms for re-ranking the CBIR results. Second, we apply the Kullback-Leibler Divergence (KLD) to give the terms weights, and measures terms’ distributions in the training documents and the whole collection. Third, we employ the classification method to get the training information from the selected terms, and test them on the IR and CBIR results respectively. 2.2

Collecting Training Documents

It’s impractical for feature selection to do an exhaustive search of all possible features. In order to make the effort be a stitch in time, we propose a way of collecting the training documents for feature selection. First of all, we set up our experiments under IR systems which will be introduced in Section 3. Second, according to the IR results, we obtain top k documents of the IR result as a set D. Here a reasonable assumption is given that the top k documents are retrieved by the systems as the most relevant ones. Third, we select and rank terms from the training set D as features in Section 2.3. 2.3

Term Selection and Ranking

For a given query, we have a corresponding set D defined in Section 2.2. We also define the whole data collection as a set C. Then we rank unique terms in D by decreasing order of their KLD weights. KLD is a popular choice of expansion term weighting, which has been applied in many state-of-the-art methods [1]. KLD measures how a term’s distribution in the training document set D and its distribution in the whole collection C. The higher KLD is, the more informative

386


the term is. For a unique term in D, the KLD weight is given by Equation 1. Top maxK terms are selected and sent into the classification method in Section 2.4. KLD(t) = p(t|D)log2 {P (t|D)/P (t|C)}

(1)

where P (t|D) = c(t, D)/c(D) is the generation probability of term t from D. c(t, D) is the frequency of t in D, and c(D) is the count of words in D. P (t|C) = c(t, C)/c(C) is the document pool model. c(t, C) is the frequency of t in C, and c(C) is the count of words in C. 2.4

Classification Method

The support vector machine (SVM) [8] is widely used in text classification in recent years. Its underlying principle is structure risk minimization. Its objective is to determine a classifier or regression function which minimizes the empirical risk (that is, the training set error) and the confidence interval (which corresponds to the generalization or test set error). Given a set of training data, an SVM determines a hyperplane in the space of possible inputs. This hyperplane will attempt to separate the positives from the negatives by maximizing the distance between the nearest positive examples and and negative examples. There are several ways to train SVMs. One particularly simple and fast method is the Sequential Minimal Optimization [12] which is adopted in our study. In addition, we empirically apply the non-linear homogeneous polynomial kernel function at degree dg as follows: k(xi , xj ) = (xi ∗ xj )dg

(2)

where xi and xj are real vectors in a p-dimensional space, and p is the number of features. The exponential parameter dg is default to 1 in our study. 2.5

Weight Combination

There are two weights which are given by the original system and the classification method. In this paper, we simply combine these two weights together by Equation 3. Here α as the tuning combination parameter, will be discussed further in Section 4. W eightre−ranking = α ∗ W eightoriginal + (1 − α) ∗ W eightclassif ier 2.6

(3)

Algorithm

The pseudo code for our proposed term selection method is presented in this section. Six steps are described, which are corresponded to Section 2.1, 2.2, 2.3, 2.4 and 2.5.


387

0. Input IR results by text mining systems; Initial retrieval result by CBIR system. 1. Output Re-ranking lists of text mining systems; Re-ranking list of CBIR system. 2. Initialization k, where top k documents of each IR result of text mining systems are selected and put into the pool D; maxK, selected features according to their KLD weights as training data. 3. Training Data Collection D, containing k documents; C, the whole data collection; 4. Term Selection and Ranking Extracting terms in D with stemming; Using Equation 1 to compute the KLD weights for each term; Sorting the term weights with a decreasing order; 5. Classification and Re-ranking CF, the classifier introduced in Section 2.4; Learning the classifier CF from maxK terms and get information to represent the documents in D as the training data; For the CBIR results { Applying CF on the CBIR result as the testing data; Combining the weights by CF and the CBIR system in Equation 3; Re-ranking the CBIR results by sorting the weights. } For the IR results { Applying CF on the IR results as the testing data; Combining the weights by CF and the IR system in Equation 3; Re-ranking the IR results by sorting the weights.}

Fig. 1. An Algorithm for the Combined Term Selection Method

3

Experimental Setup

In this section, we first show our IR system in Section 3.1. Then we introduce a context-based image retrieval system as in Section 3.2. Third, we describe the CLEF 2009 medical image data set and queries in Section 3.3. 3.1

Text Retrieval System

We used Okapi BSS (Basic Search System) as our main search system. Okapi is an IR system based on the probability model of Robertson and Sparck Jones [2, 7, 13]. The retrieval documents are ranked in the order of their probabilities of relevance to the query. Search term is assigned weight based on its withindocument term frequency and query term frequency. The weighting function used is BM25.

w=

(k1 + 1) ∗ tf (r + 0.5)/(R − r + 0.5) (k3 + 1 ) ∗ qtf ∗log ∗ K + tf (n − r + 0.5)/(N − n − R + r + 0.5) k3 + qtf

⊕ k2 ∗nq ∗

(avdl − dl ) (avdl + dl) (4)

388


where N is the number of indexed documents in the collection, n is the number of documents containing a specific term, R is the number of documents known to be relevant to a specific topic, r is the number of relevant documents containing the term, tf is within-document term frequency, qtf is within-query term frequency, dl is the length of the document, avdl is the average document length, nq is the number of query terms, the ki s are tuning constants K equals to k1 ∗ ((1 − b) + b ∗ dl/avdl), and ⊕ indicates that its following component is added only once per document, rather than for each term. 3.2

Content-Based Image Retrieval System

Content-based Image Retrieval (CBIR) systems enable users to search a large image database by issuing an image sample, in which the actual contents of the image will be analyzed. The contents of an image refer to its features: colors, shapes, textures, or any other information that can be derived from the image itself. This kind of technology sounds interesting and promising. The key issue in CBIR is to extract representative features to describe an image. However, this is a very difficult research topic. In this paper, we explore three representative features for medical image retrieval as follows. 1. Color and Edge Directivity Descriptor (CEDD): is a low level feature which incorporates color and texture information in a histogram [5]. 2. Tamura Histogram Descriptor: features coarseness, contrast, directionality, line-likeness, regularity, and roughness. The relative brightness of pairs of pixels is computed such that degree of contrast, regularity, coarseness and directionality may be estimated [15]. 3. Color Histogram Descriptor: Retrieving images based on color similarity is achieved by computing a color histogram for each image that identifies the proportion of pixels within an image holding specific values (that humans express as colors). Current research is attempting to segment color proportion by region and by spatial relationship among several color regions. Examining images based on the colors they contain is one of the most widely used techniques because it does not depend on image size or orientation. 3.3

Data Set and Queries

In this paper, we use the dataset of the CLEF 2009 medical image retrieval track. It contains scientific articles from two radiology journals, Radiology and Radiographics. The size of the database is a total of 74,902 images. This collection constitutes an important body of medical knowledge from the peerreviewed scientific literature including high quality images with annotations. Images are associated with journal articles and can be part of a figure. Figure captions are made available to participants as well as the part concerning a particular subfigure if available. For each image, captions and access to the full text article through the Medline PMID (PubMed Identifier) were provided. An articles PMID could be used to obtain the officially assigned MeSH (Medical Subject


389

Table 1. Performance of Term Selection Model on Content-based Image Retrieval MAP Performance with maxK maxK = 10 maxK = 20 maxK = 30 maxK = 40 Baseline 0.0045 0.1 0.0056 0.0053 0.0046 0.0045 (24.44%) (17.78%) (2.22%) (0.00%) 0.2 0.008 0.0067 0.0049 0.0047 (77.78%) (48.89%) (8.89%) (4.44%) 0.3 0.0115 0.0083 0.005 0.0049 (155.56%) (84.44%) (11.11%) (8.89%) 0.4 0.0164 0.0095 0.0054 0.0053 (264.44%) (111.11%) (20.00%) (17.78%) α 0.5 0.0203 0.0106 0.0057 0.0058 (351.11%) (135.56%) (26.67%) (28.89%) 0.6 0.0218 0.0126 0.0069 0.0068 (384.44%) (180.00%) (53.33%) (51.11%) 0.7 0.0251 0.0133 0.0078 0.0086 (457.78%) (195.56%) (73.33%) (91.11%) 0.8 0.0251 0.0130 0.0088 0.0096 (457.78%) (188.89%) (95.56%) (113.33%) 0.9 0.0223 0.0121 0.0092 0.0100 (395.56%) (168.89%) (104.44%) (122.22%) 1.0 0.0222 0.012 0.009 0.0094 (393.33%) (166.67%) (100.00%) (108.89%)

Headings) terms. The collection was entirely in English. However, the queries are supplied in German, French, and English [11]. We focus on the queries in English. The image-based queries have been created using methods where realistic search queries have been identified by surveying actual user needs. Each query contains 2 to 4 sample images. More information can be found in [11].

4

An Empirical Study

In this section, we first present our experimental results in Section 4.1. Then we analyze and discuss the influence of our proposed model on text retrieval results and CBIR results respectively. 4.1

Experimental Results

The experimental results of the proposed model on the CBIR result are presented in Table 1. The term number maxK and the tuning combination parameter α are set to different values. The original baseline is first displayed at the third row. The remained are the performance under the settings. The values in the parentheses are the relative rates of improvement over the baseline. In addition, before tuning the parameters maxK and α, we set k = 10 as one of conclusion in our paper [19]. All our results presented in this paper are automatic, i.e., no manual query modification or iterative selection of results is allowed. In order to show the robustness of our model, we also conduct the experiments which apply the proposed model on the text retrieval result. The re-ranking results are presented in Table 2. The tuning parameters maxK and α are the same as set in Table 1. The text retrieval baseline is displayed as well. The values

390

Q.V. Hu, Z. Ye, and X.J. Huang Table 2. Performance of Term Selection Model on Text Retrieval MAP Performance with maxK maxK = 10 maxK = 20 maxK = 30 maxK = 40 Baseline 0.3520 0.1 0.3561 0.3547 0.3546 0.3534 (1.16%) (0.77%) (0.74%) (0.40%) 0.2 0.3612 0.3578 0.356 0.354 (2.61%) (1.65%) (1.14%) (0.57%) 0.3 0.3677 0.3604 0.3591 0.3568 (4.46%) (2.39%) (2.02%) (1.36%) 0.4 0.368 0.3639 0.3581 0.354 (4.55%) (3.38%) (1.73%) (0.57%) α 0.5 0.3692 0.3634 0.3564 0.3525 (4.89%) (3.24%) (1.25%) (0.14%) 0.6 0.3657 0.3612 0.356 0.3472 (3.89%) (2.61%) (1.14%) -(1.36%) 0.7 0.3622 0.3542 0.3485 0.3376 (2.90%) (0.63%) -(0.99%) -(4.09%) 0.8 0.3558 0.3455 0.3386 0.3269 (1.08%) -(1.85%) -(3.81%) -(7.13%) 0.9 0.3462 0.3345 0.3252 0.3132 -(1.65%) -(4.97%) -(7.61%) -(11.02%) 1.0 0.332 0.322 0.3093 0.3003 -(5.68%) -(8.52%) -(12.13%) -(14.69%)

in the parentheses are the relative rates of improvement over the text retrieval baseline. We also note k = 10. 4.2

Influence of Term Selection Model on Content-Based Image Retrieval

To illustrate the results in Table 1 graphically, we re-plot these data in Figure 2 and 3. The x-axis represents the tuning combination parameter α which varies from 0.1 to 1.0. The y-axis shows the MAP performance. maxK is set as {10, 20, 30 ,40} respectively. We can see that all re-ranking results of the proposed Term Selection Model on CBIR Result 0.03

0.025

maxK=10 maxK=20 maxK=30 maxK=40 Baseline

0.02

0.015

0.01

0.005

0 0.1

0.2

0.3

0.4 0.5 0.6 0.7 Alpha Tuning as the Linear Combination Parameter

0.8

0.9

Fig. 2. Term Selection Model on CBIR Result

1


391

Improvements on CBIR Result 5 4.5 4

maxK=10 maxK=20 maxK=30 maxK=40

3.5 3 2.5 2 1.5 1 0.5 0 0.1

0.2

0.3


0.8

0.9

1

Fig. 3. Improvements of Term Selection Model on CBIR Result

term selection model outperform the original CBIR baseline. It’s very successful that no matter how many terms are selected as the training features, and no matter how to combine the different weights given by the CBIR baseline and the classifier, the re-ranking results make great improvements. In particular, we believe the proposed model can make progress if we can obtain a better baseline, since the current baseline is as low as 0.0045. 4.3

Influence of Term Selection Model on Text Retrieval

The proposed model has shown its effectiveness and promising on content-based image retrieval in Section 4.2. In order to further prove the robustness of the model, we conduct the same experiments on the text retrieval results, under four maxK values and 10 tuning combination parameter α. We re-plot Table 2 in Figure 4 and 5. It’s interesting that we can not make improvements on all maxK and α. However, if we fix α within an interval, the proposed model brings significant improvements. So we can say that the proposed term selection model works on the IR result. But it depends on how to combine the classification weight and the baseline weight. Also, the improvements in Figure 5 show that the value of maxK effect the performance obviously. In summary, the proposed model can boost the text retrieval performance through tuning the parameters. 4.4

Influence of Term Number maxK

In the experiments, we try four different maxK values to show the influence of maxK on the re-ranking results. In Figure 2 and 4, the re-ranking CBIR and text retrieval results tell us when maxK equals to 10, the re-ranking performance achieves the best, no matter how α tunes. We can also observe a trend that the improvements are getting smaller, when maxK is getting bigger. This suggests us how to select terms as features at the stage of feature selecting and ranking.

392


Term Selection Model on Text Retrieval Result 0.37

0.36

0.35

0.34 maxK=10 maxK=20 maxK=30 maxK=40 Baseline

0.33

0.32

0.31

0.3 0.1

0.2

0.3


0.8

0.9

1

Fig. 4. Term Selection Model on Text Retrieval Result Improvements on Text Retrieval Result 0.05

0

maxK=10 maxK=20 maxK=30 maxK=40

−0.05

−0.1

−0.15 0.1

0.2

0.3


0.8

0.9

1

Fig. 5. Improvements of Term Selection Model on Text Retrieval Result

4.5

Influence of Tuning Parameter α

For the re-ranking CBIR results in Table 1 and Figure 2, we find that the reranking results outperform the baseline for all maxK and α. In particular, the re-ranking performance reaches the best at α = 0.7. However, for the re-ranking text retrieval results in Table 2 and Figure 5, the re-ranking results are not always better than the baseline. Based on these observations, an interval for α can be recommended as [0.1, 0.5]. Therefore, for both CBIR and text retrieval, we suggest to tune α around 0.5.

5

Related Work

Lots of previous work has been done on feature selection. In the early 1992, Kira and Rendell [10] described a statistical feature selection algorithm called


393

RELIEF that uses instance based learning to assign a relevance weight to each feature. Later in 1994, John, Kohavi and Pfleger [9] addressed the problem of irrelevant features and the subset selection problem. They presented definitions for irrelevance and for two degrees of relevance (weak and strong). They also state that features selected should depend not only on the features and the target concept, but also on the induction algorithm. Further, they claim that the filter model approach to subset selection should be replaced with the wrapper model. In a comparative study of feature selection methods in statistical learning of text categorization, Yang and Pedersen [18] evaluated document frequency (DF), information gain (IG), mutual information (MI), a χ2 -test (CHI) and term strength (TS); and found IG and CHI to be the most effective. Blum and Langley [3] focussed on two key issues: the problem of selecting relevant features and the problem of selecting relevant examples. Xing, Jordan and Karp [17] successfully applied feature selection methods to a classification problem in molecular biology involving only 72 data points in a 7130 dimensional space. They also investigated regularization methods as an alternative to feature selection, and showed that feature selection methods were preferable in the problem they tackled. Guyon and Elisseeff [6] gave an introduction to variable and feature selection. They recommend using a linear predictor of your choice and select variables in two alternate ways: (1) with a variable ranking method using a correlation coefficient or mutual information; (2) with a nested subset selection method performing forward or backward selection or with multiplicative updates.

6

Conclusions

In this study, our contributions are four-fold. First, we propose a term selection model to improve the content-based image retrieval performance. Second, we introduce our term selection and ranking method to collect the training documents. Later we employ the classification method to classify and re-ranking the baselines. Third, we evaluate the proposed model on the CLEF 2009 medical image data. The experimental results confirm the model works very well on the CBIR system. Furthermore, we also conduct the same experiments on the text retrieval result, which shows the robustness of the proposed model since it works well on the text retrieval baseline. Fourth, for the term number maxK and the tuning combination parameter α, our experimental results suggest that maxK equal to 10 and α vary around 0.5.

References [1] Amati, G.: Probabilistic models for information retrieval based on divergence from randomness. PhD thesis, Department of Computing Science, University of Glasgow (2003) [2] Beaulieu, M., Gatford, M., Huang, X., Robertson, S., Walker, S., Williams, P.: Okapi at TREC-5. In: Proceedings of TREC-5, pp. 143–166. NIST Special Publication (1997) [3] Blum, A.L., Langley, P.: Selection of relevant features and examples in machine learning. Artif. Intell. 97(1-2), 245–271 (1997)

394


[4] Datta, R., Joshi, D., Li, J., Wang, J.Z.: Image retrieval: Ideas, influences, and trends of the new age. ACM Comput. Surv. 40(2), 1–60 (2008) [5] Gasteratos, A., Vincze, M., Tsotsos, J.: Cedd: Color and edge directivity descriptor. a compact descriptor for image indexing and retrieval. In: Gasteratos, A., Vincze, M., Tsotsos, J.K. (eds.) ICVS 2008. LNCS, vol. 5008, pp. 312–322. Springer, Heidelberg (2008) [6] Guyon, I., Elisseeff, A.: An introduction to variable and feature selection. J. Mach. Learn. Res. 3, 1157–1182 (2003) [7] Huang, X., Peng, F., Schuurmans, D., Cercone, N., Robertson, S.: Applying machine learning to text segmentation for information retrieval. Information Retrieval Journal 6(4), 333–362 (2003) [8] Joachims, T.: Transductive inference for text classification using support vector machines. In: ICML 1999: Proceedings of the Sixteenth International Conference on Machine Learning, pp. 200–209. Morgan Kaufmann Publishers Inc., San Francisco (1999) [9] John, G.H., Kohavi, R., Pfleger, K.: Irrelevant features and the subset selection problem. In: ICML, pp. 121–129 (1994) [10] Kira, K., Rendell, L.A.: A practical approach to feature selection. In: ML 1992: Proceedings of the Ninth International Workshop on Machine Learning, pp. 249– 256. Morgan Kaufmann Publishers Inc., San Francisco (1992) [11] Muller, H., Kalpathy-Cramer, J., Eggel, I., Bedrick, S., Radhouani, S., Bakke, B., Kahn Jr., C., Hersh, W.: Overview of the clef 2009 medical image retrieval track. In: CLEF working notes 2009, Corfu, Greece (2009) [12] Platt, J.C.: Fast training of support vector machines using sequential minimal optimization, pp. 185–208 (1999) [13] Robertson, S.E., Walker, S.: Some Simple Effective Approximations to the 2Poisson Model for Probabilistic Weighted Retrieval. In: Proceedings of the 17th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Dublin, Ireland, July 3-6, pp. 232–241. ACM/Springer (1994) [14] Smeulders, A.W.M., Worring, M., Santini, S., Gupta, A., Jain, R.: Content-based image retrieval at the end of the early years. IEEE Trans. Pattern Anal. Mach. Intell. 22(12), 1349–1380 (2000) [15] Tamura, H., Mori, S., Yamawaki, T.: Textural features corresponding to visual perception. IEEE Transactions on Systems, Man and Cybernetics 8, 460–473 (1978) [16] Wang, J.Z., Boujemaa, N., Del Bimbo, A., Geman, D., Hauptmann, A.G., Tesić, J.: Diversity in multimedia information retrieval research. In: MIR 2006: Proceedings of the 8th ACM International Workshop on Multimedia Information Retrieval, pp. 5–12. ACM, New York (2006) [17] Xing, E.P., Jordan, M.I., Karp, R.M.: Feature selection for high-dimensional genomic microarray data. In: ICML 2001: Proceedings of the Eighteenth International Conference on Machine Learning, pp. 601–608. Morgan Kaufmann Publishers Inc., San Francisco (2001) [18] Yang, Y., Pedersen, J.O.: A comparative study on feature selection in text categorization. In: ICML 1997: Proceedings of the Fourteenth International Conference on Machine Learning, pp. 412–420. Morgan Kaufmann Publishers Inc., San Francisco (1997) [19] Ye, Z., Huang, X., Lin, H.: Towards a better performance for medical image retrieval using an integrated approach. In: Proceedings of the 10th Workshop of the Cross-Language Evaluation Forum (CLEF 2009), Corfu, Greece, September 30 October 2 (2009)

Modeling User Knowledge from Queries: Introducing a Metric for Knowledge Frans van der Sluis and Egon L. van den Broek Department of Human Media Interaction University of Twente P.O. Box 217, 7500 AE Enschede, The Netherlands [email protected], [email protected]

Abstract. The user’s knowledge plays a pivotal role in the usability and experience of any information system. Based on a semantic network and query logs, this paper introduces a metric for users’ knowledge on a topic. The finding that people often return to several sets of closely related, well-known, topics, leading to certain concentrated, highly activated areas in the semantic network, forms the core of this metric. Tests were performed determining the knowledgeableness of 32,866 users on in total 8 topics, using a data set of more than 6 million queries. The tests indicate the feasibility and robustness of such a user-centered indicator.

1

Introduction

Lev Vygotsky (1896 - 1934) and Jean Piaget (1896 - 1980) were the first to indicate the importance of a user’s knowledge base for interpreting information. New knowledge should not be redundant but should have sufficient reference to what the user already knows. Moreover, sufficient overlap fosters the understandability of information (Van der Sluis and Van den Broek, in press), in particular deep understanding (Kintsch, 1994); is an important determinant of interest (Schraw and Lehman, 2001); and, through both understandability and interest, the knowledge base has an indirect effect on motivation (Reeve, 1989). Concluding, it is of pivotal importance to have a model of the user’s knowledge allowing to measure the distance between a topic and the user’s Knowledge Model (KM). Several types of models have been used for user (knowledge) modeling; e.g., weighted keywords, semantic networks, weighted concepts, and association rules. A core requirement is the connectedness of concepts, which makes a semantic network particularly useful for knowledge modeling. Query logs offer an unobtrusive, large source, storing basic information about the search history (e.g., queries and some click through data) for a limited amount of time (e.g., a cookie will often expire within 6 months) (Gauch et al., 2007). The use of a query log to base a KM on assumes that what one has searched for, one has knowledge about. We will give three arguments to support this assumption: 1) The user has to know a word before being able to posit it to a search system. This is well known as the vocabulary problem, where users have difficulty finding the right word to represent their exact information need; A. An et al. (Eds.): AMT 2010, LNCS 6335, pp. 395–402, 2010. c Springer-Verlag Berlin Heidelberg 2010

396

F. van der Sluis and E.L. van den Broek

i.e., they are confined to the words they know (Furnas et al., 1987). 2) A query log represents the history of searches. As a user performs a query to learn new information, a query log primarily indicates that what has recently been learned. 3) Users often return to very specific domains when performing a new search (Wedig and Madani, 2006). These domains are indicative of what is familiar to the user; i.e., domains the user has at least above-average knowledge about. Reliability is a salient problem for user profiles based on implicit sources. Logs can contain noise; e.g., due to sharing a browser with a different user. Moreover, polysemy is a structural problem for information retrieval. Polysemy refers to the need of context in order to know the intentioned meaning of a word. Any knowledge metric based on implicit sources has to deal with this uncertainty. The remainder of this paper looks at the feasibility of measuring the amount of knowledge a user has about a certain topic. A semantic KM is introduced in Section 2, using a query log as data source. The idea that users posit many queries on closely related topics that are very well-known to them, forms the basis for the metric of knowledge presented in Section 3. Some examples of the use of the metric are shown in Section 4. Finally, the results are discussed, reflecting on the feasibility of a web system adaptive to the knowledge base of users.

2

Knowledge Model

The KM was build on WordNet version 3.0. WordNet is a collection of 117, 659 related synonym sets (synsets), each consisting of words of the same meaning (Miller, 1995). As input data, an AOL query log was used (cf. Pass et al., 2006). This log consists of 21, 011, 340 raw queries, gathered from 657, 426 unique user ID’s. As in general a large proportion of all queries is caused by a small percentage of identifiable users (Wedig and Madani, 2006), only a subset of the most active 5% of users was used. One outlier was removed, having 106, 385 queries compared to 7, 420 for the next highest user. The final subset consisted of 32, 866 users and was responsible for 29.23% of all unique queries. The procedure used to analyze the query logs is an iterative one, analyzing each unique query present in the (subset of the) log. For each query (q), the total set of synsets touched upon (Sq ) was computed as follows: Sq (q) = ∪ S(w), w∈q

S(w) = {s ∈ W |w ∈ s}.

(1)

Here, S(w) gives the set of all possible synsets s of lexical dictionary W (i.e., WordNet) related to the word w of query q. All possible lemmata for each word w were used to retrieve the synsets s. With the analysis of the query log noise reduction was disregarded; no effort was spent to reduce the effects of polysemy. This resulted in substantial noise within the data set. On average, for the subset of the data used, a query loaded upon 13.92 synsets. The results of the analyses were stored as a large collection of (user, query, synset, part-of-speech) tuples. This collection allowed for the retrieval of the number of times a user posited a query on a specific synset Q(s).

Modeling User Knowledge from Queries

3

397

Similarity Metric

The distance between a certain topic and the KM indicates the amount of knowledge a user has on that topic. We exploit this by measuring the semantic distance k(t) (See Equation 6) between a synset t, representing the topic, and the KM. This is done by looking at the number of activated synsets A near the topic synset t, which is inspired by the finding that people stick to certain domains for their searches (Wedig and Madani, 2006). This leads to a large number of (unique) queries in a specific domain and, accordingly, to very few queries in non-familiar domains. By looking at areas of activation, the method attends one of the core problems of user profiling: polysemy. It is expected that polysemy will lead to the random, isolated, activation of synsets and not to the activation of an area of synsets. The number of activated synsets An (t) is defined as: An (t) = #{s ∈ Sn (t)|Q(s) > 0},

(2)

where Q(s) is the previously defined number of queries per synset s (See Section 2) and Sn is the set of synsets in exactly n steps related to topic t (Gervasi and Ambriola, 2003): S0 (w) = {s ∈ W |w ∈ s},

(3)

Sn (w) = Sn−1 ∪{s ∈ W |r(s, s ) ∧ s ∈ Sn−1 (w)}.

1 Weight (d)

1 8000 6000 4000 2000

Weight (a)

Related Synsets (Sn)

Here, w is the word of interest (i.e., the topic t), s represents all the synsets in lexical dictionary W (e.g., WordNet) of which the word w is part of, and r(s, s ) is a boolean function that indicates whether or not there is any relationship between synset s and s‘. Please note that S0 is the same as S from equation 1 and that Sn (w) has a memory: it includes the synsets up to n. Following Gervasi and Ambriola (2003), hyponymy relationships are excluded from function r(s, s ). The absolute count A is not directly useful for two reasons. First, the growth in synsets for higher ns is approximately exponential (cf. Figure 1). This can lead to very high values of An (t) for higher ns. Second, a normalized value is preferable, allowing to estimate the limits of the function kN (t). Consequently,

0.5

0 0

5 Steps (n)

10

Fig. 1. Average related synsets

0.5

0 0 5 10 Activated Synsets (A)

Fig. 2. Activation limit function a

0

5 Steps (n)

Fig. 3. Distance function d

10

decay

398


there is a need for a strict limiting function for the contribution of An (t) to kN (t). Let c1 be a constant and An be defined in Equation 2; then, the weighted activation a at n steps from topic t is given by: an (t) = 1 − (c1 )An (t) .

(4)

The constant c1 is indicative for how many activated synsets are needed. Since lower values of An (t) are very informative (as opposed to higher values), we choose for a value of 12 . Figure 2 illustrates this idea. It is unlikely that a user has as much knowledge on synsets n = 3 steps away as on synsets n = 1 step away. Therefore, a decay function is needed, giving a high penalty to activation far away. We use the following exponential decay function to achieve this: d(n) = (c2 )n ,

(5)

where n is the number of steps of the connection and c2 a constant determining the distance penalty. The output of this function is illustrated in Figure 3 for c2 = 12 over n = 10 steps. The idea clearly shows from this figure. Activation in the KM close to the input synsets are given a higher weight, to such an extent that twice as much weighted activation is needed at n = 1, four times as much activation at n = 2, etc. Combining Equations 4 and 5, the knowledge a user has on a topic synset t can be estimated by the function k(t): kN (t) =

N

d(n)an (t).

(6)

n=0

The result is a weighted count of how many related synsets a user has previously touched upon, able to indicate how close a topic is to the user’s knowledge. The maximum of the function is 2.00, the minimum is 0. A short review of the behavior of kN (t) helps interpreting its range. Consider some hypothetical users and their unweighted activation values of An (t), as shown in Table 1. The values of kN (t) are shown in the last column. An interpretation of the values of An (t) is that the first user has relatively little knowledge about the topic, the second and third have reasonable knowledge, and the fourth has a lot of knowledge. Therefore, as a rule of thumb, we use the value of 0.85 as a threshold. Table 1. Exemplar users User 1 2 3 4

A0 0 1 0 1

A1 1 1 3 3

A2 1 1 3 4

A3 1 1 3 4

A4 1 1 3 4

A5 1 1 3 5

k 0.48 0.98 0.85 1.38


4

399

Proof of Concept

As a proof of concept, the KM and metric were tested on eight topics. The performed test method defines, for all users, their knowledge about the topic. This was done through the following steps. Each topic is represented by a word; subsequently, for every word, the corresponding synsets were collected. Next, for each user, kN (t) was computed. A value of N = 5 was used. The distribution of users over kN (t) is illustrated by a 10-bins histogram, counting the number of users per bin of kN (t), shown in Figure 4. Furthermore, the total number of “knowledgeable users” (K) for each topic was calculated by counting the number of users with kN (t) > 0.85; i.e., the previously defined threshold. The values of K for each topic are shown in Table 2. Three of the eight topics were derived from a list of topics that Wedig and Madani (2006) identified as being sticky topics: those topics that a group of users often return to. The sticky topics are: lottery, travel, and money. Figure 4(a-c) illustrates that for all three topics there is a particular user group active in the topic and a greater set of users inactive; i.e., with kN (t) < 0.85). From these distributions of users, the knowledge metric can identify knowledgeable users. Furthermore, Figure 4d shows a very uncommon word: malonylurea (Gervasi and Ambriola, 2003). This word refers to a chemical compound often used in barbiturate drugs. Being a word previously unknown to the authors of this article, the expectation was that this word will not have many users common to the word. This expectation was correct, as K = 0. The topic vacation is, besides a very common word, also a somewhat peculiar word concerning its semantic relatedness. The word corresponds to two synsets 4

1

2 1

4

x 10 Users

Users

2

x 10

1

2 1

0

0

0

0

0 1 2 Knowledge Metric (k)




(b) Travel

(c) Money

4

x 10

x 10 Users

2 1

2 1

(d) Malonylurea

4

4

x 10 2

x 10 Users

4

Users

Users

2

(a) Lottery

Users

4

x 10

Users

4

x 10

1

2 1

0

0

0

0





(e) Vacation

(f) University

(g) School

(h) University and School Fig. 4. Distribution of users over topic relatedness

400

F. van der Sluis and E.L. van den Broek Table 2. Number of knowledgeable users per topic Topic Health Lottery Travel

K 5,411 3,624 3,960

Topic Money Malonylurea Vacation

K 3,082 0 4,952

Topic University School University and School

K 7,551 14,077 4,701

that after following its relationships for n = 3 steps still lead to only S3 (t) = 8 synsets. Compared to the average, this is a very low number of synsets; see also Figure 1. However, the metric can still identify (at least part of) the knowledgeable users: K = 4, 952; see Figure 4e). Finally, two somewhat related words were analyzed: university (Figure 4f) and school (Figure 4g). Both are very popular topics, respectively having K = 7, 551 and K = 14, 077 users. Moreover, the topics were combined by taking only those users regarded as knowledgeable users (K) on both topics and averaging their values of kN (t). This was still a large number of users: K = 4, 701; see also Figure 4h. Hence, a concatenation of topics also seems possible.

5

Discussion

Founded on the notion that people perform most of their searches on a few salient topics, this paper introduced a metric of knowledge. Queries of users were monitored and the synsets used were identified. This revealed that areas of the semantic network on which a user had posed queries before were most activated. This provided a metric of users’ knowledge on a topic. Using the metric introduced in Section 3, Section 4 showed that it is indeed feasible to give an indication of a user’s knowledge. Moreover, we showed that this indication can be founded on i) relatively limited data, ii) only a restricted KM, and iii) a noisy data set, as no effort was put into noise reduction. So, if based on a measure of the spread of activation, the possibility of “measuring knowledge” seems to be quite robust and can, thus, be applied to make information systems adaptive. Several improvements can be made to the source, model, and metric. To start with the source, the use of queries provides limited information. This is partly due to the inherent ambiguity of the often short, keywords-based, queries. However, users also, often, only search for items that are at a certain moment in their interest. Hence, using only this source gives a skewed KM. A more elaborate approach could for example be the use of the content of a user’s home directory. The metric itself has room for improvement as well. For example, the constants c1 and c2 can be further optimized. Moreover, the implementation of semantic relationship, as introduced in Section 3, can be improved. Currently, it is a simple distance measure: the n relations a synset is away from a topic. Other measures have been proposed as well; e.g., Hirst-StOnge, Resnik, LeocockChodorow, and Jiang-Conrath (Budanitsky and Hirst, 2006). However, not all features will be suitable for our purpose – a measure of distinctness between two


401

items of knowledge (or, information) is needed, which is consistent with how the user will experience it (Van der Sluis et al., in press). A similar notion holds for the threshold of .85, used to indicate when a user is knowledgeable: user tests are needed to compare the metric to how the user perceives his knowledgeableness. The last argument of perceived relatedness and knowledgeableness is intrinsically interweaved with the used model of knowledge. For the purpose of modeling knowledge, a semantic model such as WordNet is distinct from a true reflection of knowledge. Besides the absence of named entities in WordNet, one of the most salient problems is the small number of relations needed to converge a large part of all synsets. This causes the effect seen in all examples shown in Figure 4, where almost all users tend to obtain a low relationship with the topic of interest. This is even the case for very uncommon words such as malonylurea. Hence, a more elaborate model can hold much more power in distinguishing between parts of the model, alleviating the need for a strict decay function; see also Figure 3. Every KM that is based on implicit sources will be particularly successful in identifying true positives; i.e., topics on which a user has knowledge. In contrast, the identification of false negatives forms a more substantial challenge. No one data source will cover all the knowledge a user has. Every KM will also, to a lesser extent, suffer from false positives. This can occur when people simply forget about a topic or when they share their account with a different user. However, this is less of a problem when looking at the spread of activation, as this spread indicates that a user has often returned to that topic. Moreover, when using not up-to-date logs, this problem should be less prominent. LaBerge and Samuels (1974) noted about comprehension: “The complexity of the comprehension operation appears to be as enormous as that of thinking in general” (p. 320). Notwithstanding, by looking at the spread of activation extracted from a user’s query history, it is feasible to infer part of the comprehensibility: the relatedness of a (new) topic to a model of the user’s knowledge. This metric will pave the way to adaptive web technology, allowing systems to directly aim for a user’s understandability, interest, and experience.

Acknowledgements We would like to thank Claudia Hauff, Betsy van Dijk, Anton Nijholt, and Franciska de Jong for their helpful comments on this research. This work was part of the PuppyIR project, which is supported by a grant of the 7th Framework ICT Programme (FP7-ICT-2007-3) of the European Union.

References Budanitsky, A., Hirst, G.: Evaluating wordnet-based measures of semantic distance. Computational Linguistics 32(1), 13–47 (2006) Furnas, G.W., Landauer, T.K., Gomez, L.M., Dumais, S.T.: The vocabulary problem in human-system communication. ACM Commun. 30(11), 964–971 (1987)

402


Gauch, S., Speretta, M., Chandramouli, A., Micarelli, A.: User profiles for personalized information access. In: Brusilovsky, P., Kobsa, A., Nejdl, W. (eds.) Adaptive Web 2007. LNCS, vol. 4321, pp. 54–89. Springer, Heidelberg (2007) Gervasi, V., Ambriola, V.: Quantitative assessment of textual complexity. In: Merlini Barbaresi, L. (ed.) Complexity in Language and Text, pp. 197–228. Plus Pisa University Press, Pisa, Italy (2003) Kintsch, W.: Text comprehension, memory, and learning. American Psychologist 49(4), 294–303 (1994) LaBerge, D., Samuels, S.J.: Toward a theory of automatic information processing in reading. Cognitive Psychology 6(2), 293–323 (1974) Miller, G.A.: Wordnet: a lexical database for english. ACM Commun. 38(11), 39–41 (1995) Pass, G., Chowdhury, A., Torgeson, C.: A picture of search. In: Proc. 1st Intl. Conf. on Scalable Information Systems. ACM Press, New York (2006) Reeve, J.: The interest-enjoyment distinction in intrinsic motivation. Motivation and Emotion 13(2), 83–103 (1989) Schraw, G., Lehman, S.: Situational interest: A review of the literature and directions for future research. Educational Psychology Review 13(30), 23–52 (2001) Van der Sluis, F., Van den Broek, E.L.: Applying Ockham’s razor to search results: Using complexity measures in information retrieval. In: Information Interaction in Context (IIiX) Symposium, ACM, New York (in Press) Van der Sluis, F., Van den Broek, E.L., Van Dijk, E.M.A.G.: Information Retrieval eXperience (IRX): Towards a human-centered personalized model of relevance. In: Third International Workshop on Web Information Retrieval Support Systems, Toronto, Canada, August 31 (2010) Wedig, S., Madani, O.: A large-scale analysis of query logs for assessing personalization opportunities. In: KDD 2006: Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 742–747. ACM, New York (2006)

Computer-Assisted Interviewing with Active Questionnaires Seon-Ah Jang, Jae-Gun Yang, and Jae-Hak J. Bae∗ School of Computer Engineering & Information Technology, University of Ulsan, Ulsan, Republic of Korea [email protected], {jgyang,jhjbae}@ulsan.ac.kr

Abstract. Computer-assisted interviewing systems have various benefits, as compared to paper-and-pencil surveys. The engine for processing questionnaires, however, should be reprogrammed when the questionnaire is changed since its processing logic is hard-coded in the system. As such, the engine for processing questionnaires depends on the questionnaires. This study makes the engine for processing questionnaires independent of questionnaires using an active document model. In this model, machines can process documents with rules specified in the documents. The active questionnaire, which is an active document, is composed of questions, control logic, and a knowledgebase. Among these, control logic expresses a method of processing questions in an executable XML. In this paper, we propose a framework for processing active questionnaires and describe its implementation. Keywords: Questionnaire, Active Documents, XML, ERML, Logic Programming, Computer-Assisted Interviewing.

1 Introduction Traditionally, companies and government offices mainly use paper forms in order to exchange and manage necessary information. In addition, each organization adopts a certain document management system based on the technology of Electronic Data Interchange (EDI) in order to reduce costs and improve productivity. As a result, user interfaces are gradually replaced with electronic form documents and many application programs have been developed based on these documents. These form documents, however, do not include processing logics, but rather only define the appearance and content of documents as is often the case with paper documents. Recently, advanced electronic form documents can include information related to the business process that expresses the flow of the document processing, as well as the user interface and business data. This kind of form document is called an active document [1]. A questionnaire is no exception. Computer-assisted interviewing (CAI) is widely used to overcome shortcomings [2] of paper-and-pencil surveys. CAI systems have provided various benefits in terms of time, costs, return rates, and reliability of ∗

Corresponding author.

A. An et al. (Eds.): AMT 2010, LNCS 6335, pp. 403–410, 2010. © Springer-Verlag Berlin Heidelberg 2010

404

S.-A. Jang, J.-G. Yang, and J.-H.J. Bae

responses, by providing an environment in which respondents can conveniently answer the questions. Despite these advantages, they still suffer from system maintenance problems. Electronic questionnaires also consist of the structure and content of questions, just like their paper counterparts. The methods for processing questions are generally hard-coded in the CAI system. When preparing for a new questionnaire, the system developer should modify the engine for processing questionnaires to reflect the new control flow in the questionnaire. In this paper, we propose a CAI system that adopts active documents in order to solve the maintenance problems, and implement the system as the Web Interview System with Active Documents (WINAD) in order to examine its usefulness.

2 Previous Work There is SSS(Simple Survey System)[2] which has processing logics in questionnaires. SSS only includes three predefined functions (routing logic, fills and computed variables) in XML schemas in questionnaires. However, based on active documents, WINAD can handle any control logic which can be described in Prolog. Table 1. Approaches to Active Documents

Research Concept of active document

Technology used Key idea

Language for behavior representation Application areas

AHDM (Active Hypertext Document Model) [3] Active document combines structured and hyperlinked information, like any other hypertext document, but also contains associated procedural information in the form of scripts or program fragments XML, link, Xpointer, CSS DOM Realizes applications through hypertext documents with embedded application logic

Displet [4] Active document is a document that can provide some autonomous active behavior; it may be displayed, printed, searched, perform computations, produce additional documentation, perform animations XML, XSLT, Java Associating XML elements to Java classes that are able to perform behaviors

Script language (Tcl / Java language OTcl) Computer supported workflow systems

Multi-document agent applications

Total Shadow and ActiveForm [1] Jima [5] Active document is a content-aware, autonomous, proactive, adaptive, and context-aware document

Active document includes in itself data, business rules, and data integrity constraints that are implied in documents as declarative knowledge to support the automation of document processing

Jini

XML, SLT, JSP, Prolog Implements active Form documents imply documents knowledge of through mobile themselves, namely, the agent technologies methods of processing and context documents that reflect information the intention of their infrastructure designer Java language ERML Ubiquitous computing and communication environment

Intelligent web applications, intelligent form document processing systems

Several studies have employed the concept of active documents, such as in electronic publishing, workflow systems, and mailing systems. Table 1 summarizes various approaches to including control logics in documents. The control logics specify how to process documents and how to exhibit their active behaviors. They are

Computer-Assisted Interviewing with Active Questionnaires

405

compared in terms of five criteria, which are the concept of active document, technology used, key idea, language for behavior representation, and application areas. Accordingly, we can see the limitations discussed below. First, procedural languages have been generally used in expressing document behaviors. Their behavior is event-driven and can be described by rules more naturally than by procedures. Second, different languages are used together in implementing the components of active documents. The use of multiple languages leads to a difficulty in maintaining the compatibility of active document processing systems in a heterogeneous environment. Finally, it takes a lot of time to represent and modify the active behaviors of documents in Java or a script language. They are not flexible enough to be used in the dynamic environment of business today. To cope with these limitations, we write active documents in Prolog and XML.

3 Active Questionnaire Model 3.1 Active Questionnaires We can separate logics for processing questions from the engine for processing questionnaires and then include the logics in the questionnaires. In this case, it becomes possible to change the questionnaire without modifying the engine. This kind of questionnaire is referred to as an active questionnaire, and its constituents are shown in Fig. 1. An active questionnaire is composed of questions, rules, a knowledgebase, and a query. Among these, questions are expressed in XML, and rules specify how to process questions in an executable XML, i.e., Executable Rule Markup Language (ERML). A knowledgebase and a query are also expressed in ERML [1, 6].

Fig. 1. The Active Questionnaire Model

3.2 Types of Questions 3.2.1 Control Logics for Questions There are two main categories of questions: closed-ended and open-ended [7]. A closed-ended question allows the respondent to select an answer from a given number of options, whereas an open-ended question asks the respondent to give his or her own answer. To achieve the best response rates, questions should flow logically from one to the next. We have devised control logics for processing questions according to question type, as shown in Table 2 [8, 9]. 3.2.2 Representation of Active Questionnaires Fig. 3 shows a part of a paper questionnaire, which includes two control logics for questions: Fill and Check response. This paper questionnaire is represented in an active questionnaire in Fig. 2. Naturally, answer, fill and checkResponse should have

406

S.-A. Jang, J.-G. Yang, and J.-H.J. Bae Table 2. Control Logics for Questions

Control Logic Check

Compute

Explanation

duplicate response limit add subtract multiply divide

Fill Arrange

question instance

Skip

To check redundancy in responses To get mandatory responses To restrict the number of responses To do addition needed in answering the current question To do subtraction needed in answering the current question To do multiplication needed in answering the current question To do division needed in answering the current question To fill the current question with phrases taken from or based on previous responses To change the order of questions To change the order of answers To control the sequence of questions for each respondent.

answer('3', none) answer('4', none) answer('5', none) 3. Do you have a car? (You have to answer this question.) Yes No 4. (If you select in question 3)

① ②

①

What is the brand of your car? ( ) 5. Are you satisfied with the car "none" that you have? Satisfied Not satisfied checkResponse('3') :- answer('3', Value), Value == none skip('4') :- answer('3', Value), Value == 2 skip('5') :- answer('3', Value), Value == 2 fill('5',Value) :- answer('4', Value)

① ②

Fig. 2. An Active Questionnaire 3. Do you have a car? (You have to answer this question.) Yes No 4. (If you select in question 3) What is the brand of your car? ( ) 5. Are you satisfied with the car "none" that you have? Satisfied Not satisfied

① ② ①

②

①

Fig. 3. A Paper Questionnaire

been translated into ERML in the form of XML, but they remain as Prolog clauses for the convenience of explanation. In the case of this questionnaire, the element consists of user responses, and the element consists of control logics for processing questions appropriate to the types of each question. There is a


407

Prolog rule that directs the questionnaire system to fill the "none" area of question 5 with the answer to question 4. Fig. 4 shows how this rule is expressed in ERML. It is eventually consulted into an inference engine of the questionnaire system.

fill('5',Value) :- answer('4', Value)

:- fill 5 Value , answer 4 Value

Fig. 4. Control Logic Fill in Prolog (left) and in ERML (right)

4 The Framework of Processing Active Questionnaires 4.1 The Structure of the WINAD System In this section we describe a CAI system that adopts active questionnaires. The system is the WINAD and has three components, namely, a questionnaire client, a web server, and a questionnaire designer, as shown in Fig. 5. The Questionnaire Client is composed of the User Interface and the Questionnaire Clerk. The User Interface obtains questions that respondents request, translates them into HTML, and then displays them on a web browser ( -1, -2). It receives the consequences of the control logic for the current question through the Questionnaire Clerk (Send Inference Result), and reflects them in the questionnaire displayed on the web browser. In addition to this, it delivers question objects to the Questionnaire Clerk (Send Question Object). The Questionnaire Clerk hands question numbers and their corresponding answers to the Question Processor in the Web Server ( -1). It also receives question numbers and consequences of control logics for the questions from the Question Processor in the Web Server ( -2). The Web Server is composed of the Question Processor, Control Logic Translator, and Web Container. The Question Processor receives question numbers and answers from the Questionnaire Clerk ( -1). It checks whether or not there are control logics corresponding to the question numbers. If a question has a control logic, it infers the consequence, which is a command for processing questions, and transmits the command to the Questionnaire Clerk. In an active questionnaire, control logics for processing questions are specified by rules and a knowledgebase in XML. The Control Logic Translator changes the representation language of control logics from Prolog to XML and conversely. It is composed of two modules. The XML2Prolog changes the representation of control logics into Prolog in order to be run in the Question

④ ④

⑨

⑨

⑨

408


⑦ ⑧

Processor ( -2, ). The Control Logic Translator receives the rules and knowledgebase from the Control Logic Extractor in the Web Container. Finally, the third component, the Web Container, consists of the Control Logic Extractor and Content Manager. The Control Logic Extractor separates questions, rules, and knowledgebase from an active questionnaire. The questions are delivered to the Content Manager ( -1), and the rules and knowledgebase are delivered to the XML2Prolog module ( -2). The Content Manager gives requested questions to the Questionnaire Client ( -2), or processes the knowledgebase of a questionnaire that a respondent finishes ( , ). Questionnaire Designer creates an active questionnaire with an XML editor. The active questionnaire is stored and distributed as an XML document, which consists of three elements: the question, rule, and knowledgebase ( , ).

⑦ ⑦ ④ ⑩⑪

①②

Fig. 5. System Architecture of WINAD

4.2 Interviewing with Active Questionnaires Fig. 6 shows a questionnaire screen that is displayed by the Questionnaire Client of the WINAD System. The active questionnaire includes three control logics for questions: Skip, Fill, and Check response types. Fig. 7 shows the screen where control logic Skip is performed according to the answer “No” to question 3 and where the questions 4, 5, and 6 are hidden. After the answer “Yes” to question 7, three questions 8, 9, and 10 remain on the screen. In order to implement the Skip function to ignore useless questions, we do not need additional program codes in the Content Manager. This function is specified in the control logic of questions and takes effect on the display when the logic is performed by the Question Processor. Fig. 8 is an example that shows how the control logic Fill is performed, in which previous answers are filled into current questions. Namely, it is the case in which the response for question 1 is filled into the "none" area of question 2. Fig. 9 shows the screen where the control logic Check response is performed to confirm whether or not there has been any response for question 1.


409

Fig. 6. Questionnaire Screen

Fig. 7. Execution of Control Logic Skip

Fig. 8. Execution of Control Logic Fill

Fig. 9. Execution of Control Logic Check response

5 Conclusions and Further Research Computer-assisted interviewing systems are devised to overcome the shortcomings of paper-and-pencil surveys and to provide a convenient environment for respondents to answer questions. However, when a new questionnaire is prepared, the engine for processing questions should be modified or re-implemented because the procedures for processing questions are generally hard-coded in the engines. This means the engine for processing questionnaires depends on the questionnaires. To maintain the engine efficiently, we make it independent of questionnaires using the active document model. In this model, machines can process documents with rules specified in the documents.

410


A questionnaire that follows the active document model, is referred to as an active questionnaire. It consists of questions, rules, a knowledgebase, and a query. Among these, questions are expressed in XML, and rules are control logics that specify how to process questions in ERML. A knowledgebase and a query are also expressed in ERML. In order to examine the usefulness of active questionnaires, we have designed and implemented a web interview system. We have demonstrated that the engine for processing questionnaires can be made independent of questionnaires in the system. The independence implies that there is no need to modify the engine for each questionnaire. Now, we have a plan to improve the WINAD system in three ways; to develop various control logics for processing questions, to enhance the convenience and efficiency of user interactions, and finally, to implement intelligent interviewing procedures. Acknowledgments. This work was supported by the 2009 Research Fund of University of Ulsan.

References 1. Nam, C.-K., Jang, G.-S., Bae, J.-H.J.: An XML-based Active Document for Intelligent Web Applications. Expert System with Applications 25(2), 165–176 (2003) 2. Bethke, A.D.: Representing Procedural Logic in XML. Journal of Software 3(2), 33–40 (2008) 3. Köppen, E., Neumann, G.: Active Hypertext for Distributed Web Applications. In: 8th IEEE International Workshops on Enabling Technologies: Infrastructure for Collaborative Enterprise (1999) 4. Bompani, L., Ciancarini, P., Vitali, F.: Active Documents in XML. ACM SIGWEB News letter 8(1), 27–32 (1999) 5. Werle, P., Jansson, C.G.: Active Documents Supporting Teamwork in a Ubiquitous Computing Environment. In: PCC Workshop 2001 & NRS 01, Nynashamn, Sweden (2001) 6. Jang, S.-A., Yang, J.-G., Bae, J.-H.J., Nam, C.-K.: A Framework for Processing Questionnaires in Active Documents. In: 2009 International Forum on Strategic Technologies (IFOST 2009), pp. 206–208 (2009) 7. Brace, I.: Questionnaire Design: How to Plan, Structure and Write Survey Material for Effective Market Research. Kogan Page Publishers (2008) 8. Jang, S.-A., Yang, J.-G., Bae, J.-H.J.: Design of Questionnaire Logic in Active Documents. In: 32nd KIPS Fall Conference 2009, vol. 16(2), pp. 945–946 (2009) (in Korean) 9. Jang, S.-A., Yang, J.-G., Bae, J.-H.J.: Flow Control of Survey in Active Documents. In: KIISE Korea Computer Congress 2009, vol. 36(1D), pp. 283–288 (2009) (in Korean)

Assessing End-User Programming for a Graphics Development Environment Lizao Fang and Daryl H. Hepting Computer Science Department, University of Regina, Canada [email protected], [email protected]

Abstract. Quartz Composer is a graphics development environment that uses a visual programming paradigm to enable its users to create a wide variety of animations. Although it is very powerful with a rich set of programming capabilities for its users, there remain barriers to its full use, especially by end-users. This paper presents a prototype end-user programming system that is designed to remove the barriers present in the native Quartz Composer environment. The system, called QEUP, is based on earlier work with cogito. It provides direct access to samples of Quartz Composer output without requiring any of the manual programming involved in Quartz Composer. In order to assess the impacts of QEUP, a user study was conducted with 15 participants. Preliminary results indicate that there may be benefit to using QEUP when first learning Quartz Composer, or when learning new capabilities within it.

1

Introduction

The power of the visual programming paradigm, which began with ConMan [1], is evident in the Quartz Composer graphics development environment (GDE) now available on Apple computer systems. This direct-manipulation style of programming makes clear the relationships between different parts, called patches, and it affords alteration of those relationships by changing the connections between patches. See Figure 1 for a view of the programming interface. Other modifications are also possible through other features of the interface. Yet, if the user would like to explore different variations possible from this simple program comprising the 5 patches in Figure 1, he or she must be responsible for all of the changes. This necessity leads to two questions that are of interest here: 1. given an unsatisfactory output, which changes can be made to improve it? 2. after evaluating several alternatives, how can one easily return to, or reprogram, the best output amongst them? Following from the classification developed by Kochhar et al. [2], graphics development environments may be classified as either manual, automated, or augmented. Manual systems require complete involvement of a human to construct A. An et al. (Eds.): AMT 2010, LNCS 6335, pp. 411–423, 2010. c Springer-Verlag Berlin Heidelberg 2010

412

L. Fang and D.H. Hepting

(a) Editor

(c) Patch Creator

(b) Patch Inspector

(d) Viewer

Fig. 1. Quartz Composer, with the sample used throughout this paper

a graphics application; automated systems require no involvement of a human; and augmented systems support some notion of the development process as a collaborative effort between human and computer. Quartz Composer, with its visual programming paradigm, is a manual environment. This paper describes the development and testing of an augmented system, called QEUP (for Quartz Composer End-user Programming) which is based on cogito [3]. As well, QEUP is an end-user programming system, which can be used by end-users with little or no programming experience. According to the spectrum of software-related activities proposed by Fischer and Ye [4], the entry level of visual programming is higher than that of end-user programming. Can additional supports for end-user programming remove the barriers that remain with Quartz Composer, highlighted by the two questions posed earlier? The rest of this paper is organized as follows. Section 2 presents some background work. Section 3 describes the software that is being studied. Section 4 describes the design of the user study. Section 5 describes the results of the user study. Section 6 presents some conclusions and opportunities for future work.

Assessing End-User Programming for a Graphics Development Environment

2

413

Background

Although visual programming systems do enable end-user programming to some extent, the differences between these two areas are highlighted here. 2.1

Visual Programming

Visual programming employs pictorial components to specify programs. In many visual programming environments (VPEs), these pictorial components are connected according to the required data flow. In some other systems, the pictorial components are constructed based on flowcharts. Many VPEs allow users to drag and drop these components. The most exciting feature of visual programming is not the colourful graphical components, but its ability to develop, test and modify in an interactive way [5]. The barriers to visual programming are lower than conventional programming [6], and everyone with some necessary training would be able to manage to create simple programs in a VPE. Visual programming provides an interactive way for users to perform programming. Some systems support real-time, or approximately real-time, computing. Furthermore, some VPEs such as FPL (First Programming Language) [7] and Quartz Composer are well-suited for end-users because the systems eliminate syntactic errors [6]. Some other samples of VPEs include: ConMan [1], LabVIEW [8], Alice [9], and Scratch [10]. However, visual programming has not been widespread. Some researchers [6,11,12] point out reasons, which are summarized as follows: – visual programming is not radical enough to describe dynamic processes – pictorial components in visual programming increase abstraction, because they are symbolic objects – pictorial components waste precious screen real estate – visual programming inhibits details – visual programming does not scale well – visual programming has no place for comments – visual programming is hard to integrate with other programming languages, such as text. 2.2

End-User Programming

End-user programming (EUP) is defined as the activities that end-users, with no or minimal programming knowledge, create functions or programs. EUP also is called end-user modifiability [13,14], end-user computing [15], and end-user development [16]. The proposed scope of EUP varies amongst different researchers. Ye and Fischer [4] defined EUP as activities performed by pure end-users, who only use software rather than develop software. However, Blackwell [17] categorized EUP into five categories: activities based on a scripting language; activities performed in visual programming environments; activities performed in graphical rewrite

414


systems; activities relying on spreadsheet systems; and activities with examplebased programming. The scope of EUP brought forward by Myers, Ko and Burnett [18] covers almost all software-related activities. End-users of EUP systems are those people having limited programming knowledge. The entry level into EUP systems is supposedly relatively low. We propose the following requirements for EUP systems - in the context of GDEs to answer the question of which software systems support EUP. 1. support for creative activities: designers cannot envision the results created by end-users. The system enables end-users to create their own programs. 2. ordinary end-users are able to benefit: the only requirement could be that users have experience of using computers. Some domain-oriented systems, such as MatLab [19], require solid domain knowledge that makes them inaccessible outside of the target domain. In contrast, Microsoft Excel is widely accepted by users from various domains. 3. easy to learn and easy to use: many end-users may not have the patience to learn a complex system. They do not have much time, nor do they want to spend a lot time learning software applications. After a short training period, end-users are able to recognize and comprehend a large percentage of functions provided by the EUP systems. 4. fault-tolerant and interactive: the system should render results without crashing, even though the results could be unreasonable. Maintaining responsiveness will help to engage the end-user.

3

Quartz Composer, cogito and QEUP

Three different software packages were involved in this paper. Each of them is described in more detail here: Quartz Composer, cogito, and QEUP. For each section, a usage scenario describes how a sample, which presents the movement of a cartoon sun, can be refined in each system. The actor in each scenario, named Jimmy, is an end-user with little programming experience. In addition, the implementation of QEUP is described. 3.1

Quartz Composer

Quartz Composer is a visual programming environment as well as a graphics development environment. There are four main windows in Quartz Composer (See Figure 1): the Editor (Figure 1(a)) window for constructing the program; the Patch Inspector (Figure 1(b)) window for setting specific parameter values; the Patch Creator (Figure 1(c)) window for adding new patches to the program; the Viewer (Figure 1(d)) window for displaying the output in real-time. To perform programming, end-users drag and drop patches then connect them by dragging lines from source ports to destination ports. The values produced from the source patch are passed to the destination patch through their ports.


415

Usage Scenario Step 1: Jimmy opens the example file, the Editor and Viewer windows pop up. In the Editor window (Figure 1(a)), he sees 5 boxes connected by lines. Step 2: The cartoon sun is moving in the horizontal direction, and Jimmy wants to add a movement effect in the vertical direction. He selects the LFO patch and clicks the “Patch Inspector” to access the Patch Inspector window. Jimmy changes the value of the Amplitude from 0 to 0.2. Step 3: Jimmy removes the horizontal movement effect by disconnecting the line between the Random patch (top) and the Billboard patch. Step 4: To add new patches to the program, Jimmy drags new patches from the Patch Creator window to the Editor window. Step 5: Jimmy refines the parameter values for the new patches, as described above, to produce a satisfying animation. 3.2

cogito

Hepting [3] developed cogito, which was designed to address drawbacks of traditional visualization tools. In this case, cogito presents the end-user with a variety of outputs, intended to show the breadth of alternatives available by exploring the parameter values of the various patches. Users can iteratively select promising alternatives to establish parameter values for exploration, refine them in the New Space (Figure 2(b)) dialogue box, and generate new alternatives for consideration on the Viewer (Figure 2(a)) window. Usage Scenario Step 1: Jimmy opens the example file, which is then shown in the first cell of the Viewer window (Figure 2(a)).

(a) Viewer Fig. 2. cogito’s user interfaces

(b) New Space

416


Step 2: Jimmy selects the animation in the first cell and the background of the cell turns green. He clicks the “New” button on the bottom, then accesses the New Space dialogue box, which shows what he has selected so far. Jimmy clicks to add values in the “Billboard 1 | inputX” and “Billboard 1 | inputY”. Step 3: He clicks the “OK” button and eight new animations are displayed in the Viewer window (Figure 2(a)). Two of them are interesting, so Jimmy selects them and the backgrounds of their cells turn green as well. Step 4: Jimmy navigates between screens by clicking the arrow buttons and continues selecting other appealing outputs. Step 5: Jimmy clicks the “New” button again to refine his selections. He continues to explore until he decides on his favourite animations. 3.3

QEUP

QEUP is an example-based end-user programming system. Users start with an example, following the bottom-up recursive learning method. Specific examples aid visualization [20] and help users create concrete mental models. Unlike Quartz Composer, which is a manual system, QEUP is an augmented system. In Quartz Composer, users have to track the value of each parameter in order to identify its effect on the final result. However, in QEUP, users are able to set multiple values for each parameter, and the system processes and generates many outputs each time. Users’ attention is shifted from setting values to making decisions and selecting outputs from diverse alternatives. Usage Scenario Step 1: Jimmy selects the example file. The Editor (Figure 3(a)) window is displayed and the example animation is shown (Figure 3(a), circle 4). Patch names are listed in the Patches control list (Figure 3(a), circle 1), along with their parameters and values. Jimmy realizes that new patches must be defined to the system from the Description Configuration dialogue box (Figure 3(b)) before he can explore their parameter values. Step 2: Jimmy begins to edit patch descriptions. These descriptions include enforceable constraints on value ranges and types. Jimmy realizes that all these activities are helping him to learn about the example, and he becomes more confident that he could tackle more complicated configurations quite easily. After finishing the description of a patch, Jimmy realizes that his description is displayed in the Editor window (Figure 3(a), circle 2). Step 3: In the Editor window, he navigates to a parameter, and clicks the “Add” button under the Values control list. In the Add Values (Figure 3(d)) dialogue box that pops up, he realizes that the values he added to his description appear as defaults under the Value List. He sees that he can also input new values manually, or use other patches to generate values. He doesn’t change anything at this time. Step 4: Jimmy clicks “Build” button in order for the system to produce the outputs, based on his choices. In the Viewer (Figure 3(c)) window, he reviews the outputs and selects those which he finds interesting.


(a) Editor

417

(b) Description Configuration

(c) Viewer

(d) Add Values

Fig. 3. QEUP’s user interfaces

Step 5: By clicking the “Explore” button, Jimmy transfers his selections’ data back to the Editor window, where he continues to refine his choices. He is confident about making changes here, because he knows that he is able to load a history of his previous work (Figure 3(a), circle 3). The implementation of the QEUP functionality is divided into four phases. XML is a cornerstone of the system because most documents involved are in XML format. Phase 1: After the example file (.qtz) is successfully loaded, the system converts the document from binary property list to XML format. The XML document is processed and related information, such as patch names, parameter names, and values, is picked up from the document. As well, the Description document (eup description.xml ) is processed. New patches that do not have descriptions are identified by the system.

418


Phase 2: When users write the description for the new patches, the description information is saved into eup description.xml, which will be processed again if the users edit the description. Phase 3: The system begins to produce outputs when the Build button on the Editor window is clicked. During the processes of generating outputs, at first the Template document (eup standarizedSample.xml ) is produced based on the XML format of the example document. The Template document inherits the XML structure of the example document, but removes those data which vary in different outputs. This Template document is created only once for each example file. A cog document is used to save the parameters and their values. The system processes and generates an XML scheme document that includes any restrictions defined by users. Then, the XML scheme document is employed to validate data in cog document. If the validation succeeds, the data in the cog document constructs a set of combinations, each of which will be used to produce an output. Based on the Template document, the first eight combinations are processed. The system produces eight outputs and puts them in the Viewer window. Phase 4: Other outputs will be dynamically produced when users navigate to the next screen. Users might select several satisfying outputs and transfer them to the Editor window for further exploration. During this process, the system maps the selected outputs to corresponding combinations mentioned in Phase 3. Then, the values of the parameters are appended to the Values list in the Editor window.

4

User Study

A user study was conducted in order to assess the impacts of QEUP for Quartz Composer. The between-subjects study was designed to look at the users’ experience with Quartz Composer in three different cases. 4.1

Participants

15 participants took part in the study, with ages ranging from 18 to 32. All of them were students at the University of Regina. Their areas of study were diverse, but most had taken at least one Computer Science course. Regarding their level of programming knowledge, 4 reported a low level; 5 reported a medium level, and 6 reported a high level. 14 participants reported no experience with Quartz Composer, and 1 participant reported low experience on Quartz Composer. They have at most medium experience on visual programming. The participants were randomly grouped assigned to 3 groups. Participants in the first group used Quartz Composer directly (QC Group). Participants in the second group used cogito followed by Quartz Composer (cogito Group). Participants in the third group used QEUP followed by Quartz Composer (QEUP Group).


4.2

419

Materials and Task Design

Each participant also encountered the following documents: a pre-task questionnaire, which covered aspects of participants’ background; a tutorial manual, which provided a standard introduction to the software systems being used; and a post-task questionnaire, which captured aspects of their experience with the software system(s) used by the participant. During each part of the study, participants began with a very simple example (shown in Figure 1). It only had 5 patches: an Image Importer (to load the image), a Billboard (to display the image), an LFO (low-frequency oscillator) and 2 Random patches, used to move the Billboard around the screen. Participants were requested to work on the input example, refine the example, and produce appealing outputs. 4.3

Procedure

Participants completed a consent form and the pre-task questionnaire. The QC Group received some training and then used Quartz Composer, for a period of at most 15 minutes. The cogito Group received some training and then used cogito followed by Quartz Composer, each for a period of at most 15 minutes. The QEUP Group received some training and then used QEUP, followed by Quartz Composer, each for a period of at most 15 minutes. Participants were asked to talk about what they were doing as they navigated the software applications, using a think-aloud protocol. Each participant also completed the post-task questionnaire. All operations using the software were recorded from the computer screen, as well as audio from the participants’ interactions (the participants themselves were not video recorded).

5

Results and Analysis

We analyzed all participants’ performance on Quartz Composer. All data in following tables in this section is collected from performance on Quartz Composer. The participants’ performance is analyzed from three aspects: time to complete the task, attempts to set values, and the primary operation performed by participants. The final outputs in Quartz Composer are determined by parameters’ values and the connection relationship amongst patches. Time to complete the task. In the study, the time spent on Quartz Composer was limited to 15 minutes, but participants were able to stop before that time. Table 1 shows the time spent by all participants from all three groups on Quartz Composer. The total and average time spent is shortest for the QEUP Group. All participants are beginners with respect to Quartz Composer. But participants using QEUP seemed able to more efficiently produce satisfying outputs within Quartz Composer compared to other participants. Users communicate with computers based on two channels: explicit and implicit [21]. The explicit

420

L. Fang and D.H. Hepting Table 1. Time to complete task (min) Group

Participant Times

Total Avg.

QC

10.00 13.28 15.00 15.00 15.00 68.28 13.66

cogito

14.25 15.00 15.00 15.00 15.00 74.25 14.85

QEUP

9.22 10.25 11.00 11.75 15.00 57.22 11.44

Table 2. Setting values on sample Group QC

User Performance Set-value operations

28

30

40

14

30

Set-value kept

20

20

22

7

10

Ratio cogito

1.40 1.50 1.82 2.00 3.00 1.94

Set-value operations

0

17

26

29

61

Set-value kept

0

13

17

13

27

Ratio QEUP

Avg.

1.00 1.31 1.53 2.23 2.26 1.67


0

21

9

18

16

Set-value kept

0

17

7

12

10

Ratio

1.00 1.24 1.29 1.50 1.60 1.32

channel is based on the user interface and the implicit channel relies on the knowledge that the users and the computers both have. Because the participants in this study have no, or very limited, experience on Quartz Composer, their communication with the computer through the explicit channel is not very different. Therefore, the improvement of the communication through the implicit channel may be the factor that results in less time spent in the QEUP Group. The QEUP Group might have gained necessary knowledge after they have some experience on the QEUP system. Attempts to set values. All participants performed programming based on the example. In order to set a suitable value for a parameter, participants might try several times. We calculated the ratio of set-value attempts made to the setvalues attempts kept, which is meant to represent how many trial values were needed in order to successfully customize a parameter. We wanted to evaluate impacts of exposure to cogito and QEUP on Quartz Composer performance. The data in this table is the data related to the example only, operations on new patches were not considered. Table 2 shows data from the three groups. The minimum ratio is 1 and so it is reasonable that this is the ratio for participants who did not set any values.


421

The QEUP Group has the smallest ratio, which may indicate that use of the QEUP system helps participants to make better decisions in Quartz Composer. The significance of using cogito on this aspect is less obvious. As well, Table 2 supports the assertion that the participants’ knowledge of Quartz Composer in the QEUP Group is improved by using QEUP first. Primary operation: setting values or connection/disconnection. Table 3 provides the number of set-value and connection/disconnection operations attempted, from which a ratio is calculated. If ratio is great than 1, setting values seems to be the primary operation. If it is less than 1, connecting/disconnecting patches is said to be the primary operation. The QC Group has no participants whose primary operation was connecting/disconnecting patches. However, there is 1 in the cogito Group and 3 in the QEUP Group. In the QC Group, we observed that participants expended much effort on tracking parameters’ values. However, participants in cogito Group and QEUP Group tend to be less concerned about tracking parameters’ values. Patch Inspector (Figure 1(b)) window for setting values is a window on the second level in Quartz Composer. It is accessible by clicking Patch Inspector button on the Editor Window. Connection/disconnection is the operation in the Editor Window, which is the first level window. The operations of setting values are on a lower level, though they are important. Using QEUP and cogito systems might have capabilities to inspire participants to move from a lower level to a higher level. In addition, the impacts of using QEUP on this aspect are more noticeable. Table 3. Setting values and connection/disconnection Group QC

User Performance Set-value operations

30

28

45

14

30

(Dis)connection operations

29

26

30

5

6

Ratio cogito


16

17

33

29

61

(Dis)connection operations

52

17

21

9

14

Ratio QEUP

Set-value operations (Dis)connection operations Ratio

6

1.03 1.08 1.50 2.80 5.00

0.31 1.00 1.57 3.22 4.36 0

9

21

18

18

29

13

26

6

2

0.00 0.69 0.81 3.00 9.00

Conclusion and Future Work

From analysis of the user study, it appears that QEUP may be a useful complement to Quartz Composer, which may help end-users to acquire knowledge

422


and to form mental models. Those participants starting with QEUP, followed by Quartz Composer, seemed better able to cope with the barriers that emerge in Quartz Composer. It may be that experience with QEUP can provide a headstart in learning the Quartz Composer graphical development environment. Furthermore, it may provide a similar benefit when an end-user wishes to better understand new features and patches within Quartz Composer. A longer-term study would be needed to better assess both potential outcomes hinted at from these results. As well, a comprehensive study would be needed to evaluate whether the QEUP system could be an alternative to replace Quartz Composer. Acknowledgements. The authors wish to acknowledge support from the Natural Sciences and Engineering Research Council of Canada, the Canada Foundation for Innovation, the Saskatchewan Innovation and Science Fund, and the University of Regina.

References 1. Haeberli, P.E.: Conman: a visual programming language for interactive graphics. In: Proc. SIGGRAPH 1988, pp. 103–111. ACM, New York (1988) 2. Kochhar, S.: et al.: Interaction paradigms for human-computer cooperation in graphical-object modeling. In: Proc. Graphics Interface 1991, pp. 180–191 (1991) 3. Hepting, D.: Towards a visual interface for information visualization. In: Proc. Information Visualisation 2002, pp. 295–302 (2002) 4. Ye, Y., Fischer, G.: Designing for participation in socio-technical software systems. In: Stephanidis, C. (ed.) HCI 2007, Part I. LNCS, vol. 4554, pp. 312–321. Springer, Heidelberg (2007) 5. Burnett, M., et al.: Toward visual programming languages for steering scientific computations. IEEE Comput. Sci. Eng. 1(4), 44–62 (1994) 6. Myers, B.A.: Taxonomies of visual programming and program visualization. J. Vis. Lang. Comput. 1(1), 97–123 (1990) 7. Cunniff, N., et al.: Does programming language affect the type of conceptual bugs in beginners’ programs? SIGCHI Bull. 17(4), 175–182 (1986) 8. Johnson, G.W.: LabVIEW Graphical Programming: Practical Applications in Instrumentation and Control. McGraw-Hill School Education Group, New York (1997) 9. Pierce, J.S., et al.: Alice: easy to use interactive 3D graphics. In: Proc. UIST 1997, pp. 77–78. ACM, New York (1997) 10. Resnick, M., et al.: Scratch: programming for all. CACM 52(11), 60–67 (2009) 11. Kahn, K.: Drawings on napkins, video-game animation, and other ways to program computers. CACM 39(8), 49–59 (1996) 12. Brooks Jr., F.P.: No silver bullet essence and accidents of software engineering. Computer 20(4), 10–19 (1987) 13. Fischer, G., Girgensohn, A.: End-user modifiability in design environments. In: Proc. CHI 1990, pp. 183–192. ACM, New York (1990) 14. Girgensohn, A.: End-user modifiability in knowledge-based design environments. PhD thesis, University of Colorado at Boulder, Boulder, CO, USA (1992) 15. Brancheau, J.C., Brown, C.V.: The management of end-user computing: status and directions. ACM Comput. Surv. 25(4), 437–482 (1993)


423

16. D¨ orner, C., et al.: End-user development: new challenges for service oriented architectures. In: Proc. WEUSE 2008, pp. 71–75. ACM, New York (2008) 17. Blackwell, A.F.: Psychological issues in end-user programming. In: End User Development, pp. 9–30. Springer, Netherlands (2006) 18. Myers, B.A., Ko, A.J., Burnett, M.M.: Invited research overview: end-user programming. In: Proc. CHI 2006, pp. 75–80. ACM, New York (2006) 19. Gilat, A.: MATLAB: An Introduction with Applications. New Age (2005) 20. Lieberman, H.: An example based environment for beginning programmers. Instructional Science 14(3-4), 277–292 (1986) 21. Fischer, G.: User modeling in human–computer interaction. User Modeling and User-Adapted Interaction 11(1-2), 65–86 (2001)

Visual Image Browsing and Exploration (Vibe): User Evaluations of Image Search Tasks Grant Strong, Orland Hoeber, and Minglun Gong Department of Computer Science, Memorial University St. John’s, NL, Canada A1B 3X5 {strong,hoeber,gong}@cs.mun.ca

Abstract. One of the fundamental challenges in designing an image retrieval system is choosing a method by which the images that match a given query are presented to the searcher. Traditional approaches have used a grid layout that requires a sequential evaluation of the images. Recent advances in image processing and computing power have made similarity-based organization of images feasible. In this paper, we present an approach that places visually similar images near one another, and supports dynamic zooming and panning within the image search results. A user study was conducted on two alternate implementations of our prototype system, the findings from which illustrate the benefit that an interactive similarity-based image organization approach has over the traditional method for displaying image search results.

1

Introduction

Image search tasks can be divided into two fundamentally different categories: discovery and rediscovery. Within a rediscovery task, the searcher knows precisely what image they are looking for and seeks to either find it in the search results collection, or decide that it is not present. In contrast, when a searcher is performing a discovery task, the mental model of the image for which they are searching is often vague and incomplete. Within the search results collection, there may be many images that match the desired image to various degrees. The primary activities for the searcher in such discovery tasks are browsing and exploration. In this paper, we evaluate how visual image browsing and exploration, as implemented in Vibe, can assist searchers in preforming discovery tasks within the domain of image search. The fundamental premise is that a visual approach to image organization and representation that takes advantage of the similarities between images can enhance a searcher’s ability to browse and explore collections of images. Vibe is an example of a web information retrieval support system (WIRSS) [5]; its purpose is to enhance the human decision-making abilities within the context of image retrieval. The primary method of image retrieval used on the Web is based on keyword search. Search engines merely adapt their document retrieval algorithms to the context of images and present the results A. An et al. (Eds.): AMT 2010, LNCS 6335, pp. 424–435, 2010. c Springer-Verlag Berlin Heidelberg 2010

Vibe: User Evaluations of Image Search Tasks

425

in a scrollable list ranked on query relevance. While list interfaces are easy to use there is limited ability to manipulate and explore search results. To facilitate an exploration of a collection of image search results, Vibe arranges the images by content similarity on a two-dimensional virtual desktop [9,10]. The user can dynamically browse the image space using pan and zoom operations. As the user navigates, an image collage is dynamically generated from selected images. At the broadest zoom level, the images in the collage are those that best represent the others in their respective neighbourhoods, providing a high-level overview of the image collection. As the searcher zooms in toward an image of interest, more images that are visually similar to the area of focus are dynamically loaded. The benefit of this interaction method is that the user has the ability see as little or as much detail as they wish; a single unified interface provides both a high-level overview and details of a subset of the image collection. Two different methods for organizing the collection of images in Vibe are discussed and evaluated in this paper. The original design of Vibe displays images in irregular patterns [9], following a messy-desk metaphor. In a preliminary evaluation of the interface, we found that once searchers zoomed into a particular area of interest in the image space, they sometimes experienced difficulties scanning the irregularly placed images within the display. A potential solution to this difficulty is to align the images in the messy-desk arrangement into a more structured neat-desk layout in order to enhance the ability of searchers to linearly scan the images. This method maintains the similarity-based organization of the images, but relaxes the use of distance between pairs of images to represent a measure of their similarity. Where user productivity and enjoyment are concerned, we feel that the characteristics of Vibe have merit. The results of a user evaluation conducted in a controlled laboratory setting are reported in this paper. The evaluation compares three image search interfaces: messy-desk Vibe, neat-desk Vibe, and a scrollable grid layout similar to that found in Web image search engines. The remainder of this paper is organized as follows. Section 2 provides an overview of image retrieval and organization. Section 3 outlines the specific features of Vibe and the techniques used to construct the similarity-based image organization. Section 4 describes the user evaluation methods, followed by the results of the study in Section 5. The paper concludes with a summary of the research contributions and an overview of future work in Section 6.

2

Related Work

Techniques for finding specific images in a large image database has been studied for decades [2]. Most current Web-based image search engines rely on some form of metadata, such as captions, keywords, or descriptions; the matching of queries to images is performed using this metadata. Manual image annotation is tedious and time consuming, whereas the results of automatic annotation are still unreliable. Hence, methods for performing retrieval using image content directly,

426

G. Strong, O. Hoeber, and M. Gong

referred as Content-based Image Retrieval (CBIR) [7,2], have been extensively studied. While CBIR approaches normally assume that users have clear search goals, Similarity-based Image Browsing (SBIB) approaches cater to users who wish to explore a collection of images, but do not have a clearly defined information need [4]. The challenge of SBIB is to arrange images based on visual similarities in such a way as to support the browsing and exploration experience. This paper investigates whether SBIB techniques, as implemented in Vibe, can improve users’ image searching experience and performance. Several SBIB techniques have been proposed. Torres et al. [11] prescribe ways to enhance CBIR results by browsing them in spiral or concentric ring representations. The position and size of the images vary with their measure of similarity to the query. In Chen et al.’s approach [1], contents of image databases are modelled in pathfinder networks. The result is a branched clustering constructed with reference to the histogram or texture similarity between images. Snavely et al. [8] provide an interesting way to arrange and browse large sets of photos of the same scene by exploiting the common underlying 3D geometry in the scene. The image browsing technique evaluated in this paper is derived from Strong and Gong’s previous work [9,10]. We adopt their idea of organizing images in 2D space by training a neural network. Two alternative approaches to laying out the images are provided and studied.

3

Vibe

The Vibe technique can arrange images in two alternative ways, which are referred to as messy-desk and neat-desk layouts, respectively. Both layouts place images on a 2D virtual desktop so that visually similar images are close to each other. The difference is that images can be positioned at arbitrary locations in the messy-desk layout, but have to be aligned to a grid in the neat-desk layout. Vibe also supports dynamic pan and zoom operations within the image search results space, allowing the searcher to easily browse and explore the images. The rest of this section discusses the methods for generating these two layouts, and the techniques for supporting interactive exploration and browsing. 3.1

Feature Vector Generation

In order to organize images based on similarity, we need to define a way of measuring the similarity between any two images. Here the similarity is computed using the Euclidean distance between two feature vectors, which are extracted from images to represent the salient information. In this paper, the color-gradient correlation is used since it is easy to calculate and offers good organizational performance [10]. To compute the color-gradient correlation for an input image I, we first compute the gradient magnitude lp and gradient orientation θp for each pixel p. We then divide the colour and gradient orientation spaces into Nc and Nθ bins,


427

respectively. Assuming that functions C(p) and Θ(p) give us the colour and gradient orientation bin indices for pixel p, the sum of gradient magnitudes for all pixels belonging to the k th colour and gradient orientation bin can be computed using: mk = lp (1) p∈I∧C(p)×NΘ +Θ(p)=k

where N = Nc × Nθ is the total number of bins. In practice, we set Nc = 8 and Nθ = 8, resulting a 64-dimensional feature vector F (I), and then normalize the final vector. 3.2

Messy-Desk Layout

Given a collection of T images, the messy-desk layout tries to position them on a 2D virtual desktop, so that visually similar images are placed together. This layout is generated by training a Self-Organizing Map (SOM), a process similar to the one discussed in [9]. A SOM is a type of artificial neural network that is trained through unsupervised learning. It is used here to map N-dimensional vectors to 2D coordinate space. SOMs consist of M ×M units, where each unit x has its own N-dimensional weight vector W (x). For dimension reduction we ensure that M × M T , making it possible to map distinct vectors to unique locations in the SOM. The SOM training process requires multiple iterations. During each iteration all images in the collection are shown to the SOM in a random order. When a particular image I is shown, the goal is to find the best match unit B and then update the weight vectors in B’s neighbourhood proportionally based on the distance between B and the neighbouring unit in the SOM. After the SOM converges, the coordinates of the best match unit B(I) for each image I gives us the mapping in 2D. The SOM’s topology preserving property ensures that images that have similar vectors are mapped to locations that are closer to each other, and vice versa. 3.3

Neat-Desk Layout

The messy-desk layout groups visually similar images together, allowing users to quickly narrow down the search results to a small area of the virtual desktop. However, preliminary evaluations found that users sometimes have difficulty locating the exact image they want because the irregular image layout makes it hard to remember which images have already been inspected. To address this problem, we propose another way to organize images, referred to as the neat-desk layout. The neat-desk layout constrains images positions to be aligned to a grid. Since a trained SOM cannot guarantee one image per unit, we cannot simply use a SOM has the same number of units as the grid we want to align the images to. Instead, we generate the neat-desk layout from the messy-desk layout. As shown in Figure 1, given the collection of images and their 2D locations in the

428


left top

right

bottom top

bottom

Fig. 1. Converting from a messy-desk to the neat-desk layout using a k-d tree

messy-desk layout, the k-d tree algorithm is used to arrange the images into a neat-desk layout. The algorithm starts by finding the median value among the horizontal coordinates of all images, and uses this to split the collection into left and right halves. It then computes the median value among the vertical coordinates of images in each half, so that each half is further split into top and bottom quarters. The above two steps are repeated until each node contains at most one image. At the end, all images are contained in the leafs of a balanced binary tree. Based on the position of each leaf, we can assign a unique location to its associated image in the neat-desk layout. In the messy-desk approach, two images that are very similar to one another will be placed in close proximity. The resulting gaps and irregular placement of images provide a good representation of the visual clustering, but make sequential evaluation of images difficult. The neat-desk layout produces a more regular layout, at the expense of losing the visual encoding of the degree of similarity. 3.4

Determining Display Priority

While the above layouts handle the positioning of the images in a collection, it is impractical to display all images at those positions when the collection is large. To facilitate the selection of images to display at run time, we pre-assign priorities to all images. The priority assignment is based on the criteria that the more representative images should have higher priorities to allow them to be selected first. For the messy-desk layout, the images’ priorities are determined using a multiresolution SOM [9]. The bottom SOM, the one with the highest resolution, is obtained using SOM training procedure described in Section 3.2. The upper level SOMs are generated from the lower level ones directly without training. This is done by assigning each unit in an upper level SOM the average weight vector of its children in the lower level SOM. The average weight vector is then used to find the best matching image for each unit in the upper level SOMs. The upper level images represent their neighbourhoods below and are given a higher priority for display. The same principle is applied for the neat-desk layout. The bottom level grid holds all images, each in its assigned location. An upper level grid contains a


429

Fig. 2. The layout of images using messy-desk Vibe (top row) and neat-desk Vibe (bottom row) for the same collection of images at three different levels of zoom. Note the visual similarity of images that are near to one another.

quarter of the grid points, with each point p linking to four child locations in the lower level grid. To select a single image for the grid point p, we first compute the average vector using images mapped to p’s four child locations, and then pick the image that has the vector closest to the average. 3.5

Browsing Interface

Given the images and their mapped locations in either messy-desk or neatdesk layouts, the browsing interface selectively displays images at their mapped locations based on the users’ pan and zoom interactions with the interface [9]. The number of images shown depends on the display resolution, the zoom level, and the user specified image display size. If the system is unable to fit all of the available images inside the viewing area, the ones with higher display priorities are shown. Figure 2 shows the three different levels of zoom for both the messydesk and neat-desk layout methods. Panning is implemented using a mouse drag operation, which translates the current viewing area. Zooming adjusts the size of the viewing area and is achieved using the normal mouse wheel operations. Zooming out enlarges the viewing area and allows users to inspect the overall image layout on the virtual desktop, whereas zooming in reduces the viewing area, making it possible to show the images in a local region in greater detail. It is worth noting that the zooming operation only changes the image display size when room is available (i.e., the view is at the lowest level and there are no “deeper” images); otherwise it is

430


provides a filtering operation that pulls and pushes images into and out of the view area. The browsing interface also provides two ways for adjusting the display size of the images. First, the users can use the combination of the control key and mouse wheel to change the size of all displayed images, which also affects the total number of images that can be shown within the limits of the current view. Secondly, users are able to selectively enlarge an image of interest with a double-click.

4

Evaluation

In order to explore the differences between the traditional grid layout of image search results and the interactive content-based approach implemented in Vibe, a user evaluation was conducted in a controlled laboratory setting. In this study, the messy-desk Vibe (Vibe-m) and the neat-desk Vibe (Vibe-n) are compared to a grid layout (Grid). In order to reduce the interaction differences between the systems being studied, the Grid was implemented as a single scrollable grid (rather than the more common multi-page approach). 4.1

Methods

Although a number of options are available for studying search interfaces [6], we conducted a user evaluation in a laboratory setting in order to obtain empirical evidence regarding the value of the similarity-based approach for image search results representation. The controlled environment of the study allowed us to manage and manipulate the factors that we believed would have an impact on a participants performance and subjective reactions. At the same time, we were also able to ensure that the search tasks each participant performed were the same. The study was designed as a 3 × 3 (interface × search task) between-subjects design. Each participant used each interface only once, and conducted each search task only once. To further alleviate potential learning effects, a GraecoLatin square was used to vary the order of exposure to the interface and the order of the task assignment. Prior to performing any of the tasks, participants were given a brief introduction to the features of each of the three interfaces. A set of three situated search tasks were provided to the participants, for which they used either Vibe-m, Vibe-n, or the Grid. For each task, participants were given a scenario in which they were asked to find five images that were relevant to the described information need (see Table 1). The tasks were chosen to be somewhat ambiguous, requiring the participants to explore the search results in some detail. The images used for all three datasets were obtained from Google Image Search by searching with the corresponding keywords. In addition, the order of images displayed in the Grid follow the order returned by Google search. For each task, measurements of time to task completion, accuracy, and subjective measures were made. Pre-study questionnaires were administered to


431

Table 1. Tasks assigned to participants in the user evaluation Query

Information Need

“Eiffel Tower” Find five images of sketches of the Eiffel Tower. “Notre Dame” Find five images of the stained glass windows of the Notre Dame Cathedral. “Washington” Find five images of Denzel Washington.

determine prior experience with image search, educational background, and computer use characteristics. In-task questionnaires measured perceptions of quality of the search results and ease of completing the task. Post-study questionnaires followed the guidelines of the Technology Acceptance Model (TAM) [3], measuring perceived usefulness and ease-of-use, along with an indication of preference for an image search interface. 4.2

Participant Demographics

Twelve individuals were recruited from the student population within our department to participate in this study. They reported using a wide range of systems for the purposes of searching for images. These included the top search engines (e.g., Google, Bing, and Yahoo), other online services (e.g., Flickr, Picasa, and Facebook), and desktop software (e.g., iPhoto, Windows Photo Gallery, and file browsers). As a result, we can conclude that all of the participants in the study were very familiar with the traditional grid-based approach to image layout.

5

Results

5.1

Time to Task Completion

The average time required to complete the three tasks with the three interfaces are illustrated in Figure 3. Clearly, these results are somewhat varied. For the “Eiffel Tower” and “Notre Dame” tasks, participants performed better using

Fig. 3. Average time to task completion measurements from the user evaluation

432


both versions of Vibe than the Grid. However, which version of Vibe performed better was different between the two tasks. For the “Washington” task, participants performed better using the Grid than either version of Vibe. ANOVA tests were preformed on these measurements to determine whether their differences were statistical significant. Among these results, only three were significant. For the “Notre Dame” task, the time taken to complete the task using Vibe-m was faster than both the Grid (F(1, 7) = 12.4, p < 0.05) and Vibe-n (F(1, 7) = 8.15, p < 0.05). For the “Washington” task, the time to completion using the Grid was faster than Vibe-m (F(1, 7) = 6.49, p < 0.05). For the rest of the pair-wise comparisons, the differences were not statistically significant. For most combinations of tasks and interfaces, there was a high degree of variance in the time to task completion measurement, indicating that the ability to complete the tasks is more a function of the skill and interest of the participant than the interface used to browse, explore, and evaluate the image search results. One aspect of particular note is the situation where the Grid allowed the participants to complete the “Washington” task faster than with either version of Vibe. Within Vibe, the system was effective in grouping images with similar global features, but not very effective in putting together images with similar local features. Since the images that contain people are strongly influenced by the background, these images are not necessarily placed together in Vibe. While participants were able to navigate to a location of interest easily, if they were unable to find enough relevant images in that location (e.g., images of Denzel Washington), they were hesitant to zoom out and continue exploring. As a result, it took them longer to find the images than sequentially searching the image space. Nevertheless, this suggests that the users were able to use the spatial layout information presented in the Vibe interface effectively. As the methods for grouping images based on local features improve, issues such as this will be eliminated. 5.2

Accuracy

After the participants completed the tasks, the five selected images were carefully inspected to verify their relevance to the information need. ANOVA tests across all three tasks indicate that there are no statistically significant differences in the accuracy when using the different interfaces (“Eiffel Tower”: F (2, 11) = 1.29, p = 0.32; “Notre Dame”: F (2, 11) = 1.00, p = 0.41; “Washington”: F (2, 11) = 0.346, p = 0.72). The average number of errors ranged from zero to 0.75. This result indicates that the exploratory nature of Vibe neither helped nor hindered the participants in deciding the relevance of individual images to the search task. 5.3

Subjective Reactions

After each task was complete, participants were asked to indicate their degree of agreement to statements related to the quality of the search results and the ease at which they were able to complete the task (using a five-point Likert


(a) quality of search results

433

(b) ease of the search task

Fig. 4. Average response to statements related to the search tasks

scale where high values indicated agreement). The average responses to these questions are reported in Figure 4. For the “Eiffel Tower” and “Notre Dame” tasks, one can readily see that participants perceived the search results to be of higher quality and the tasks to be easier to perform when using either version of Vibe compared to the Grid. For the “Washington” task, it appears that since there was some difficulty with Vibe being able to organize the local features of people in the images properly, the participants provided similar responses for all three interfaces. The statistical significance of these results were evaluated using pair-wise Wilcoxon-Mann-Whitney tests. Significance was found only for certain comparisons in the “Notre Dame” query. For the quality of search results measure, only the Grid vs. Vibe-n (Z = −2, 055, p < 0.05) comparison was statistically significant. For the ease of search task measure, only the Grid vs. Vibe-m (Z = −2.494, p < 0.05) and Grid vs. Vibe-n (Z = −2.494, p < 0.05) comparisons were statistically significant. Since the data from these in-task questionnaires was rather sparse, questions related to the overall perception of the usefulness and ease of use of the interface were collected in the post-study questionnaire, using the TAM instrument. Since this data was not collected in the context of a particular task, aggregate results of all participants and all TAM statements are shown in Figure 5. WilcoxonMann-Whitney tests were performed on the responses using a pair-wise grouping

Fig. 5. Average response to statements regarding to the usefulness and ease of use of the interface

434


Table 2. Statistical analysis (Wilcoxon-Mann-Whitney tests) of the responses to the TAM questions Grid vs. Vibe-m

Grid vs. Vibe-n

Vibe-m vs. Vibe-n

Usefulness Z = −7.578, p < 0.001 Z = −7.966, p < 0.001 Z = −0.967, p = 0.334 Ease of Use Z = −2.775, p < 0.05 Z = −2.206, p < 0.05 Z = −0.785, p = 0.432

of the interfaces. The results from this statistical measure are reported in Table 2, showing that participants found either version of Vibe more useful and easy to use than the Grid. The differences between Vibe-m and Vibe-n were not found to be statistically significant. 5.4

Preference

At the end of the study, participants were asked to indicate their preference for an image search interface. Four participants indicated a preference for Vibe-m (33%), six for Vibe-n (50%), and two for the Grid (17%). This clearly indicates a high degree of preference for the dynamic layout and interactive features of Vibe. A Wilcoxon signed rank sum test found statistical significance (Z = −2.309, p < 0.05) in the preference of Vibe over the Grid. The preference between the messy-desk and neat-desk layouts was not statistically significant (Z = −0.632, p = 0.53).

6

Conclusions and Future Work

In this paper, we present an interactive visual interface that supports the browsing and exploration of image search results (Vibe). Two different versions of Vibe were created and studied in comparison to the commonly used grid layout. The messy-desk layout version of Vibe places images on a 2D virtual desktop, using the distance between images to represent their similarity. The neat-desk layout adds structure to the image arrangement. Both versions of Vibe provide dynamically generated collages of images, which can be interactively panned and zoomed. As the searcher zooms into an area of interest and more space is created in the view, more images from the search space are dynamically displayed. This interaction results in a filtering and focusing of the search space, supporting the searcher in discovering relevant images. As a result of the user evaluation, we conclude that Vibe can improve the time it takes to find relevant images from a collection of search results. However, there are situations where the overhead of browsing and exploring outweighs the time saved in finding relevant images. Further study is required to examine the boundary conditions for increasing or decreasing searcher performance. During the study, the perception of search results quality and ease of completing the tasks was higher for Vibe than for the grid layout. However, the degree and significance of this result was dependent on the task. By the end of


435

the study (after each participant was exposed to each of the three interfaces), measurements of usefulness and ease of use showed a clear and statistically significant preference for Vibe. These results indicate that the the participants were able to see the value in using Vibe for their image search tasks, even though the time taken to find relevant images was not necessarily improved. Further validation of this outcome was provided by the fact that 83% of the participants preferred to use Vibe over a grid layout. In terms of the differences between the messy-desk and neat-desk layout, no clear conditions were found in this study indicating when one layout method was superior to the other. Whether a participant found one or the other easier to use may simply be a matter of personal preference. However, further study to identify the value of one layout method over the other will be of value.

References 1. Chen, C., Gagaudakis, G., Rosin, P.: Similarity-based image browsing. In: Proceedings of the IFIP International Conference on Intelligent Information Processing, Beijing, China, pp. 206–213 (2000) 2. Datta, R., Joshi, D., Li, J., Wang, J.Z.: Image retrieval: Ideas, influences, and trends of the new age. ACM Computing Surveys 40(2), 1–60 (2008) 3. Davis, F.D.: Perceived usefulness, perceived ease of use, and user acceptance of information technology. Management Information Systems Quarterly 13(3), 319– 340 (1989) 4. Heesch, D.: A survey of browsing models for content based image retrieval. Multimedia Tools and Applications 42(2), 261–284 (2008) 5. Hoeber, O.: Web information retrieval support systems: The future of Web search. In: Proceedings of the IEEE/WIC/ACM International Conference on Web Intelligence - Workshops (International Workshop on Web Information Retrieval Support Systems), pp. 29–32 (2008) 6. Hoeber, O.: User evaluation methods for visual Web search interfaces. In: Proceedings of the International Conference on Information Visualization, pp. 139–145. IEEE Computer Society Press, Los Alamitos (2009) 7. Smeulders, A., Worring, M., Santini, S., Gupta, A., Jain, R.: Content-based image retrieval at the end of the early years. IEEE Transactions on Pattern Analysis and Machine Intelligence 22(12), 1349–1380 (2000) 8. Snavely, N., Seitz, S.M., Szeliski, R.: Photo tourism: Exploring photo collections in 3d. In: Proceedings of the ACM International Conference on Computer Graphics and Interactive Techniques, pp. 835–846 (2006) 9. Strong, G., Gong, M.: Browsing a large collection of community photos based on similarity on GPU. In: Proceedings of the International Symposium on Advances in Visual Computing, pp. 390–399 (2008) 10. Strong, G., Gong, M.: Organizing and browsing photos using different feature vectors and their evaluations. In: Proceedings of the International Conference on Image and Video Retrieval, pp. 1–8 (2009) 11. Torres, R.S., Silva, C.G., Medeiros, C.B., Rocha, H.V.: Visual structures for image browsing. In: Proceedings of the International Conference on Information and Knowledge Management, pp. 49–55 (2003)

Contextual Recommendation of Social Updates, a Tag-Based Framework Adrien Joly1,2 , Pierre Maret3 , and Johann Daigremont1 1

3

Alcatel-Lucent Bell Labs France, Site de Villarceaux, F-91620 Nozay, France [email protected] 2 Université de Lyon, CNRS, INSA-Lyon, LIRIS, UMR5205, F-69621, France Université de Lyon, Laboratoire Hubert Curien, UMR CNRS 5516, F-42000 Saint-Etienne, France [email protected]

Abstract. In this paper, we propose a framework to improve the relevance of awareness information about people and subjects, by adapting recommendation techniques to real-time web data, in order to reduce information overload. The novelty of our approach relies on the use of contextual information about people’s current activities to rank social updates which they are following on Social Networking Services and other collaborative software. The two hypothesis that we are supporting in this paper are: (i) a social update shared by person X is relevant to another person Y if the current context of Y is similar to X’s context at time of sharing; and (ii) in a web-browsing session, a reliable current context of a user can be processed using metadata of web documents accessed by the user. We discuss the validity of these hypothesis by analyzing their results on experimental data.

1

Introduction

On Social Networking Services (such as Facebook1 , Twitter2 , LinkedIn3 ) and other collaboration software, people maintain and create new social ties by sharing personal (but not necessarily private) social updates regularly to their community, including status messages and bookmark notifications. As depicted on Figure 1, a social update is a short message sent to a group of interested persons (e.g. a community). It can consist of a one-sentence news or question, an anchor, a picture, or a comment, to share their current thoughts, activities, intentions and needs. On most of these tools, social updates are not meant to be consumed in a push fashion, i.e. like emails, that are aimed at specific recipients and stacked in their inboxes. Instead, community members can go through the list of short social 1 2 3

http://www.facebook.com/ http://www.twitter.com/ http://www.linkedin.com/

A. An et al. (Eds.): AMT 2010, LNCS 6335, pp. 436–447, 2010. c Springer-Verlag Berlin Heidelberg 2010

Contextual Recommendation of Social Updates

437

Fig. 1. A status update from Twitter, and a bookmark update from Delicious

updates of the people or subjects (e.g. hashtags on twitter) they follow, to get a quick feeling of awareness about those they care about. However, as the number of people and subjects being followed increases, the time required to get through to the social updates they emit also increases, causing a loss of productivity. Additionally, as social updates are broadcast in real-time, they create frequent interruptions that can reduce people’s ability to focus on a demanding task, especially when the social updates are not relevant for this task (because it would induce a costly cognitive switch). In response to this emerging problem of information overload, we propose a framework to rank social updates according to real-time distances in-between users’ contexts, which are processed on social and meta-descriptions of the web documents the users are looking at. The two underlying hypothesis that we are supporting in this paper are: (i) a social update shared by person X is relevant to another person Y if the Y ’s current context is similar to X’s context at time of sharing; and (ii) in a web-browsing session, a reliable current context of a user can be processed using a combination of tags and metadata of web documents accessed by the user. In the next section, we motivate our approach by explaining how contextawareness and current web techniques can be leveraged to improve awareness. In section 3, we survey existing work related to our problem. In section 4, we describe our contextual recommendation framework to provide relevant social updates to people. In section 5, we present our experimental setup and results to evaluate the human response to this approach. We will then discuss these results and propose future work.

2

Motivation

Vyas et al. proved [18] that informal ties between co-workers are essential to improve awareness, and thus better collaboration. In previous studies [9,8], we have identified that contextual information about users could be leveraged to assist the sharing of social updates, and thus maintain these ties, while reducing interruptions. Context was defined by Dey [5] as ”information that can be used to characterize the situation of an entity. An entity is a person, place, or object that is considered relevant to the interaction between a user and an application, including the user and applications themselves”. In most context-aware applications, researchers have been relying on sensors to extract contextual information. As tagging becomes a common practice on the Internet, rich contextual information can also emerge from human-generated content.

438

A. Joly, P. Maret, and J. Daigremont

Despite some semantic issues related to ambiguity of terms combined in folksonomies, the increasing amount of tags given by Internet users on digital resources [12,11] (e.g. web pages tagged on delicious4 ) have become good indexing features for recommender systems [7,15]. With the growing use of twitter and geotagging applications in mobility, tags are now emerging from places, events and other real-world entities [13], which implies exciting opportunities to create new ambient intelligence, ambient awareness, augmented reality, and other social applications.

3

Background

To the best of our knowledge, the closest existing solution to our problem is a web-based service and mobile application called My6sense 5 . This software can filter entries from RSS and other feeds (including social updates) according to the user’s preference. This content-based filtering technique relies on a profile which contains the user’s subjects of interest, and this profile is continuously evolving by tracking which entries are consulted by the user. Similarly, SoMeONe [1] is a collaboration tool that can recommend contacts (and their selection of web resources) to users by identifying communities of interest around topics, using collaborative filtering. The names of users’ bookmark categories are leveraged as topics. groop.us [3] applied the same approach while relying on tags attributed to web pages by users (folksonomies from a social bookmarking website) instead of hierarchical categories. In these approaches, recommendations are based on documents that were explicitly selected and shared (bookmarked) by users. Despite the evolving design of user profiles, the filtering is not adaptive to the user’s current context. It is possible to provide collaboration opportunities by recommending people that are currently browsing similar documents [4,2], based on a TF-IDF analysis [17] of their content, users’ context being represented by weighted term vectors. These recommendations can also include some information about people’s current activities, as identified by a software module that tracks users’ actions on their computer (e.g. chat sessions, office documents being edited, etc...) [6]. However these efforts don’t leverage tags proposed by web users. In the PRUNE framework [10], contextual entities (e.g. person, place or resource) and events can be extracted from heterogeneous sources like RSS feeds, web APIs and manual user entry. The Notes that Float application leverages this framework to attach such contextual information to notes added by the user, so that their individual visibility depends on their relevance with the user’s current content, which relies on their similarity with the context at the time these notes were added. However, we have found no evidence that tags were leveraged in this application. Moreover, previous collaborative systems imply potential privacy issues. 4 5

http://del.icio.us/ http://www.my6sense.com/


4

439

Contextual Recommendation Framework

After having identified that contextual information about people can be leveraged to further describe the documents they are browsing/editing, and thus to recommend these documents to people that are in a similar context, we have reviewed several techniques and applications that are relevant for ranking documents. In this section, we study the case of enterprise employees working on computers, then we present a framework and software implementation of an adapted social update recommender system which considers web-browsing context as a relevance criteria, and leveraging their tags as features. 4.1

Case Study

As a motivating case, we propose to consider an enterprise environment where employees work on individual networked computers. They don’t know everybody in the enterprise, and are possibly spread across several offices in several cities, or even different countries. Such organizations traditionally rely on hierarchies of managers to coordinate the efforts of workers and transfer information to the relevant parties. We propose an internal social networking tool that allows every worker to share and retrieve relevant information about the current interests and status of their colleagues, while reducing unnecessary task interruptions and network maintenance time. This system will rely on various stream/feed to leverage personal current thoughts, activities, intentions and questions, and must respect their privacy (e.g. case of private browsing). 4.2

Contextual Tag Clouds

As users are working on computers in the case study presented above, most contextual information about their current activity can be extracted from the software they use (e.g document edition and viewing). In this study, we assume that descriptions about the web sites they are currently browsing (e.g. to find some reference on an ongoing task), can provide clues on the user’s current activity. Users’ context can thus be modeled as a set of weighted terms, combining metadata and tags about these browsed web pages. As these terms can potentially reveal private or confidential information, users must be able to quickly visualize and edit them before submitting to a recommender system. We propose

Fig. 2. Sample contextual tag cloud: a set of words describing one’s context

440


the name of ”Contextual Tag Clouds” to refer to these human-readable contexts based on a weighted combination of tags and other descriptive terms, as depicted on Figure 2. 4.3

Data Flow and User Interaction

As depicted on Figure 3, contextual information is extracted from usermanipulated content (in our case, descriptions of web pages currently browsed) by sniffers running on every user’s computer. For privacy control reasons, no contextual information will be ever sent to any remote party without user confirmation. Contextual information is represented by a set of weighted keywords, and represented as a tag cloud. Also running on the user’s computer, an aggregator gathers events from all these sniffers, and queries several web services to generate weighted tags, in order to combine them in a contextual tag cloud that represents the user’s current context.

Fig. 3. Overview of the contextual recommendation loop

When posting a social update (e.g. a tweet ), the user can attach his/her current contextual tag cloud, so that the contextual filter (i.e. a recommender system running in the infrastructure) can match it with other users’ last contextual tag cloud, using a relevance function. This social update will then be recommended to users whose last contextual tag cloud is relevant to the one attached to the social update. That way, every user gets a dynamic (near real-time) list of recent social updates, sorted by decreasing relevance score as they browse the web. Like with regular social networking and microblogging services, these short social updates can be quickly read by users to remain aware of relevant activities going on in their communities. They can also decide to reply to social updates or to call their author.


4.4

441

Ranking Model

The theoretical framework that we designed to solve our relevance ranking problem relies on a vector space model, five weighting functions, an aggregation operator and a relevance function. The weighting functions are equivalent to context interpreters [5]: they transform raw data (in our case, URLs of web documents being browsed by the user) into higher-level information (contextual tags) by abstraction. The contextual tag cloud model is equivalent to the vector space model proposed by Salton [17]. Notice that, in this paper, the word tag refers to terms, whatever their origin. Traditionally, a set of terms resulting from the analysis of a document d is formalized as a vector of weights v(d) = [w(t1 , d), w(t2 , d), ..., w(tN , d)] attributed for each term t = [t1 , t2 , ..., tN ∈ R]. The specificity of our model lies in the combination of five functions applied on browsed documents (and their crowd-sourced description: tags) to compute the weights: 1) the Metadata function counts occurrences of term t with different coefficients (α, β, γ), depending of the position of this term in document d’s metadata: w1 (t, d) = α ∗ |t ∈ Td | + β ∗ |t ∈ Kd | + γ ∗ |t ∈ Dd | where |t ∈ Td | is the number of occurrences of the term t in the title of the document d, |t ∈ Kd | in its keywords set, and |t ∈ Dd | in its description text. 2) the SearchQuery function counts the number of occurrences of term t in a search query Qd , when the analyzed document d contains a list of search results: w2 (t, d) = |t ∈ Qd | 3) the DomainNames function adds the domain names (including subdomains) Nd from document d’s URL as terms: w3 (t, d) = |t ∈ Nd | 4) the SocialBookmarks function counts the number of people who publicly bookmarked the document d using the term t as tag: w4 (t, d) =

tag(p, d, t)

p∈P

where each person p is in the set of people P that are using this bookmarking service, and where tag(p, d, t) has a value of 1 or 0, whether or not this person p bookmarked the document d using term t as a tag.

442


5) the SemanticAnalyzer function counts the number of occurrences of semantically-defined entities (i.e. concepts and instances) that are represented by the term t, when they are identified in the document d: w5 (t, d) = |t ∈ Rd | Rd = [∀e ∈ Ed , repr(e)] where repr(e) is the textual representation of a semantic entity e, Rd is the set of textually represented entities Ed found in the document d. This function is further described in the next part of this section. Additionally, we define an aggregation operator and a relevance function that leverage the vectors resulting from the weighting functions above: The aggregation operator is the addition of given weighted term vectors, after their individual normalization. The normalized form v of vector v conforms N to t=t1 vt = 1, with weight values vt ∈ R in the range [0, 1]. Thus, the aggregation operator applied to a set of vectors V = [v1 , v2 , .., vM ] acts as the following function: M aggr(V ) = vt t=1

The relevance function between normalized vectors, like in traditional vectorbased models, relies on cosine similarity. Thus, the relevance of a tag cloud vector R with another tag cloud vector S is computed by: relevance(R, S) =

R·S RS

which returns a relevance score r ∈ R in the range [0, 1], 1 being the maximum relevance score (i.e. contextual equality). 4.5

Software Implementation

The framework described above was designed as a modular architecture, according to the data flow depicted in Figure 3, in which software modules communicate through RESTful HTTP requests. In this section, we present the implementation of these modules: – A Firefox extension6 acts as a context sniffer and a notifier. For sniffing, it hooks on the browser’s events related to opening, closing and switching web pages, and transmits these events with the corresponding URLs to the local Context Aggregator for processing. At the end of the flow, the recommended social updates are displayed in the side-bar of the browser. 6

http://www.firefox.com/


443

– The Context Aggregator handles local events with their attached contextual information, and runs weighting functions on this information to produce an aggregated (and thus normalized) contextual tag cloud for the Contextual Filter, using the aggr() function defined in the previous section. The weighting functions are implemented as five interpreters that turn URLs into contextual clouds (i.e. weighted term vectors). The Metadata interpreter parses the title, description and keywords elements from the HTML source code of each web page to produce the corresponding weighted terms, with the following parameter values: α = 50 per term appearing in the title, β = 10 in the keywords field, and γ = 1 in description field. The SearchQuery interpreter extracts query terms from Google Search7 result pages. The SocialBookmarks interpreter gathers tags given by users about a web page, when existing on the Delicious social bookmarking service. The SemanticAnalyzer gathers textual representations of semantic entities that were identified in the web page, thanks to the SemanticProxy web service8 . – The Contextual Filter receives contextual clouds gathered and interpreted by users’ aggregator, computes relevance scores between them using the relevance() function, and recommend best-ranked social updates to each user (through their notifier ). Social updates are gathered by subscription to the users’ declared third-party social feeds/streams (e.g. their Twitter account). This software ecosystem is functional and gives a good sense of the benefits of our approach. In the next section, we present an evaluation of the underlying framework.

5

Evaluation

In order to evaluate the validity of our hypothesis on relevance of contextually recommended social updates, we gathered browsing logs and social updates from 8 volunteers during one week, ran our algorithms on these logs to generate 1846 contextual clouds (every 10 minutes), and asked the volunteers to rank the quality of a selection of social updates. In this section, we define the experimentation plan we followed, explain its setup, then discuss the results obtained. 5.1

Experimentation Plan

The evaluation of our hypothesis relies on two measures: (i) the relevance of social updates with the context of their author at the time of posting, and (ii) their relevance for other users in similar contexts. As the quality of recommendations is to be evaluated by users with their own browsing behavior, implied contexts, and own social updates, we did not rely on existing evaluation data sets such as the ones from TREC, nor follow a scenario-based experiment. 7 8

http://www.google.com/ http://semanticproxy.opencalais.com/

444


During one week, volunteers browsed web pages using Firefox and produced social updates (i.e. shared statuses and bookmarks), while the provided sniffing extension was logging the required browsing events to a local database. At the end of this period, they were proposed to review these log entries, so that they could remove privacy-critical entries when needed (e.g. private activities, and other noisy data that is irrelevant to this study), and then send us their log. Afterwards, we ran our algorithms on the browsing logs and social updates provided by volunteers, to produce personalized survey forms containing ranked recommendations for each volunteer. We asked each volunteer to fill two personalized surveys. In the first survey, we asked volunteers to rate the perceived relevance of three random social updates with five contextual clouds generated from their own web browsing data. For each context, only one of the proposed social updates was actually a well-ranked match. In order to support them in remembering those contexts, we provided the list of web pages that were being browsed by the volunteer at that time. In the second survey, volunteers rated the relevance of their own social updates with their contextual tag cloud at the time of posting. 5.2

Experimental Setup and Process

In this section we provide the process and parameters that we set to generate these personalized surveys from the logs provided by volunteers. Because the experiment was not interactive, we indexed contextual clouds and social updates on a common time line with a period of 10 minutes. Contextual clouds are generated from the list of URLs involved in a web browsing event, i.e. when the page was opened, selected or closed. Indexing a social update consists of associating it with the contextual cloud of the last context snapshot at the time of posting this update. If there is no known context information in the previous snapshot, we use the one before the previous. Every indexed contextual cloud is processed to split multiple-word tags, cleaned from punctuation and other non-literal characters, filtered against a stop-words list, and then normalized so that the sum its tags’ weights equals 1. Only the first 20 tags (with highest weights) are displayed to volunteers. As shown on Figure 2, a contextual tag cloud can contain diverse kinds of terms, such as words in various languages, word combinations and acronyms. Then, we ran the recommendation algorithm on the contextual and social indexes in order to produce a relevance matrix for each participant. In order to generate a participant’s personalized survey, we selected 5 heterogeneous contexts (i.e. the most dissimilar to each other) that were matched (by the recommender) with at least one highly-ranked social update. The second survey was simply generated by correlating users’ social updates with their corresponding context. 5.3

Results

As stated above, the results are twofold: we gathered scores given by every participant on (i) the relevance of social updates with the context of their posting, and (ii) the relevance of social updates for other people with similar contexts.


445

Relevance of contextualized social updates: In order to measure the consistency of contextual clouds as reference documents for recommending social updates, we asked the participants to rate the relevance of each of their own social updates (e.g. their tweets and bookmarks) to the contextual cloud representing their current situation at time of posting/sharing. Over a total of 59 social updates, their authors rated an average relevance to context of 50.3%. The following distribution of ratings is observed: 19 social updates were ranked 1 (low relevance), 10 were ranked 2, 14 were ranked 3, and 16 were ranked 4 (high relevance). These social updates are gathered from several social streams: 54% are status updates posted on Twitter, 29% are bookmarks instantly shared through Delicious. By further analyzing these specific types of social streams, we discovered an average relevance score of 71% for shared bookmarks, and 38% for status updates from Twitter. It is natural that new bookmarks are more relevant to their context, as the web document that is bookmarked is usually being browsed by the user, and thus represented in the corresponding contextual clouds. Concerning status updates, Naaman et al. [14] proved that only 41% of social updates from twitter are actual statuses about the current activity of the person (categorized as ”me now” by the authors). The similarity of this proportion with our average contextual relevance score for status updates gives some proof, although preliminary, about the consistency of our results. Relevance of recommendations: As explained in the previous section, social updates proposed to users are voluntarily not all relevant. Our goal is to observe a correlation between the relevance scores given by participants and the rankings computed by the system. Thus, we rely on a Mean Percentage Error (based on MAE, Mean Absolute Error) to define the following accuracy function: Q |relevance(Cq , Uq ) − rating(Cq , Uq )| accuracy = 1 − q=1

in which, for each proposed social update q, relevance(Cq , Uq ) is the relevance score of the social update Uq with the contextual tag cloud Cq , as evaluated by the ranking algorithm. Whereas, rating(Cq , Uq ) is the actual relevance score, as given by the volunteer. Both scores are values in the range [0, 1], represented as percents. As rating() scores are given by volunteers in the [1, 4] grade rating range, they are converted to percents with the following formula: rating =

(grade − 1) 3

We observed an average accuracy value of 72%. As a natural behavior of recommender systems, the best-ranked ratings (mostly in Rank 3 ) are slightly overestimated by the recommendation algorithm, whereas low relevance ratings (Rank 1 ) given by participants are higher than expected. From the list of relevance ratings expected by the recommendation system, 63% are low ranked (Rank 1 ), whereas 19% are medium-high (Rank 3 ). The high

446


number of low-ranked scores and the medium ranking of better scores expected by the algorithm reveals that highly similar contextual clouds were rare in our small scaled experiment. By increasing the number of participants, more similar contexts would be found, thus the average scores would naturally increase.

6

Conclusion

In this paper, we proposed a theoretical framework, a privacy-aware implementation and its evaluation to rank social updates by contextual relevance, for reducing information overload. Through the analysis of experimental results, we evaluated a combined weighting scheme based on social and meta-descriptions of web pages being accessed by the user, as a contextual criteria for recommending relevant social updates. This study explores the potential of our novel recommendation approach based on contextual clouds. Despite the small scale of this preliminary experiment, our results are promising. The average accuracy of recommended social updates: 72%, is significant for a web recommender system. We observed that the relevance perceived by users increases as social updates reflect the current activity of their authors. In order to improve the performance of our system, we intend: – to improve the quality of context with emergent semantics of tags [16]. – to broaden the range of context, by developing additional context sniffers, including documents, and physical context information from mobile devices. – to find more precise relevance factors between specific types of social updates and contextual properties, after having carried out a higher-scale experiment. – and to improve the scalability of the system when used simultaneously by numerous users (currently: O(n2 ) complex), e.g. using tag clustering.

References ´ 1. Agosto, L.: Optimisation d’un Réseau Social d’Echange d’Information par Recommandation de Mise en Relation. Ph.D. thesis, Université de Savoie, France (2005) 2. Bauer, T., Leake, D.B.: Real time user context modeling for information retrieval agents. In: CIKM 2001: Proceedings of the Tenth International Conference on Information and Knowledge Management, pp. 568–570. ACM, New York (2001) 3. Bielenberg, K., Zacher, M.: Groups in social software: Utilizing tagging to integrate individual contexts for social navigation. In: Digital Media, p. 120. University Bremen. Master of Science in Digital Media, Bremen, Germeny (2005) 4. Budzik, J., Fu, X., Hammond, K.: Facilitating opportunistic communication by tracking the documents people use. In: Proc. of Int. Workshop on Awareness and the WWW. ACM Conference on CSCW 2000, Philadelphia, Citeseer (2000) 5. Dey, A.K.: Providing Architectural Support for Building Context-Aware Applications. Ph.D. thesis, Georgia Institute of Technology (2000) 6. Dragunov, A., Dietterich, T., Johnsrude, K., McLaughlin, M., Li, L., Herlocker, J.: TaskTracer: a desktop environment to support multi-tasking knowledge workers. In: Proceedings of the 10th International Conference on Intelligent Uer Interfaces, pp. 75–82. ACM, New York (2005)


447

7. Hotho, A., J¨ aschke, R., Schmitz, C., Stumme, G.: Information retrieval in folksonomies: Search and ranking. In: The Semantic Web: Research and Applications, pp. 411–426 (2006) 8. Joly, A.: Workspace Awareness without Overload: Contextual Filtering of Social Interactions. In: Smart Offices and Other Workspaces, Workshop of the Intelligent Environments 2009 Conference, Ambient Intelligence and Smart Environments, pp. 297–304. IOS Press, Amsterdam (2009) 9. Joly, A., Maret, P., Daigremont, J.: Context-Awareness, the Missing Block of Social Networking . International Journal of Computer Science and Applications 4(2) (2009), Special Issue on Networking Mobile Virtual Knowledge 10. Kleek, M.V., Karger, D.R., Schraefel, M.C.: Watching through the web: Building personal activity and Context-Aware interfaces using web activity streams. In: Proceedings of the Workshop on Understanding the User - Logging and Interpreting User Interactions in Information Search and Retrieval (UIIR-2009), in Conjunction with SIGIR-2009, Boston, MA, USA (2009) 11. Marlow, C., Naaman, M., Boyd, D., Davis, M.: Position paper, tagging, taxonomy, flickr, article, toread. In: Collaborative Web Tagging Workshop at WWW 2006, 31–40 (2006), http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.74.8883 12. Mathes, A.: Folksonomies-cooperative classification and communication through shared metadata. Computer Mediated Communication (2004) 13. Naaman, M., Nair, R.: ZoneTag’s collaborative tag suggestions: What is this person doing in my phone? IEEE Multimedia 15(3), 34–40 (2008) 14. Naaman, M., Boase, J., Lai, C.: Is it really about me? message content in social awareness streams. In: Proceedings of the 2010 ACM Conference on Computer Supported Cooperative Work, pp. 189–192. ACM Press, Savannah (2010) 15. Niwa, S., Doi, T., Honiden, S.: Web page recommender system based on folksonomy mining for itng 2006 submissions. In: Third International Conference on Information Technology: New Generations, ITNG 2006, pp. 388–393 (2006) 16. Rattenbury, T., Good, N., Naaman, M.: Towards automatic extraction of event and place semantics from flickr tags. In: SIGIR 2007: Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 103–110. ACM, New York (2007) 17. Salton, G., McGill, M.J.: Introduction to Modern Information Retrieval. McGrawHill, Inc., New York (1986) 18. Vyas, D., Van De Watering, M., Eliëns, A., Van Der Veer, G.: Engineering Social Awareness in Work Environments. In: Stephanidis, C. (ed.) UAHCI 2007 (Part II). LNCS, vol. 4555, pp. 254–263. Springer, Heidelberg (2007)

Semantic Web Portal: A Platform for Better Browsing and Visualizing Semantic Data Ying Ding1, Yuyin Sun1, Bin Chen2, Katy Borner1, Li Ding3, David Wild2, Melanie Wu2, Dominic DiFranzo3, Alvaro Graves Fuenzalida3, Daifeng Li1, Stasa Milojevic1, ShanShan Chen1, Madhuvanthi Sankaranarayanan2, and Ioan Toma4 1

School of Library and Information Science, Indiana University 2 School of Computing and Informatics, Indiana University 47405 Bloomington, IN, USA 3 Tetherless World Constellation, Rensselaer Polytechnic Institute, NY, USA 4 School of Computer Science, University of Innsbruck, Austria {dingying,yuysun,binchen,katy,djwild,daifeng,madhu, yyqing,chenshan}@indiana.edu, {dingl,agraves,difrad}@cs.rpi.edu, {ioan.toma}@uibk.ac.at

Abstract. One of the main shortcomings of Semantic Web technologies is that there are few user-friendly ways for displaying, browsing and querying semantic data. In fact, the lack of effective interfaces for end users significantly hinders further adoption of the Semantic Web. In this paper, we propose the Semantic Web Portal (SWP) as a light-weight platform that unifies off-the-shelf Semantic Web tools helping domain users organize, browse and visualize relevant semantic data in a meaningful manner. The proposed SWP has been demonstrated, tested and evaluated in several different use cases, such as a middle-sized research group portal, a government dataset catalog portal, a patient health center portal and a Linked Open Data portal for bio-chemical data. SWP can be easily deployed into any middle-sized domain and is also useful to display and visualize Linked Open Data bubbles. Keywords: Semantic Web data, browsing, visualization.

1 Introduction The current Web is experiencing tremendous changes to its intended functions of connecting information, people and knowledge. It is also facing severe challenges in assisting data integration and aiding knowledge discovery. Among a number of important efforts to develop the Web to its fullest potential, the Semantic Web is central to enhancing human / machine interaction through the representation of data in a machine-readable manner, allowing for better mediation of data and services [1]. The Linked Open Data (LOD) initiative, led by the W3C SWEO Community Project, is representative of these efforts to interlink data and knowledge using a semantic approach. The Semantic Web community is particularly excited about LOD, as it marks a critical step needed to move the document Web to a data Web, toward enabling powerful data and service mashups to realize the Semantic Web vision. A. An et al. (Eds.): AMT 2010, LNCS 6335, pp. 448–460, 2010. © Springer-Verlag Berlin Heidelberg 2010

Semantic Web Portal: A Platform for Better Browsing and Visualizing Semantic Data

449

The Semantic Web is perceived to lack user-friendly interfaces to display, browse and query data. Those who are not fluent in Semantic Web technology may have difficulty rendering data in an RDF triple format. Such perceived lack of user-friendly interfaces can hinder further adoption of necessary Semantic Web technologies. D2R server or various SPARQL endpoints display query results in pure triple formats such as DBPedia (e.g., displaying the resource Name: http://dbpedia.org/page/Name) and Chem2Bio2RDF (e.g., displaying the SPARQL query result on “thymidine” as http://chem2bio2rdf.org:2020/snorql/?describe=http%3A%2F%2Fchem2bio2rdf.org% 3A2020%2Fresource%2FBindingDBLigand%2F1):they aren’t, however, intuitive and user friendly. Enabling user-friendly data displays, browsing and querying is essential for the success of the Semantic Web. In this paper, we propose a lightweight Semantic Web Portal (SWP) platform to help users, including those unfamiliar with Semantic Web technology, allowing all users to efficiently publish and display their semantic data. This approach generates navigable faceted interfaces allowing users to browse and visualize RDF triples meaningfully. SWP is aligned with similar efforts within medical domains funded by NIH in the USA toward the facilitation of social networking for scientists and facile sharing of medical resources. The main architecture of the SWP is based upon Longwell (http://simile.mit.edu/wiki/Longwell_User_Guide) and the Exhibit widget (http:// simile-widgets.org/exhibit/) from MIT’s SIMILE project (http://simile.mit.edu/). We further extend the system by adding Dynamic SPARQL Query module, Customized Exhibit View module, Semantic Search module and SPARQL Query Builder module to enhance the functionality and portability of the system. This paper is organized as follows: Section 2 discusses related work; Section 3 introduces the SWP infrastructure; Section 4 discusses and exemplifies portal ontology; Section 5 demonstrates four use cases for deploying SWP; Section 6 evaluates and compares SWP to related systems, and; Section 7 presents future work.

2 Related Work Research on Semantic Web portals began fairly early, in the nascent 2000s. A number of Semantic Web portal designs and implementations were published in research literature such as SEAL (SEmantic portAL) [2] and Semantic Community Portal [3]. Lausen et al [4] provided an extensive survey on a selection of Semantic Web portals published before 2005. Many research groups are currently maintaining their group portals using Semantic Web technologies. For example, Mindswap.org was deployed as “the first OWL-powered Semantic Web site” [5] and Semantic Mediawiki [6] has been used to power several groups’ portals, such as the Institute of Applied Informatics and Formal Description Methods (AIFB, aifb.kit.edu) and Tetherless World Constellation (tw.rpi.edu). Meanwhile, there are many domain-specific Semantic Web portals coming from winners of the “Semantic Web challenge” [7] including CS AKTive Space [8], Museum Finland [9], Multimedia E-Culture demonstrator [10], HealthFinland [11] and TrialX [12]. While these Semantic Web portals are nicely crafted, most of them are too complicated to be replicated by non-specialists. Visualizations are one of the key components of a Semantic Web portal ([13], [14]). There are some general-purpose tools for visually presenting Semantic Web data, including

450

Y. Ding et al.

linked data browsers such as Tabulator (http://dig.csail.mit.edu/2005/ajar/ajaw/ tab.html) and OpenLink Data Explorer (http://linkeddata.uriburner.com/ode), as well as data mashup tools such as sigma (aggregated instance description, sig.ma) and swoogle (aggregated semantic web term definition, swoogle.umbc.edu). These tools render RDF triples directly via faceted filtering and customized rendering. SIMILE’s Longwell can be used to enable faceted browsing on RDF data, and Exhibit can further enable faceted visualization (e.g., map, timeline). It is notable that these tools differ from information visualization tools, which have more emphasis on rendering data into a graphical format.

3 SWP Architecture The SWP is a lightweight portal platform to ingest, edit, display, search and visualize semantic data in a user-friendly and meaningful way. It can convert a current portal based on relational databases into a Semantic Web portal, and allows non-Semantic Web users to create a new Semantic Web portal in a reasonable period of time without professional training. Fig. 1 shows the overall architecture, which contains the following main components:

Fig. 1. SWP overall architecture

Data Ingestion (DI) Component: Its main function is to facilitate the conversion of the input data in various formats into RDF triples. It provides different templates and wrappers to handle some common data formats, such as text file, relational databases and Excel sheets. For example, it uses D2R MAP and offers templates to help nonSemantic Web users to semi-automatically create D2R rules to convert their relational data into RDF triples. Ontology Management (OM) Component: Its main function is to enable easy online ontology creation, editing, browsing, mapping and annotation. It is based on Vitro developed by Cornell University [15]. Vitro provides similar functions as Protégé (http://protege.stanford.edu/), but it is online based. Vitro will be further developed and improved by the NIH-funded VIVO project. Faceted Browsing (FB) Component: Based on Longwell, SWP mixes the flexibility of the RDF data model with faceted browsing to enable users to explore complex RDF triples in a user-friendly and meaningful manner. This faceted browser can be multifiltered, where, for example, for a research group portal, users can browse either all


451

the existing presentations by one research group or only those within one specific year AND at a specific location; for a health center portal, a doctor can know the number of patients who have diabetes AND live in Monroe County, Indiana. Semantic Visualization (SV) Component: It is based on Exhibit developed by MIT Simile project and Network Workbench by the Cyberinfrastructure for Network Science Center at Indiana University ([16], [17], [18]). It displays or visualizes RDF data in tile, timeline, Google map and table formats. It also enables the faceted visualization so that userscan visualize all of the research group members, or only those group members who share common research interests; and Semantic Search (SS) Component: It enables a type-based search that can categorize federated RDF triples into different groups based on ontologies. It is based on Lucene (http://lucene.apache.org/) and integrated with pre-defined portal ontologies to provide type-based searches. For example, if users key in “semantic web” as search query to SWP, they will receive RDF resources which contain the string “semantic web,” wherein these resources are further categorized as person, project, publication, presentation, and event. Subclasses of a Person group can be further categorized into Academic, Staff or Student. SWP acts as a stand-alone Semantic Web portal platform which can be deployed in any domain or application to input, output, display, visualize and search semantic data. Currently, it has been deployed to: (1) a middle-size research group to semantically manage topics of people, paper, grant, project, presentation and research; (2) a specialty Linked Open Data chem2bio2rdf dataset to display the relationship and association among gene, drug, medicine and pathway data; (3) an eGov dataset to facilitate faceted browsing of governmental data, and; (4) a health center to enable federated patient, disease, medication and family ties to be grouped, associated and networked. For more details, please see Section 5.

4 Portal Ontology Deploying SWP is domain specific. The user needs to create one or more portal ontologies to convert current relational databases into RDF triples. Creating an appropriate ontology is therefore a critical part of SWP. It should facilitate user queries, and meaningfully display and visualize RDF data. There are some generic requirements for creating ontologies for SWP: 1) the ontology should reflect the database schema of its original datasets; 2) the identified main concepts or relationships from commonly used user queries should be included in ontologies; 3) to enable interoperability, the portal ontologies should try to reuse existing popular ontologies, such as using FOAF to represent people (http://en.wikipedia.org/wiki/FOAF_% 28software%29) , using DOAP (http://en.wikipedia.org/wiki/Description_of_ a_Project) to represent projects, using Bibontology (http://bibliontology.com/) to represent publications and using SIOC (http://sioc-project.org/) to represent online communities, and; 4) Obeying Linked Open Data (LOD) rules (http://www.w3.org/ DesignIssues/LinkedData.html): using HTTP URIs for naming items, making URIs dereferencable and trying to use URIs from other Linked Open Data as much as possible to facilitate easy mapping. Here we use the Information Networking Ontology Group (INOG) to demonstrate the principle of creating an ontology for research networking of people and sharing medical resources. Part of this ontology group has been implemented in the Research

452

Y. Ding et al.

Group Portal use case in Section 5. INOG is one of the efforts funded by NIH and led by University of Florida [19] and Harvard University [20]. It aims to create modularized ontologies to enable a semantic “facebook” for medical scientists to network and share lab resources. The overall INOG framework is shown in Fig. 2. The core part of the framework are the INOG, including the VIVO ontology (modeling research networking) and Eagle-I ontology (modeling medical resources). These two ontologies share some common URIs and map other related URIs, and are aligned with popular ontologies such as FOAF, SIOC, DOAP and BIBO. This enables us to link our data with some existing Linked Open Data sets, such as FOAF, DBPedia and DBLP. Also, in order to model the expertise of scientists and categorize medical resources, we use existing domain ontologies such as MeSH (http://www.ncbi.nlm.nih.gov/mesh), SNOMED (http://www.nlm.nih.gov/research/umls/Snomed/snomed_main.html), Biomedical Resource Ontology (http://bioportal.bioontology.org/visualize/43000) and Ontology for Biomedical Investigation( http://obi-ontology.org/page/Main_Page) to provide categories or controlled vocabularies.

Fig. 2. Information Networking Ontology Group framework

5 Use Cases In this section, we demonstrate that SWP can be easily deployed to different domains to create various Semantic Web portals. Research Group Portal Research Group portals are one of the most common portals used in academic settings. Professors need to manage their research labs, groups or centers in an efficient way to conduct, disseminate and promote their research. The traditional research group websites are normally not easy to maintain, browse and search, especially when the size of groups reaches a certain level. The following use case is based on a mid-size research group (the Information Visualization Lab (IVL) in the School of Library and Information Science at Indiana University Bloomington (http://ivl.slis.indiana.edu/). There are approximately 30 group members, consisting of


453

one professor, several senior research staff and programmers, PhD and master students and hourly students. It has, at any point in time, around ten externally-funded projects, mostly from NIH and NSF. The major activities and datasets for this research group are people, papers, courses, presentations, events, datasets, software, hardware and funding. Previously all data has been stored in a relational database (e.g., PostgresSQL) with about 20 main tables and more than 50 bridge tables to inter-connect different datasets. One of the major bottlenecks is that it is not simple to harvest all items relating to one entity. For example, it is very difficult to group all information about one group member. Users have to go to the publication page to get information on publications, the presentation page to get information on presentations and the research page to get information on projects. This harvesting limitation also generates the problem of maintaining and updating the data.

Fig. 3. List view of SWP

Fig. 4. Graph view of SWP

Fig. 5. Screenshots of SWP’s semantic visualization

454

Y. Ding et al.

Using SWP, we create a machine-readable semantic version of this research group portal (http://vivo-onto.slis.indiana.edu/ivl/). We used D2R to convert around 70 relational tables into RDF triples based on the VIVO ontology version 0.9. This portal enables faceted browsing and semantic visualization. For example, by clicking People, users see the list view of federated information for each group member, including his or her publications, presentations, research interest and projects. Using a faceted browser, users can further narrow down their searches. Among all the group members, SWP can display group members who are only interested in the Network Workbench Tool research topic. The default view is List view (see Fig. 3), and Graph view provides basic graph overlay of RDF triples and highlights some nodes with labels (see Fig. 4). Exhibit view contains several view formats, such as tile, timeline, map and table views (see Fig. 5). Tile view groups entities based on multiple criteria, such as grouping presentations based first on year, then on presenter’s last name. Timeline view shows timelines on grouped entities, such as presentations at different time slots. Table view displays entities in table format. Map view uses Google Map to view grouped entities based on locations. All of these views enable faceted visualization so that users, for example, can view presentations in 2005 AND in Indianapolis. The current semantic search function is very limited. Longwell only provides Lucene text search. Since the People page groups all the related information about one person together, by going to the People page and searching “network,” users can locate people who are interested in “Network Workbench Tool” or who published their papers in “Network Science” conference.

Fig. 6. Screenshots of the Health Center Portal Fig. 7. Screenshots of eGov Portal

Health Center Portal Indiana University (IU) Health Center (http://healthcenter.indiana.edu/index2.html) provides comprehensive health services to meet the medical and psychological needs of students, spouses and dependents. It serves more than 40,000 potential patients around campus, and each patient can access his or her information online. Doctors and medical staff can pull out the related information about a group of patients from this portal for diagnosis and analysis purposes. It currently uses a relational database and is powered by workflow.com enterprise solutions. IU Health Center data are stored in more than 100 tables and contain information such as person, insurance, medication, clinical document, surgery, immunization, allergies and family ties. We deployed SWP to IU Health Center and created an easy-to-use Semantic Web portal (see Fig. 6). As it is useful for doctors and staff to look at the overall


455

information at one place, this portal groups together all information related to one patient, such as medication, diagnosis, doctor, disease, location and time factors. The faceted browser allows users to select different criteria by which to view data. For example, the right side of Fig. 6 shows the H1N1 flu patients’ geographical distribution in the Bloomington area. Doctors can further narrow down the geo maps by selecting different time periods or patient status. eGov Portal eGov’s current initiative of adopting Semantic Web technology makes converting governmental data into RDF triples and providing meaningful browsing and searching supports essential. In this example, we use Ozone and Visibility data from the EPA’s Castnet project (http://www.epa.gov/castnet/) and convert them into RDF triples. The problem here is that while these datasets have data on Ozone and Visibility for each of the Castnet sites, they do not have data on where these sites are located. Using a second dataset from the EPA’s site (http://www.epa.gov) that has data on the location of each Castnet site, we created this Web application as seen in Fig. 7. In the left side of Fig. 7, yellow dots represent a single Casetnet site and the size of dots corresponds to the average Ozone reading for that site. Users can apply filters to narrow down the results of Castnet sites. When a Castnet site is clicked, a small pop-up opens that displays more information on that site and provides a Web link which takes users to another page. The right side of Fig. 7 displays a timeline for all the Ozone and Visibility data available for that site based on Google Visualization API. Chem2bio2rdf Portal/Linked Open Data Portal This use case demonstrates the potential of using SWP to provide better browsing and searching support for some of LOD bubbles. A systems chemical biology network called chem2bio2rdf has been created by integrating bio2rdf and Linking Open Drug Data (LODD) to allow links between compounds, protein, targets, genes and diseases. The chem2bio2rdf contains 18 datasets in the domain of systems chemical biology and is grouped into five categories: chemical (pubchem, ChEBI), chemogenomics (KEGG ligand, CTD chemical, BindingDB, Matador, PubChem BioAssay, QSAR, TTD, DrugBank), biological (UNIPROT), systems (KEGG, Reactome, PPI, DIP), phenotype (OMIM, CTD disease, SIDER) and literature (PubMed). The result is a SPARQL endpoint to support RDF queries (http://chem2bio2rdf.org) and a userfriendly SWP at (http://chem2bio2rdf.org/exhibit/drugbank.html).

6 Evaluation To evaluate SWP’s usability, we conducted a user evaluation based on 14 users. The survey results demonstrate that semantic web technology provides better integrated information with positive feedback by 78% of our users. As for the faceted browser, more than 57% of users agreed that such function shortens the time they required to find desired information. Additionally, users were very positive about the visualizations function of SWP. Among the 6 methods of visualization available, map view received the highest aggregate score in users’ satisfaction, while graph view the lowest., The survey did reveal limitations to user satisfaction with the SWP., some users

456

Y. Ding et al.

felt that too much information is integrated. The predefined filtering conditions need refinement in the faceted-browsing function. users suggested that visualization views should be based on the data type, potential user needs, user system configuration and final output, and currently these views did not match their expectations. Another evaluation approach is a straightforward comparison of the difference between portals with and without SWP, where we take the afore-mentioned Research Group Portal and chem2bio2rdf Portal as examples. The Research Group Portal comparison demonstrates that the SWP version provides several value-added features (e.g., federating related information about one entity in one place) than the non-SWP version. The second chem2bio2rdf Portal comparison explains that SWP can provide better user-friendly browsing support for Linked Open Data bubbles than normal SPARQL endpoints (see Fig. 8).

Fig. 8. Normal LOD display vs. SWP LOD display

Seven related systems have been identified herein: Disco (http://www4.wiwiss. fu-berlin.de/bizer/ng4j/disco/), Marbles (http://marbles.sourceforge.net/), Zitgist (http://zitgist.com/), Dipper (http://api.talis.com/stores/iand-dev1/items/dipper.html), mSpace (http://mspace.fm/), jSpace (http://www.clarkparsia.com/jspace/), sigma (http://sig.ma), Exhibit (http://www.simile-widgets.org/exhibit/) and Tabular (http:// www.w3.org/2005/ajar/tab). We compare SWP with nine systems (see Table 1, Disco (http://www4.wiwiss.fu-berlin.de/bizer/ng4j/disco/), Marbles (http://marbles.source forge.net/), Zitgist (http://zitgist.com/), Dipper (http://api.talis.com/stores/ianddev1/items/dipper.html), mSpace (http://mspace.fm/), jSpace (http://www.clarkparsia. com/jspace), sigma (http://sig.ma), Exhibit (http://www.simile-widgets.org/exhibit/) and Tabular (http://www.w3.org/2005/ajar/tab), where the major function of these systems is to display RDF triples. Except for Dipper and mSpace, these systems only display RDF triples in plain property-value pairs. mSpace provides RSS news style display with headings, pictures and content. Dipper displays RDF triples in plain property-value pairs and provides further categorization of these RDF triples. Sigma allows users to provide feedback on each triple by either accepting or rejecting it. Disco and Marbles only display RDF triples based on the input URI, while the others have their own data sources and ontology. Sigma has the largest data source compared to the others, and also mashes up data from other APIs. Exhibit and Tabular


457

both provide different view types to render the data, such as table view, map view, timeline view. Only mSpace, jSpace and Exhibit provide faceted browsers. In mSpace and jSpace, users can add or delete different facets based on their own needs. None of the systems, however, provide semantic search and visualization. Marble, Zitgist and Tabulator trace data provenance by adding the data source from which the RDF triple is derived. Sigma provides data provenance by allowing users to provide trust of these data sources. Only jSpace provides user-friendly SPARQL template based on the user-selected paths. Tabulator uses the selected data to generate SPARQL query. Through these comparisons, SWP can be enhanced by adding provenance to RDF triples (e.g., Sigma), improving SPARQL query builder (e.g., jSpace) and providing more output formats (e.g., Dipper).

7 Conclusion and Future Work In this paper, we propose a SWP platform which enables faceted browsing, semantic visualization and semantic search functions of RDF triples. It can be deployed to any domain or application that needs to integrate, federate and share data. It has been tested in several different domains, and requires users to create their own portal ontologies. Some future improvements to this platform include: •

• •

•

•

Dynamic SPARQL queries: Currently MIT Simile toolsets (e.g., Exhibit) cannot process dynamic SPARQL queries. It can only read static JSON files. In order to make searching and browsing more interactive, we need to find a way to let Exhibit handle dynamically generated JSON files, mainly via asynchronized service requests; Online ontology management: Currently the OM component is not fully integrated from Vitro to SWP,; Data ingestion: Currently, SWP only has the read function of RDF triples to display them in different ways. To implement the write function of SWP, data has to be converted separately to become the input of SWP. Also, there is no user-friendly way to let end users add, delete and update their instance data. Vitro provides some good examples for addressing this issue, but the integration of Vitro and SWP has to be investigated; Semantic visualization: Currently the semantic visualization of SWP is very limited, with only naïve displays of RDF graphs and labeling nodes. The network analysis is not yet implemented. Future work will be focused on visualizing network and identified paths of the network which are associated with user queries, and; Semantic Search: Currently SWP uses Lucene indexing, and the type-based search is very limited. We need to identify a better way to integrate Vitro semantic search with SWP. Meanwhile, we are exploring the potential integration of semantic associations to discover complex relationships in semantic data. As RDF data forms semantic graphs, with nodes and links that have embedded semantics, graph mining technologies can be applied to identify and rank semantic nodes and relationships. By weighing semantics of surrounding nodes and links, semantic associations can be calculated based on ranking of available paths of nodes [21].

No

No

No

No

Provenance

User-friendly SPARQL template

No

Yes

No

No

Have faceted browser? Semantic search Visualization

Yes

Yes

No

No

No

No

No

No

No

No

Yes

No

No

No

No

Yes

Yes

Yes

No

No

No

Yes

Yes

No

Yes

No

No

Yes(crawldata fromweb,donot haveown ontology) No

No

No

Yes

No

Yes

Yes

Displaythedata in different views

Yes

Yes

Yes

No

No

No

Displaythedata in different views.

Yes

No

Yes

Yes

Yes

Yes

Displaythedata in different views.

No(mashing up relateddatafrom differentdata sources) No

User-friendly displayRDF triples:RSSnews styleofdisplay (heading,picssk, andcontent) Purelypropertyvaluepairdisplay

No(just displayingdata contained in the inputURI) No

Haveown dataand ontology?

Classifyproperty-value pairsbasedonpredefinedcategories

Purelyproperty-value pairdisplay

Readallthe information available forthese entities,and displays itsothat userscaneasily readand understand related,contextual information. No

Purelypropertyvaluepairdisplay

Purely property-value pairdisplay

DisplayRDF triples

Usercan add/delete filters tothe faceted browser

SWP BrowseRDFdata indifferent views type,suchaslist, graph, map, timeline,table. ProvideuserfriendlySPARQL querybuilder, semanticsearch.

Tabulator BrowseRDFdata andselectpartofit todisplayin different views type,suchastable, map,calendar, timelineand SPARQL template.

Exhibit DisplayRDFtriples indifferent views, includingTabular View,Timeline View,MapView andTileView

Sig.ma DisplayRDF triples gathered fromcrawled sourcesorother APIs Usercanprovide theirfeedbackto acceptorrejectthe resourcesfortheir ownpurposes Purelypropertyvaluepairdisplay

jSpace DisplayRDFtriples Providethreeviews: data,web,andsocial network views User-friendly SPAQRLbuilder throughuserselected paths

mSpace Viewdatawith facetedbrowser

Dipper DisplayRDFtriples ina givenURI Categorizeproperties intoseveralpre-defined classes Exporttheoutputdata in different formats:JSON, RDF/XML,Turtle,NTriple

Zitgist

Provide DataViewerand QueryBuilderfor RDFtriples

Marbles

DisplayRDF triplescontained ina givenURI. Providethree views:full, summaryand photo views

DisplayRDF triples contained ina givenURI

Major functions

Disco

Table 1. Comparison of SWP with related systems

458 Y. Ding et al.


459

This paper addresses the issue of lacking user-friendly displaying and browsing support for semantic data. The Semantic Web is moving successfully from theory development to real data gathering and application building. It is now important to provide user-friendly methods that allow normal users to feel the beauty of semantic data and Semantic Web technologies. This paper confirms that SWP can make Semantic Web meaningful to both Semantic Web specialists and the public. SWP can be easily deployed into any middle-sized domain, and is also useful for displaying and visualizing Linked Open Data bubbles. Acknowledgments. This work is funded by NIH VIVO Project (UF09179).

References 1. Berners-Lee, T., Hendler, J., Lassila, O.: The semantic web: Scientific American. Scientific American (2001) 2. Maedche, A., Staab, S., Stojanovic, N., Studer, R., Sure, Y.: SEAL - A Framework for Developing SEmantic Web PortALs. In: Proceedings of the 18th British National Conference on Databases: Advances in Databases, pp. 1–22 (2001) 3. Staab, S., Angele, J., Decker, S., Erdmann, M., Hotho, A., Maedche, A., Schnurr, H., Studer, R., Sure, Y.: Semantic community Web portals. Comput. Netw., 473–491 (2006) 4. Lausen, H., Ding, Y., Stollberg, M., Fensel, D., Hernandez, R., Han, S.: Semantic web portals: state-of-the-art survey. Journal of Knowledge Management 9(5), 40–49 (2005) 5. Maryland Information and Network Dynamics Lab Semantic Web Agents Project (2004), http://www.mindswap.org/first.shtml 6. Krötzsch, M.: Semantic Media Wiki (2010), http://semantic-mediawiki.org/wiki/Semantic_MediaWiki 7. Bizer, C., Maynard, D.: Semantic Web Challenge (2010), http://challenge.semanticweb.org/ 8. Schraefel, M.C., Shadbolt, N.R., Gibbins, N., Glaser, H., Harris, S.: CS AKTive Space: Representing Computer Science in the Semantic Web. In: Proceedings of the 13th International Conference on World Wide Web, pp. 384–392. ACM Press, New York (2004) 9. Hyvönen, E., Junnila, M., Kettula, S., Mäkelä, E., Saarela, S., Salminen, M., Syreeni, A., Valo, A., Viljanen, K.: Publishing Museum Collections on the Semantic Web: The Museumfinland Portal. In: Proceedings of the 13th International World Wide Web Conference on Alternate Track Papers & Posters, pp. 418–419. ACM Press, New York (2004) 10. MultimediaN N9C Eculture project: Multimedia E-Culture demonstrator, http://e-culture.multimedian.nl/index.shtml 11. Suominen, O., Hyvönen, E., Viljanen, K., Hukka, E.: HealthFinland-A National Semantic Publishing Network and Portal for Health Information. Web Semantics: Science, Services and Agents on the World Wide Web 7(4), 287–297 (2009) 12. Applied Informatics, Inc.: TrialX: Enabling Patients to Find New Treatments (2010), http://trialx.com/ 13. Cole, W.G., Shererta, D.D., Hsu, G., Fagan, L.M., Carlson, R.W.: Semantic Visualization Of Oncology Knowledge Sources. In: Proc. Annu. Symp. Comput. Appl. Med. Care, pp. 67–71 (1995) 14. Padgett, T., Maniquis, A., Hoffman, M., Miller, W., Lautenschlager, J.: A Semantic Visualization Tool for Knowledge Discovery and Exploration in a Collaborative Environment, https://analysis.mitre.org/proceedings/Final_Papers_Files/ 171_Camera_Ready_Paper.pdf

460

Y. Ding et al.

15. Devare, M., Corson-Rikert, J., Caruso, B., Lowe, B., Chiang, K., McCue, J.: VIVO: Connecting People, Creating a Virtual Life Sciences Community. D-Lib Magazine 13(7/8) (2007), http://www.dlib.org/dlib/july07/devare/07devare.html 16. Bruce, H., Huang, W., Penumarthy, S., Börner, K.: Designing Highly Flexible and Usable Cyberinfrastructures for Convergence. In: Bainbridge, W.S., Roco, M.C. (eds.) Progress in Convergence - Technologies for Human Wellbeing, Boston, vol. 1093, pp. 161–179. Annals of the New York Academy of Sciences, MA (2007) 17. Neirynck, T., Börner, K.: Representing, Analyzing, and Visualizing Scholarly Data in Support of Research Management. In: Proceedings of the 11th Annual Information Visualization International Conference, IEEE Computer Society Conference Publishing Services, Zürich, Switzerland, pp. 124–129 (2007) 18. NWB Team: Network Workbench Tool, http://nwb.slis.indiana.edu 19. Conlon, M.W.: VIVO: Engabling National Networking of Scientists. University of Florida, Cornell University, Indiana University, Washington University of St. Louis, Ponce School of Medicine, Weill Cornell Medical College, The Scripps Research Institute: NIH/NCRR, 1U24RR029822-01 (2009) 20. Nadler, M.D., Marshall, L.: Networking Research Resources Across America (Eagle-I project). Harvard Medical School, Dartmouth College, Jackson State Univesity, Morehouse School of Medicine, Montana State University, Oregon Health and Science University, University of Alaska Fairbanks, University of Hawaii Manoa, University of Puerto Rico: NIH CTSA (2009) 21. Anyanwu, K., Maduko, A., Sheth, A.: SemRank: ranking complex relationship search results on the semantic web. In: Proceedings of the 14th International Conference on World Wide Web, pp. 117–127 (2005)

NicoScene: Video Scene Search by Keywords Based on Social Annotation Yasuyuki Tahara, Atsushi Tago, Hiroyuki Nakagawa, and Akihiko Ohsuga Graduate School of Information Systems, The University of Electro-Communications, Tokyo, Japan [email protected]

Abstract. As there are increasing needs to view a huge number of videos on the Web in short time, the video summary technology is actively investigated. However, there exists trade-offs of costs and precision of summaries. In this paper, we propose a system called NicoScene to search desirable scenes from the videos provided in a video hosting service called Nico Nico Douga. We use the feature of the service in which we can attach comments to videos and treat the comments as social annotation. By some experiments, we demonstrate the advantages of NicoScene in particular the search precisions.

1

Introduction

Huge amount of storage and the rapidly spreading broadband Internet have enabled video hosting services such as YouTube1 . According to a report [Australian 10], the ratio of audience of TV is decreasing and that of the video hosting services is increasing. In addition, people are beginning to use services such as Nico Nico Douga2 in which they can annotate comments to shared videos synchronously with respect to the playing time. This is because such services can be used as highly bidirectional communication media. These services would become more important in building culture and arousing public opinion. For example, the users of the services replied to a questionnaire of their supporting political parties answers that are different from those of the general public. However, it is needed to enable the videos to be viewed efficiently in a limited time because the easy procedures to publish the videos on the Web make the number of the videos rapidly increase. The means of efficient viewing can be divided into the two categories, that is, video classification and video summary. Video classification decreases the number of the videos to be viewed. This approach classifies the videos by analyzing the tags and the contents so that the users can easily find the videos they want to view. It also uses techniques such as filtering to recommend videos matching the users’ preferences. The video summary approach enables the users to view videos in shorter time by summarizing 1 2

http://youtube.com/ http://www.nicovideo.jp/ “Nico Nico Douga” in Japanese is literally translated into “smiley video”.

A. An et al. (Eds.): AMT 2010, LNCS 6335, pp. 461–474, 2010. c Springer-Verlag Berlin Heidelberg 2010

462

Y. Tahara et al.

the videos. The summaries are made by classifying the scenes in the videos and picking up and playing the scenes of only some specific classes. The word “scene” here means a fragment of a video including some specific events. Annotation is a technique to attach to the scenes meta-information that are used for searching the scenes. There are two approaches of annotation, that is, automatic methods such as video analysis and manual annotation. The approach to attach comments in the bidirectional media mentioned before can be considered as social annotation in which the general public annotate manually. In this paper, we focus on scene search for the video summary approach and propose a search technique using social annotation. The reason to use social annotation is to decrease the annotation costs for a huge number of videos in the Web. In addition, by experiments, we demonstrate that our technique can search scenes with high precisions and sufficient recalls. This paper is organized as follows. Section 2 describes Nico Nico Douga that is a representative example of bidirectional service and the target of our research. Section 3 makes clear the issues in using the comments as social annotation. Section 4 proposes the system called “NicoScene” to address the issues. Section 5 describes the implementation of NicoScene. Section 6 describes the experiments to demonstrate the effect of our approach and discusses the results. Section 7 discusses how much our system addresses the issues described in Section 3. Section 8 compares related work with our proposal and examines the advantages of our approach. Section 9 provides some concluding remarks and future work.

2

Nico Nico Douga

We deal with videos published on Nico Nico Douga as socially annotated media. In this section, we describe a summary of Nico Nico Douga and its feature of attaching comments to videos. 2.1

Summary

Nico Nico Douga is one of the most popular video hosting services in Japan with more than thirteen million subscribers3 , about three million videos, and more than two billion comments. This popularity is due to the unique user interface (shown in Figure 1) in which the users can attach comments to each scene of videos individually by designating the time from the beginning of each video and the comments are superimposed on the videos. This commenting feature provides togetherness to the audience and highly bidirectional communication between video uploaders and the audience. The uploaders can have incentive because they can easily obtain the audience’s opinions. Nico Nico Douga is now a field of very lively and creative activities such as videos with advanced video processing techniques and original stories. It has also a considerable impact to the research community such as an analysis of video reuse networks [Hamasaki 08]. 3

Users cannot view the videos on Nico Nico Douga without subscription.

NicoScene: Video Scene Search by Keywords Based on Social Annotation

463

Comments

Comment Input Field

Comments Comment Input Field

Fig. 1. User interface of Nico Nico Douga: playing a video and attaching comments

2.2

Attaching Comments to Scenes

A Nico Nico Douga user can attach comments to any scenes during viewing a video. Other users can read the comments that move from right to left on the corresponding scene. The uploader of the video can use the uploader-specific commenting functionality to attach such texts as the explanations of the scenes and the lyrics of the songs. The types of comments varies depending on the scenes. For example, “gooru !!!” (Japanese for “goal”) is attached to a goal scene of a soccer video, “ooooo” (Japanese for “ooh”) to a surprising scene, and “wwwww” (“w” is the acronym of a Japanese word for “laugh”) to a funny scene. On a scene in which a specific character or person appears, we sometimes see comments related to it. More comments appear on a scene with more interesting contents. The comments in Nico Nico Douga have the potential to be useful video annotation because they usually include information about the contents of the scenes. However, there are so many noises in the comments that we cannot use all the comments as annotation. In this paper, we propose a system to search scenes by estimating the degree of attention to the scenes on the basis of the number of the comments and examining the contents of the scenes on the basis of the contents of the comments.

3 3.1

Issues in Scene Search Based on Comments Dependency on the Number of Comments during a Fixed Time

The number of viewers and comments of a video increases as the time length of the video does. However, we consider that the time when a scene appears in the

464

Y. Tahara et al.

video does not affect the number and the contents of the comments. Therefore it is not desirable if the number of comments affect the search results. For example, the search results should not be different between the case in which five hundred comments are attached to a video and the case in which one thousand comments are attached to the same video. 3.2

Dependency on Video Clipping

As Nico Nico Douga limits the time length of a video, the uploaders need to clip long videos. It is left to the uploaders’ discretion which part of a video they clip and upload even if they deal with the same video. Therefore it is not desirable if the way of clipping affects the search results. For example, suppose that we have the following three videos of the same soccer match: (1) the video including both of the first and the second halves, (2) the one including only the first half, and (3) the one including only the second half. It is not desirable if the results differ by searching the scenes of the first half from (1) and from (2) with the same query. 3.3

Representations of Attributes of Scenes

In order to search scene using the comments, the users need to know the words explicitly representing the attributes of the scenes that they want to search. However, the users usually do not know which comments are annotated to which scenes. Therefore we need to bridge the gap between the queries the users input and the representations of the attributes of the scenes included in the comments. 3.4

User Interface for Easy Understanding of Relationships between Scenes

Many existing user interfaces dealing with videos display the scenes in the order of the time sequence. Although such interfaces are useful if the users want to view the search results on the basis of time, it is not easy to understand the relationship between scenes if the users have searched them on the basis of their contents. In addition, if the users search scenes from multiple videos transversely, user interfaces based on time sequences are not considered as an appropriate way of displaying the search results.

4 4.1

Proposed System and Approach NicoScene: Scene Search System

We propose NicoScene (Figure 2) as a scene search system addressing the issues described in Section 3. NicoScene users can carry out the following operations. 1. Video search and identification 2. Scene search in the identified videos with queries of scene attributes


465

3. Search result display according to relationships to be focused on 4. Collaborative editing of the keyword ontology specifying the scene attributes After the users search videos on NicoScene, they search scenes by inputing scene attributes. The relationships to be focused on are switched by checkboxes. The ontology specifying the scene attributes is included in NicoScene. It can be edited using the specific editor.

（１）Search and specify videos

（２）Scene search with queries of scene attributes （３）Choose relationships to be focused on Scene Keyword Video Scene

Videos 動画動画（４）Collaborative editing

Keyword ontology

System

シーンシーン Scenes

シーン2（内容2）

Fig. 2. NicoScene system

4.2

Unit Scoring Based on Comments

NicoScene treats each scene as a unit that is a fragment of videos whose time length is specified in advance. In order to address the issues described in 3.1 and 3.2, the system calculates the score of each unit of a video on the basis of the comments and outputs the units with more scores than a threshold as the scene search results. The following descriptions show the procedure. 1. Divide a video with its time length T and the number of comments C into units Ui (i = 1, 2, · · ·, N ). Each unit is a video fragment with its time length t that the user can change according to the category or the type of the video (therefore T = N t). N of all the comments 2. For each unit Ui , count the number ci i=1 ci = C and the number Ki of the comments including the keywords in the keyword ontology (“keyword comments” hereafter) corresponding to the scene attributes included in the query.

466

Y. Tahara et al.

3. Calculate the basic score Si = ci + αKi , where α is the weight representing how much more important are the keyword comments than other comments. 4. Calculate the average number of comments Ct/T in order to resolve the differences according to the total number of comments C and the time length of a unit t, and normalize Si with Ct/T into Si : Si =

T Si Ct

(1)

5. Output the units Ui and their previous units Ui−1 as the search results where the unit scores Si are larger than the threshold δ. 4.3

Collaborative Editing of Keyword Ontology

We adopt the ontology approach to address the issue described in 3.3. In order to represent the attributes of the scenes to be searched, NicoScene uses a set of keywords frequently appearing in the comments in such scenes. The keywords are organized in the keyword ontology (Table 1). A keyword ontology is a set of relations between scene attributes and keywords. This keyword ontology is created and edited by the audience that heavily use Nico Nico Douga and have knowledge about the videos and the scenes. Thus the ontology would become useful because it incorporates the knowledge of the audience who are acquainted with the comments of Nico Nico Douga. Table 1. Example of keyword ontology Scene Attributes Goal Red card Rough play Overtake Taro Aso Yayoi Takatsuki ...

4.4

Keywords ”goal”, ”here comes”, ”good”, ”ooh”, ”first score” ”red”, ”sent out”, ”suspension” ”dangerous”, ”rough”, ”red card”, ”yellow card” ”overtake”, ”OT”, ”ooh”, ”overtaking” ”Aso”, ”prime minister”, ”His Excellency”, ”Rosen”, ”Miyabe” ”yayoi”, ”u’uh”, ”bean sprout”, ”35 yen” ...

User Interface Displaying Relationships to Be Focused on

In order to address the issue described in 3.4, we provide a user interface visualizing videos, scenes, and keywords as the nodes of SpringGraph [SpringGraph] (Figure 3). SpringGraph is a component that draws graphs automatically on the based of the physical model of springs. It makes us easily understand the relationships between the nodes. For example, in Figure 3, we can focus on the video 1 by removing the node of video 2 and displaying only the relationships to be focused on. We can also focus on the keyword “goal” by removing the node of the keyword “poor” (Figure 4).


467

Fig. 3. Example of user interface arranging videos (rectangle nodes), keywords (round rectangle nodes), and scenes (oval nodes) in SpringGraph

Fig. 4. Left: focusing on video 1, right: focusing on the keyword “goal”

5 5.1

NicoScene Implementation Summary

NicoScene is composed by the server side program written in Java and the client side component implemented using Adobe AIR [Adobe AIR]. The client side component accepts queries of video and scene search, displays the search results, and builds and edits the keyword ontology. The server side program accesses Nico Nico Douga according to the input search queries. It also obtains video information and comments, and manage the keyword ontology. 5.2

Displaying Scene Search Results

We implemented the user interface described in 3.5 using SpringGraph [SpringGraph] as well as the interface arranging the scenes along the usual time sequence. SpringGraph displays a pair of linked nodes as close as possible and a pair of unlinked ones as far as possible. This feature enables easily and visually understandable representations of relationships between videos such as shown in Figure 5.

468

Y. Tahara et al.

Fig. 5. Connections of Content (oval nodes), scenes (smaller rectangles), and videos (larger rectangles)

6

Experimental Evaluation of Search Precision

In this sections, we evaluate NicoScene by measuring the precision of scene search by experiments. The target of the experiments include searches of objective scenes such as goal scenes of soccer and subjective ones such as impressive scenes. The correct answers for the objective scene searches are the video units including the times on which the corresponding events occur. For the subjective ones, we picked up the units including the times recorded as impressive scenes by multiple persons who carried out the experiments. The subjects of these experiments are four students of our university. 6.1

Method of Experiments

We used a set of keywords listed up in advance by users who frequently use Nico Nico Douga. We considered a search result of NicoScene as correct if the searched unit includes a correct scene. The evaluation criteria are precision = (the number of correct search results) / (the number of searched scenes) and recall = (the number of correct search results) / (the number of correct answers).

Video Search results Correct answer Fig. 6. Definition of correct answers

For example, in Figure 6, the triangles represent the times in which the events occur. As the correct answers are the units including that times, the number of the correct answers is three. The orange units are the search results of the system (searched scenes) and the number of them is two. In this case, the first search


469

result does not overlap with any units of the correct answers and therefore it is “incorrect”. Since the other result includes a correct answer, it is a correct result. Therefore, because one of the two searched scenes is a correct result and the number of the correct answers is three, the precision is 50% (1/2) and the recall is 33.3% (1/3). Experiment 1: Objective Scene Search. As the target of the experiment of searching scenes including objectively observable events, we examined soccer goal scenes and overtaking scenes of the Formula One auto racing (“F1” hereafter). 1. Experiment 1-(1): Soccer goal scene search We searched goal scenes from seventeen soccer videos whose time lengths are ranging from 20 to 65 minutes and measured the precisions and the recalls. We examined the correctness of the results by comparing them with the times of the goal scenes identified by actually viewing the videos. We also fixed the time length of one unit t as 38 seconds because we found by a preliminary inquiry that average people recognize this time length appropriate as one scene of a soccer match. We used the keywords corresponding to the “Goal” attribute in Table 1. 2. Examined 1-(2): F1 Overtaking scene search We searched overtaking scenes from nine F1 videos whose time lengths are around 25 minutes and measured the precisions and the recalls. We examined the correctness of the results by comparing them with the times of the overtaking scenes identified by actually viewing the videos. We also fixed the time length of one unit t as 60 seconds. We used the keywords corresponding to the “Overtake” attribute in Table 1. Experiment 2: Subjective Scene Search. In this experiment, we requested four people to view a soccer match of 160 minutes (including the extra time and the penalty shoot-out) and to record the beginning times of the scenes the people thought impressive. After that, we compared the records with the search results of the system and calculated the total precision and recall as the evaluation parameters. We used the set of keywords (keyword set) { “goal”, “here comes”, “good”, “ooh”, “first score”, “rough”, “dangerous”, “red card”, “sent off”, “red”, “suspension”, “yellow card”, “yellow”, “caution”, “great”, “unlucky”}. Experiment 3: Scene Search from Other Types of Videos. The aim of this experiment is to examine how much widely applicable our system is. We tried the following tasks and measured the precisions and the recalls with changing α and δ. 1. Experiment 3-(1): Searching scenes in which a specific person is speaking from videos of political discussions that are different from sports videos because they include only small actions of the people 2. Experiment 3-(2): Searching scenes in which a specific character appears from entertainment videos

470

Y. Tahara et al.

As for 3-(1), we searched scenes in which Mr. Taro Aso, the former prime minister of Japan, is speaking from four videos of party leader meetings. As for 3-(2), from four videos of original stories based on a video game called “THE IDOLM@STER”, we searched scenes in which Yayoi Takatsuki, a character of the game, appears. We fixed the time lengths of the units as 60 seconds (3-(1)) and 40 seconds (3-(2)) by considering the length of one scene of each type of videos. We want to remark that the distributions of the comments have larger deviations than the videos used in Experiments 1 and 2. We used the keywords appearing in Table 1. In all the experiments, we changed the weight α of the keyword comments from 0 to 100 and the score threshold δ from 0 to 12. 6.2

Results of Experiments

Figure 7 shows the results of the experiments. In each graph, the solid lines denote the precisions while the dotted lines denote the recalls. The results show that scene search based on social annotation has sufficient precisions. In detail, for any types of videos, all the experiments provided high precisions with keeping the recalls more than 0.5 if we fix the weight of keyword comments α to 70 and the threshold δ to 6 (Figure 2). Table 2. Precisions and recalls of each experiment with weight α = 70 and threshold δ=6 Precisions Recalls Soccer goal scenes 0.77 0.78 0.65 0.62 F1 overtake scenes Soccer impressive scenes 0.81 0.58 Mr. Aso speaking scenes 0.77 0.77 0.66 0.56 Yayoi appearing scenes

Next, we evaluate the results in detail. The precision of Experiment 1-(2) is lower than that of 1-(1) because the numbers of videos, scenes, and comments are small. Although our approach normalizes the comment scores, we would need a considerable number of comments. The results of Experiment 2 is worse than those of 1-(1) that deals with the same type of videos (of soccer). This would be because of the wide variety of the scenes picked up by each person on the basis of her or his subjective criteria of impressiveness, while goal scenes are identified by objective events. For example, a subject of this experiment picked up player substitution scenes. Another subject picked up scenes in which the supporters are singing. The results of Experiments 3 are worse than Experiments 1 although the former ones search objective scenes in which a specific person is speaking or a specific character appears. This would be because, while the audience respond


471

Fig. 7. Precisions (solid lines) and recalls (dotted lines) of each experiment

much to the first speaking scene or the first appearance scene, their responses become weaker for the following similar scenes due to the characteristics of the videos of discussions or stories.

7

Discussions

In this section, we discuss how much our system addresses the issues described in Section 3. First we discuss the issues described in 3.1 and 3.2, that is, dependency on the number of comments during a fixed time and dependency on video clipping. The experiments show that NicoScene can search required scenes with considerable precisions by scoring the units on the basis of the number and the contents

472

Y. Tahara et al.

of comments and normalizing them. This is in particular correct in searching scenes including objective events to which the general public respond. However, the experiments also show that our approach does not much fit searching scenes each person subjectively feels important. Even if we search objective scenes, sufficient number of comments are needed to be attached to each video. We tried to address the issue described in 3.3, that is, representations of attributes of scenes, by using the keyword ontology. The experiments show that building an appropriate keyword set helps effective scene search. However, it is not easy to list up keywords appropriate to various situations. We need to examine additional approaches such as keyword reuse based on the structure of the ontology to make our system practical. As for the issues described in 3.4, that is, user interface for easy understanding of relationships between scenes, SpringGraph visualizes the relationships. This is because the spring model puts the closely related nodes near each other. Our user interface also makes it easy to grasp scenes that match multiple keywords. Thus we can understand the relationships intuitively by NicoScene.

8

Related Work

There are various approaches to annotate multimedia contents. One of such approaches that are popular is automatic annotation. This approach identifies the types of scenes by analyzing the images and/or the sounds and attaches metadata to the scenes. For example, Bloehdorn et al. [Bloehdorn 05] relates low level descriptions of multimedia data such as colors and content level descriptions such as persons, and creates and attaches semantic annotations. The automatic annotation approaches are expected to make sure considerable precisions and have produced several applications such as removing commercial messages. However, most of them have the issue of limitations of target videos. For example, we need to identify each speaking person for news videos, changes of the background images for sports videos, and scene switching for movies. Therefore, since we need to use different techniques for different types of videos, it is difficult for one single system to deal with various types of videos. This means that we cannot deal with the huge number and variety of videos on the Web at the same time. On the other hand, the manual annotation approaches are divided into the two categories of experts’ annotations and annotations by the general public. As for the former approach, experts such as producers and broadcasting company staffs attach metadata mainly for the commercial purposes. Such annotations include captions in media such as DVDs, teletexts, and closed captioning. This approach would produce highly precise annotations because experts provide them. However, it is not practical to use them for video hosting services since it is more expensive than other approaches and can be used only for commercial media. Manual annotation by the general public is called social annotation this paper deals with. While Nico Nico Douga is an application of social annotation in practice, there are other applications in the research field such as the video tag


473

game [Video Tag Game]. In social annotation, general users voluntarily attach metadata to scenes. Social annotation is considered as more useful information as more users are involved. Miyamori et al. [Miyamori 05] obtained considerable result in summarizing American football match videos. In their approach, they compared with the video of the match the uploaded time of the texts written for a TV program of a match into BBSs on the Internet. Thus they treated the texts as annotations to the video. Because the annotations are attached by the users with the background knowledge about American football, their approach has the advantages that it can precisely describe the contents of scenes and the annotation costs are low. Masuda et al. [Masuda 08] adopt a similar approach. However, if we use the annotation without any processing, we would have the problems that we cannot obtain appropriate information due to the many noises. In addition, a large amount of information often makes it difficult to identify and extract specific scenes. Nico Nico Douga is attracting attentions from various communities because it is large-scale. Research on it is getting active recently. Nakamura et al. [Nakamura 08] measured reliabilities of videos by tracing the changes of comments along the playing time and the time since uploading of each video. Hamasaki et al. [Hamasaki 08] analyzed the connections and the width of the spread of users’ creative activities by tracing the relations between the user-generated contents in Nico Nico Douga. Our approach is different from these researches of scene search and annotation as follows. We attach annotations automatically in low costs to the huge number of videos stored in the bidirectional media by estimating the degree of attention to scenes on the basis of quantitative comment analysis and analyzing the types of the comments. We can thus search scenes with considerable precisions efficiently.

9

Conclusions

In this paper, we proposed NicoScene that is a system to search scenes of videos by examining comprehensively the number and the contents of the comments of Nico Nico Douga. Experiments showed that our system can search scenes with sufficient precisions and recalls from any types of videos by fixing an appropriate weight of keyword comments and an appropriate threshold of scores. In addition, we can adjust the trade-offs between the precision and the recall by changing the threshold. This means that the threshold can be used as a parameter to adjust the number of the search results. The current research status is at an intermediate step toward the video summary technology. Therefore we consider the precisions of our experiments are sufficient if we apply our approach to a system with which we manually extract scenes from the search results and create a summary by combining them. As future work, we will carry out experiments similar to those of this paper for other various types of videos. These experiments would make clear the applicability of our approach and increase the precision. We want to improve our system to be better applied to video summary. Because the keyword set of

474

Y. Tahara et al.

NicoScene is manually built using ontology, we will investigate automatic approaches such as identifying comments with high frequency in each scene as keyword candidates by supervised machine learning techniques. We are also going to investigate knowledge-based search methods such as analyzing relationships between comments.

Acknowledgement We would like to thank the members of our laboratory for their cooperation on the experiments and discussions with us.

References Adobe AIR. Adobe Systems Incorporated, Adobe AIR, http://www.adobe.com/products/air/ Hamasaki 08. Hamasaki, M., Takeda, H., Nishimura, T.: Network Analysis of Massively Collaborative Creation of Multimedia Contents - Case Study of Hatsune Miku videos on Nico Nico Douga. In: First International Conference on Designing Interactive User Experiences for TV and Video (uxTV 2008), pp. 165–168 (2008) SpringGraph. Shepherd, M.: SpringGraph, http://mark-shepherd.com/SpringGraph/ Miyamori 05. Miyamori, H., Nakamura, S., Tanaka, K.: Generation of views of TV content using TV viewers’ perspectives expressed in live chats on the web. In: Proceedings of the 13th Annual ACM International Conference on Multimedia, pp. 853–861 (2005) Nakamura 08. Nakamura, S., Shimizu, M., Tanaka, K.: Can social annotation support users in evaluating the trustworthiness of video clips? In: Proceeding of the 2nd ACM Workshop on Information Credibility on the Web, pp. 59–62 (2008) Australian 10. The Australian, Net plan triggers ‘digital divide’ at Seven (2005), http://www.theaustralian.com.au/business/media/ net-plan-triggers-digital-divide-at-seven/story-e6frg996-1225840643633 Bloehdorn 05. Bloehdorn, S., Petridis, K., Saathoff, C., Simou, N., Tzouvaras, V., Avrithis, Y., Handschuh, S., Kompatsiaris, Y., Staab, S., Strintzis, M.G.: Semantic Annotation of Images and Videos for Multimedia Analysis. In: G´ omez-Pérez, A., Euzenat, J. (eds.) ESWC 2005. LNCS, vol. 3532, pp. 592–607. Springer, Heidelberg (2005) Masuda 08. Masuda, T., Yamamoto, D., Ohira, S., Nagao, K.: Video Scene Retrieval Using Online Video Annotation. In: Proc. of JSAI 2007, pp. 54–62 (2008) Video Tag Game. Zwol, R., Pyeyo, L.G., Ramirez, G., Sigurbj¨ ornsson, B., Labad, M.: Video Tag Game. In: Proc. of WWW 2008 (2008)

Social Relation Based Search Refinement: Let Your Friends Help You! Xu Ren1 , Yi Zeng1 , Yulin Qin1,2 , Ning Zhong1,3 , Zhisheng Huang4 , Yan Wang1 , and Cong Wang1 1

3

International WIC Institute, Beijing University of Technology Beijing, 100124, P.R. China [email protected] 2 Department of Psychology, Carnegie Mellon University Pittsburgh, PA 15213, U.S.A [email protected] Department of Life Science and Informatics, Maebashi Institute of Technology Maebashi-City, 371-0816, Japan [email protected] 4 Department of Artificial Intelligence, Vrije University Amsterdam De Boelelaan 1081a, 1081 HV Amsterdam, The Netherlands [email protected]

Abstract. One of the major problems for search at Web scale is that the search results on the large scale data might be huge and the users have to browse to find the most relevant ones. Plus, due to the reason for the context, user requirement may diverse although the input query may be the same. In this paper, we try to achieve scalability for Web search through social relation diversity of different users. Namely, we utilize one of the major context for users, social relations, to help refining the search process. Social network based group interest models are developed according to collaborative networks, and is designed to be used in more wider range of Web scale search tasks. The experiments are based on the SwetoDBLP dataset, and we can conclude that proposed method is potentially effective to help users find most relevant search results in the Web environment. Keywords: social relation, retained interest, social network based group interest model, personalized search, search refinement.

1

Introduction

Formulating a good query for search is an everlasting topic in the fields of information retrieval and semantic search, especially when the data goes to Web scale. The hard part is that users some times cannot provide enough constraints for a query since many of the users are not experienced enough. User background is a source that can be used to find user interests and the acquired interests can be added as constraints to the original vague query to refine the query process and help users get most relevant results. A. An et al. (Eds.): AMT 2010, LNCS 6335, pp. 475–485, 2010. c Springer-Verlag Berlin Heidelberg 2010

476

X. Ren et al.

In our setting for this study, we define a user interest as concepts that the users are interested in or at least familiar with. In addition to the study that we have made in [1], which shows that users’ recent interests may help to get a better refined query, we propose that in some cases, users’ social relations and social network based group interest models can help to refine the vague query too, since social relations serve as an environment for users when they perform query tasks. From the perspective of scalable Web search, this paper aims at achieving scalability through providing important search results to users. Since no matter how fast the data is growing, the size of the most important search results for users will be relatively small. Users’ social relation can be represented in the form of semantic data and serve as one kind of background information that can be used to help users acquire the most important search results. In this paper, based on SwetoDBLP [2], an RDF version of the DBLP dataset, we provide some illustrative examples (mainly concentrating on expert finding and literature search) on how the social relations and social network based group interest models can help to refine searching on the Web.

2

Social Relations and Social Networks

Social relations can be built based on friendships, coauthorships, work relationships, etc. The collection of social relationships of different users form a social network. As an illustrative example, we build a coauthor network based on the SwetoDBLP dataset, we represent the coauthor information for each author using FOAF vocabulary “foaf:knows”.

Fig. 1. Coauthor number distribution in the SwetoDBLP dataset

Fig. 2. log-log diagram of Figure 1

The social network can be considered as a graph. Each node can be an author name and the relationships among nodes can be coauthorships. An RDF dataset that contains all the coauthor information for each of the authors in the SwetoDBLP dataset has been created and released1 . Through an analysis of 1

The coauthor network RDF dataset created based on the SwetoDBLP dataset can be acquired from http://www.wici-lab.org/wici/dblp-sse

Social Relation Based Search Refinement: Let Your Friends Help You!

477

node distribution for this DBLP coauthor network, we can find it has following statistical properties: As shown in Figure 1 and Figure 2 [3,4]. The distribution can be approximately described as a power law distribution, which means that there are not many authors who have a lot of coauthors, and most of the authors are with very few coauthors. We can indicate that with this distribution characteristics, considering the scalability issue, when the number of authors expand rapidly, it will be not hard to rebuild the coauthor network since most of the authors will just have a few links. The purpose of this RDF dataset is not just to create a coauthor network, but to utilize this dataset to extract social relations from it and use them for refining the search process.

3

Search Refinement through Social Relationship

In enterprize information retrieval, expert finding is a emerging research topic [5]. The main task for this research area is to find relevant experts for a specific domain [6]. Nevertheless, a list of expert names that has nothing to do with the end user always confuse them. More convenient search refinement strategies should be developed. We propose that if the end users are familiar with the retrieved expert names, the search results may be more convenient for use. As an illustrative example, we propose a search task that needs to find “Artificial Intelligence authors” based on the SwetoDBLP dataset. Table 1. A partial result of the expert finding search task “Artificial Intelligence authors”(User name: John McCarthy) Satisfied Authors Satisfied Authors without social relation refinement with social relation refinement Carl Kesselman (312) Hans W. Guesgen (117) * Thomas S. Huang (271) Virginia Dignum (69) * Edward A. Fox (269) John McCarthy (65) * Lei Wang (250) Aaron Sloman (36) * John Mylopoulos (245) Carl Kesselman (312) Ewa Deelman (237) Thomas S. Huang (271) ... ...

Table 1 provides a partial result for the experiment of the proposed expert finding search task (Here we only consider a very simple and incomplete strategy, namely, find the author names who have at least one paper with “Artificial Intelligence” in its title). The left column is a partial list of results without social relation based refinement, which is just a list of author names without any relationship with the user. The right column is a partial list of results with social relation based refinement (The refinement is based on the social relations of the specified user that are extracted from the social network created in Section 2). Namely, the “Artificial Intelligence” authors whom the user “John McCarthy”

478

X. Ren et al.

knows are ranked to the front (As shown in the table, including himself). The results of the right column type seems more convenient for a user since the results which are ranked to the first ones seems to be familiar with the user compared to a list of irrelevant names. In a enterprise setting, if the found experts have some previous relationship with the employer, the cooperation may be smoother. In this example, a user’s collaborators appeared in two different scenarios, namely, in the coauthor network and domain experts knowledge base (here we consider SwetoDBLP as the experts knowledge base). Both of them are represented as semantic datasets using RDF, which enables the following connection. When a user tries to find domain experts, his social relations in the coauthor network are linked together with the domain experts knowledge base through the user’s name or URI. This connection brings two separate datasets together and help to refine the expert finding task.

4

Social Network Based Group Interest Models

A user and his/her friends, collaborators form a social network. the user’s interests may be affected by this social network since the network contains a group of other users who also have some interests. If they always communicate with each other, in the form of talking, collaboration, coauthoring, etc., their interests may be affected by each others’. If the user is affected by the social network based “group interests”, he/she may begin to search on the interesting topic to find relevant information. Hence, the social network based group interests may be serve as an essential environmental factor from user background for search refinement. Group Interest For a specific interest “t(i)”, its group interests for a specific author “u”, namely “GI(t(i), u)” can be quantitatively defined as: GI(t(i), u) = E(t(i), u, c) =

m

E(t(i), u, c),

c=1

1 (t(i) ∈ IctopN )

(1)

0 (t(i) ∈ IctopN )

where E(t(i), u, c) ∈ {0, 1}, if the interest t(i) appears both in the top N interests of a user and one of his/her friends’, then E(t(i), u, c) = 1, otherwise, E(t(i), u, c) = 0. For a specific user “u”, there are m friends in all, and the group interest of “t(i)” is the cumulative value of E(t(i), u, c) based on the m friends. In a word, group interest focuses on the cumulation of ranked interests from a specific user’s social network. Various models can be used to quantitatively measure and rank interests so that one can get the top N interests to produce the value of group interests. We defined 4 models in [7], here we briefly review them so that the comparison of group interests from the 4 perspectives can be made. Let i, j be positive integers, yt(i),j be the number of publications which are related to topic t(i) during the time interval j.


479

Cumulative Interest Cumulative interest, denoted as CI(t(i), n), is used to count the cumulative appear times of the interest t(i) during the n time intervals. It and can be acquired through: n CI(t(i), n) = yt(i),j . (2) j=1

It is used reflect a user’s over all interest on the specified topic within a time interval. Retained interest A person may be interested in a topic for a period of time but is likely to loose interest on it as time passes by if it has not appeared in some way for a long time. This phenomenon is very similar to the forgetting mechanism for cognitive memory retention. In [1] we introduced an retained interest model based on a power law function that cognitive memory retention [8] follows: RI(t(i), n) =

n j=1

−b yt(i),j × ATt(i) ,

(3)

where Tt(i) is the duration interested in topic t(i) until a specified time. For each −b time interval j, the interest t(i) might appear yt(i),j times, and yt(i),j × ATt(i) is the total retention of an interest contributed by that time interval. According to our previous studies, the parameters satisfy A = 0.855 and b = 1.295 [1]. Interest Cumulative Duration Interest cumulative duration, denoted as ILD(t(i)), is used to represent the longest duration of the interest t(i): ILD(t(i)) = max(ID(t(i))n ).

(4)

where n ∈ I + , ID(t(i))n is the interest duration when t(i) discretely appears (the time interval of the appeared interest is not directly continuous with the one of the previous appeared interest) for the nth time. Interest Longest Duration Interest longest duration, denoted as ICD(t(i)), is used to represent the cumulative duration of the interest t(i):

ICD(t(i)) =

n

(ID(t(i))n ).

(5)

n=1

where n ∈ I + is used to represent the nth discrete appearance of the interest t(i), and n is the total discrete appearance times of the interest t(i). The above 4 interest models are used for producing the top N interests. The corresponding group interests based the proposed models are: group cumulative interest (GCI(t(i), u)), group retained interest (GRI(t(i), u)), group cumulative duration (GCD(t(i), u)), and group longest duration (GLD(t(i), u)) respectively. Their calculation function is the same as GI(t(i), u), namely, GCI(t(i), u), GRI(t(i), u), GCD(t(i), u) and GLD(t(i), u) are special cases of GI(t(i), u).

480

X. Ren et al.

As a foundation for the development of social “group interest”, we analyzed all the authors’ retained interests values based on the SwetoDBLP dataset (more than 615,000 authors) using the introduced model, an RDF version of the interest enhanced DBLP author set has been released on the project page2 . Here we give an illustrative example on producing group interests based on retained interests. Using formula 3 and 1, and taking “Ricardo A. Baeza-Yates” as an example, a comparative list of top 7 retained interests of his own and his group retained interests (with 132 authors involved) is shown in Table 2.

Table 2. A comparative study of top 7 retained interests of a user and his/her group retained interests. (User name: Ricardo A. Baeza-Yates) Self Retained Interests Web Search Distributed Engine Mining Content Query

Value Group Retained Interests Value 7.81 Search (*) 35 5.59 Retrieval 30 3.19 Web (*) 28 2.27 Information 26 2.14 System 19 2.10 Query (*) 18 1.26 Analysis 14

Through Table 2 we can find that may be group retained interests are not the same as, but to some extent related to the user’s own retained interests (interesting terms that are marked with “*” are the same). As a step forward, we analyzed the overlap between a specific user’s own interests and his/her group interests. 50 most productive authors from the DBLP dataset (May 2010 version) are selected for the evaluation. The analysis considers 4 types of overlaps: – cumulative interest (CI(t(i), n)) and group cumulative interest (GCI(t(i), u)), – retained interest (RI(t(i), n)) and group retained interest (GRI(t(i), u)), – interest longest duration (ILD(t(i), j)) and group interest longest duration (GLD(t(i), u)), – interest cumulative duration (ICD(t(i), j)) and group interest cumulative duration (GCD(t(i), u)). The value of the overlaps are average values of the selected 50 authors. As shown in Figure 3, from the 4 perspectives, the overlaps range are within the interval [0.593, 0.667]. It means that no matter from which of these perspectives, the overlap between the users’ own interests and their group interests are at least greater than 59%. Take RI(t(i), n) vs GRI(t(i), u) and CI(t(i), n) vs GCI(t(i), u) as examples, Figure 4 shows that for most of the 50 authors, the overlaps are within the time interval [0.4,0.9]. 2

http://www.wici-lab.org/wici/dblp-sse and http://wiki.larkc.eu/csri-rdf


481

Fig. 3. Ratio of Overlap between differ- Fig. 4. A comparative study on the ent group interest values and the speci- overlap between RI and GRI, CI and fied author’s interest values GCI

Based on the results and analysis above, besides users’ own interests, their group interests and can be used as another source to refine the search process and satisfy various user needs.

5

Group Interests Based Search Refinement

In [1], according to the idea of retained interests model of a specific user, we developed a DBLP Search Support Engine (DBLP-SSE), which utilizes the user’s

Fig. 5. A screen shot of the DBLP Search Support Engine (DBLP-SSE)

482

X. Ren et al.

own retained interests to refine search on the SwetoDBLP dataset [2]. Based on the idea of group retained interest model introduced in 4, we developed a search support engine based on the SwetoDBLP dataset [2]. Figure 5 is a screen shot on the current version of the DBLP Search Support Engine (DBLP-SSE).

Table 3. Search Refinement using user’s retained interests and group retained interests Name: Ricardo A. Baeza-Yates Query : Intelligence List 1 : without any refinement (top 7 results) 1. PROLOG Programming for Artificial Intelligence, Second Edition. 2. Artificial Intelligence Architectures for Composition and Performance Environment. 3. The Mechanization of Intelligence and the Human Aspects of Music. 4. Artificial Intelligence in Music Education: A Critical Review. 5. Readings in Music and Artificial Intelligence. 6. Music, Intelligence and Artificiality. 7. Regarding Music, Machines, Intelligence and the Brain: An Introduction to Music and AI. List 2 : with user’s own interests constraints (top 7 results) interests : Web, Search, Distributed, Engine, Mining, Content, Query 1. SWAMI: Searching the Web Using Agents with Mobility and Intelligence. 2. Moving Target Search with Intelligence. 3. Teaching Distributed Artificial Intelligence with RoboRally. 4. Prototyping a Simple Layered Artificial Intelligence Engine for Computer Games. 5. Web Data Mining for Predictive Intelligence. 6. Content Analysis for Proactive Intelligence: Marshaling Frame Evidence. 7. Efficient XML-to-SQL Query Translation: Where to Add the Intelligence? List 3 : with group retained interests constraints (top 7 results) interests : Search, Retrieval, Web, Information, System, Query, Analysis 1. Moving Target Search with Intelligence. 2. A New Swarm Intelligence Coordination Model Inspired by Collective Prey Retrieval and Its Application to Image Alignment. 3. SWAMI: Searching the Web Using Agents with Mobility and Intelligence. 4. Building an information on demand enterprise that integrates both operational and strategic business intelligence. 5. An Explainable Artificial Intelligence System for Small-unit Tactical Behavior. 6. Efficient XML-to-SQL Query Translation: Where to Add the Intelligence? 7. Intelligence Analysis through Text Mining.

Table 3 shows a comparative study of search results without refinement, with user retained interests based refinement, and with group retained interests based refinement. Different search results are selected out and provided to users to meet their diverse needs. One can see that how the social network based group interests serve as an environmental factor that affect the search refinement process and help to get more relevant search results.


6

483

Evaluation and Analysis

Since the user interests and group interests are obtained from analysis based on real authors in the DBLP system. For the evaluation of the experimental results, the participants also need to be real authors in the system, preferably those with some publications distributed in different years. These constraints made finding the participants not easy. The participants are required to search for intelligence in the DBLP Search Support Engine (DBLP-SSE)3 that we developed based on the SwetoDBLP dataset [2]. Three lists of query results are provided to each of them. One is acquired based on unrefined query, and another two are refined by users’ own top 9 retained interests and top 9 group retained interests. They are required to judge which list of results they prefer. Currently, we have received evaluation results from 7 authors that have some publication listed in DBLP. Through an analysis of these results, we find that: 100% of these authors feel that the refined search results using users’ most recent RI(t(i), n) and GRI(t(i), u) are much better than the result list which does not have any refinement. 100% of them feel that the satisfaction degree of the two refined result lists are very close. 83.3% of them feel that refined results by the users’ own RI(t(i), n) is better than others. 16.7% of them feel refined results by GRI(t(i), u) are better than others. The refined list with the authors’ RI(t(i), n) is supposed to be the best one. Since the query constraints are all most related information that the users are interested in. Since the average overlap between users’ RI(t(i), n) and GRI(t(i), u) is around 63.8%, which means that interests from the author’s social network are very relevant to his/her own interests! That’s why the refined list with GRI(t(i), u) are also welcome and considered much better than the one without any refinement. It indicates that if one’s own interests can not be acquired, his/her friends’ interests also could help to refine the search process and results.

7


In this study, we provide some illustration on how the social relations and social network based interest models can help to refine searching on large scale data. For the scalability issue, this approach scales in the following way: No matter how large the dataset is, through the social relation based group interest models, the amount of most relevant results are always relatively small, and they are always ranked to the top ones for user investigation. The methods introduced in this paper are related but different from traditional collaborative filtering methods [9,10]. Firstly, both the user and their friends (e.g. coauthors, collaborators) do not comment or evaluate any search results (items) in advance. Secondly, interest retention models (both users’ own one and their group one) track the retained interests as time passed. The retained interests are dynamically changing but some of the previous interests are retained according 3

DBLP-SSE is available at http://www.wici-lab.org/wici/dblp-sse

484

X. Ren et al.

to the proposed retention function. Thirdly, following the idea of linked data [11], there is no need to have relevant information in one dataset or system. As shown in Section 3, user interests stored in different data sources are linked together for search refinement (user interests data and collaboration network data). Another example is that, if someone is recorded in the DBLP system wants to buy books on Amazon, he/she does not have to have a social relation on Amazon which can be used to refine the product search. Through the linked data from the group interests based on SwetoDBLP, the search process also could be refined. For now, semantic similarities of all the extracted terms have not been added into the retained interest models. Some preliminary experiments show that this may reduce the correlation between an author’s own retained interests and his/her group interests retention. For example, for the user “Guilin Qi”, both his current retained interests and his group interests contain “OWL” and “ontology”, which seem to be 2 different terms. But in practice, “OWL” is very related to “Ontology” (for their Normalized Google Distance [12], N GD(ontology, owl) = 0.234757, if N GD(x, y) ≤ 0.3, then x and y is considered to be semantically very related [12]). For the user “Zhisheng Huang”, the terms “reasoning” and “logic” are 2 important interests, while reasoning is very related to “logic” (N GD(logic, reasoning) = 0.2808). In our future work, we would like to use Google distance [12] to calculate the semantic similarities of interesting terms so that more accurate retained interests can be acquired and better search constraints can be found. We also would like to see whether other social network theories (such as six degree of separation) could help semantic search refinement in a scalable environment.

Acknowledgement This study is supported by the research grant from the European Union 7th framework project FP7-215535 Large-Scale Integrating Project LarKC (Large Knowledge Collider). We thank Yiyu Yao for his idea and discussion on Search Support Engine, Yang Gao for his involvement on the program development of interest retentions for authors in the SwetoDBLP dataset.

References 1. Zeng, Y., Yao, Y.Y., Zhong, N.: Dblp-sse: A dblp search support engine. In: Proceedings of the 2009 IEEE/WIC/ACM International Conference on Web Intelligence, pp. 626–630 (2009) 2. Aleman-Meza, B., Hakimpour, F., Arpinar, I., Sheth, A.: Swetodblp ontology of computer science publications. Web Semantics: Science, Services and Agents on the World Wide Web 5(3), 151–155 (2007) 3. Elmacioglu, E., Lee, D.: On six degrees of separation in dblp-db and more. SIGMOD Record 34(2), 33–40 (2005) 4. Zeng, Y., Wang, Y., Huang, Z., Zhong, N.: Unifying web-scale search and reasoning from the viewpoint of granularity. In: Liu, J., Wu, J., Yao, Y., Nishida, T. (eds.) AMT 2009. LNCS, vol. 5820, pp. 418–429. Springer, Heidelberg (2009)


485

5. Balog, K., Azzopardi, L., de Rijke, M.: Formal models for expert finding in enterprise corpora. In: Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (2006) 6. YimamSeid, D., Kobsa, A.: Expert-Finding Systems for Organizations: Problem and Domain Analysis and the DEMOIR Approach. In: Sharing Expertise: Beyond Knowledge Management, 1st edn., pp. 327–358. The MIT Press, Cambridge (2003) 7. Zeng, Y., Zhou, E., Qin, Y., Zhong, N.: Research interests: Their dynamics, structures and applications in web search refinement. In: Proceedings of the 2010 IEEE/WIC/ACM International Conference on Web Intelligenc (2010) 8. Anderson, J., Schooler, L.: Reflections of the environment in memory. Psychological Science 2(6), 396–408 (1991) 9. Goldberg, D., Nichols, D., Oki, B.M., Terry, D.: Using collaborative filtering to weave an information tapestry. Communications of the ACM 35(12), 61–70 (1992) 10. Resnick, P., Iacovou, N., Suchak, M., Bergstrom, P., Riedl, J.: Grouplens: An open architecture for collaborative filtering of netnews. In: Proceedings of the Conference on Computer Supported Cooperative Work, 175–186 (1994) 11. Bizer, C.: The emerging web of linked data. IEEE Intelligent Systems 24(5), 87–92 (2009) 12. Cilibrasi, R., Vitanyi, P.M.B.: The google similarity distance. IEEE Transaction on Knowledge and Data Engineering 19(3), 370–383 (2007)

An Empirical Approach for Opinion Detection Using Significant Sentences Anil Kumar K.M. and Suresha Department of Studies in Computer Science University of Mysore Manasagangothri Mysore, India {anilkmsjce,sureshabm}@yahoo.co.in

Abstract. In this paper we present an unsupervised approach to identify opinion of web users using a set of significant sentences from an opinionated text and to classify web user’s opinion into positive or negative. Web users document their opinion in opinionated sites, shopping sites, personal pages etc., to express and share their opinion with other web users. The opinion expressed by web users may be on diverse topics such as politics, sports, products, movies etc. These opinions will be very useful to others such as, leaders of political parties, selection committees of various sports, business analysts and other stake holders of products, directors and producers of movies as well as to the other concerned web users. We use an unsupervised semantic based approach to find users opinion. Our approach first detects subjective phrases and uses these phrases along with semantic orientation score to identify user’s opinion from a set of empirically selected significant sentences. Our approach provides better results than the other approaches applied on different data sets.

1 Introduction The rapid development of world wide web and its related technologies have fueled the popularity of the web with all sections of society. The web has been used by many firms such as governments, business houses, industries, educational institutions etc., to make them familiar and accessible globally. An individual web user is provided with an opportunity to obtain and share knowledge. The web is the origin of many research activities and one interesting area of research is to mine users opinion from web on diverse topics like politics, movies, educational institutions, products etc. The study of opinions is useful to both producers and consumers of the topic. The producers can be manufacturers of automobiles, movie producers, editor of news article, digital product manufactures etc., who are very much interested to find opinion of a user. The consumers are individual users who express their opinion and want to share it with other web users. In this paper, we attempt to find opinion of the users using a set of significant sentences from the opinionated texts. The set of significant sentences is empirically selected after observing a large collection of opinionated texts available at opinionated sites, e-commerce sites, blogs etc. Our intuition is that, the opinion from such a set of A. An et al. (Eds.): AMT 2010, LNCS 6335, pp. 486–497, 2010. c Springer-Verlag Berlin Heidelberg 2010

An Empirical Approach for Opinion Detection Using Significant Sentences

487

significant sentences reflects user’s opinion on a subject. The other sentences of the opinionated text will have opinion on features of a subject. An opinionated text discussing on digital camera may have sentences with an overall opinion of the digital camera as well as opinion on features of digital camera such as lens, battery backup, size etc. It becomes critical to detect opinion of the web user’s from such significant sentences and classify it as positive or negative. For example, consider the following opinionated texts of web users obtained from an opinionated site reviewcentre. The collected opinionated texts were retained in the original form, no attempt was made to correct the grammatical mistakes of web users from these opinionated texts. Example 1. It is a good player. I guess I was kind of expecting a better picture on my Insignia 27” LCD with a regular dvd boosted up to “near HD DVD” quality. In the end it’s all good though. If I had to do it over again I would have saved some of my money and bought a Philips DVP3140/37 player which plays just about every dvd I own without a single buzz, click, skip or lockup. That’s more than I can say for any other dvd player I have ever seen. Example 2. Easy setup and easy to use. I connected it to a 19” HDTV using the component cables that came with the TV. Using the upscale setting and a standard DVD with TV at 720p, the picture is bright and beautifully detailed. Much better picture than I had expected. I definitely recommend this player. It is thinner but a bit wider than my last player, so be sure it will fit your allotted space. Example 3. Bought a PS3 so the old Toshiba standard def just wouldn’t do anymore. Found this tv at HH Greggs about 4 months ago during a big sale they were having for about $1250. This tv overall has been great so far. The full 1080p works great for gaming and playing blu-rays. Directv HD programming is also crystal clear even though it is not in 1080p resolution. If your looking for a great tv with the same quality as a Sony, but slightly cheaper, I can fully recommend this TV. Example 1 refers to an opinion of a user on philips DVD product. The overall opinion of a user is positive. The overall user’s opinion of the product is recorded in first sentence of the opinionated text. Example 2 show user’s opinion on DVD player in a sentence between the first and last sentence. Example 3 shows user’s opinion on PS3 T.V. in the last sentence. The sentences in Bold refer to overall user’s opinion of the product in an opinionated text. We observe from the afore mentioned examples as well as from the other opinionated texts, users document their overall opinion in few sentences and use other sentences to expresses opinion on different features of the product. We believe that it is important to identify a set of significant sentences that provides an overall opinion of the users. The opinion obtained from such significant sentences is considered as the actual opinion of a web user. In this paper, we focus on finding opinion of web users only on products using a set of significant opinionated sentences. A set of significant sentences here refers to first sentence, last sentence and a sentence with maximum semantic orientation score and they are represented by SF , SL and Smax . The remainder of this paper is organized as follows: In Section 2 we give a brief description of related work. Then, in Section 3, we discuss our methodology. In Section 4, the experiments and results are discussed. Conclusion is discussed in Section 5.

488

Anil Kumar K.M. and Suresha

2 Related Work Opinion mining is a recent sub discipline of information retrieval which is not about the topic of a document, but with the opinion it expresses [1]. In literature, opinion mining is also known as sentiment analysis [7], sentiment classification [8], affective classification [21] and affective rating [16]. It has emerged in the last few years as a research area, largely driven by interests in developing applications such as mining opinions in online corpora, or customer relationship management, e.g., customer’s review analysis [21]. Hatzivassiloglou and McKeown [19] have attempted to predict semantic orientation of adjectives by analyzing pairs of adjectives (i.e., adjective pair is adjectives conjoined by and, or, but, either-or, neither-nor) extracted from a large unlabelled document set. Turney [14] has obtained remarkable results on the sentiment classification of terms by considering the algebraic sum of the orientations of terms as representative of the orientation of the document. Turney an Littman [15] have bootstrapped from a seed set, containing seven positive and seven negative words, and determined semantic orientation according to Pointwise Mutual Information-Information Retrieval (PMI-IR) method. Wang and Araki [20] proposed a variation of the Semantic Orientation-PMI algorithm for Japanese for mining opinion in weblogs. They applied Turney method to Japanese webpage and found results slanting heavily towards positive opinion. They proposed balancing factor and neutral expression detection method and reported a well balanced result. Opinion observer [6] is the sentiment analysis system for analyzing and comparing opinions on the web. The product features are extracted from noun or noun phrases by the association miner. They use adjectives as opinion words and assign prior polarity of these by WordNet exploring method. The polarity of an opinion expression which is a sentence containing one or more feature terms and one or more opinion words is assigned a dominant orientation. The extracted features are stored in a database in the form of feature, number of positive expression and number of negative expression. Kamps et al [12] have focused on the use of lexical relations defined in WordNet. They defined a graph on the adjectives contained in the intersection between the Turney’s seed set and WordNet, adding a link between two adjectives whenever WordNet indicate the presence of a synonymy relation between them. The authors defined a distance measure d (t1, t2) between terms t1 and t2, which amounts to the length of the shortest path that connects t1 and t2. The orientation of a term is then determined by its relative distance from the seed terms good and bad. Our work differs from the afore mentioned studies, by finding opinion of a user only with a few significant opinionated sentences from an opinionated text. We do not consider opinion of all other sentences found in an opinionated text. Our work uses not only adjectives but other part-of-speech like verb, adverb etc., to capture opinionated words for efficient opinion detection.

3 Methodology We collected six data sets for our work on different products like sony cybershot, nikon coolpix, cannonG3, philips dvd player etc. The first data set consist of 250 opinionated texts on five different products, collected from results of various search engines.


489

The second data set is collection of 400 opinionated texts obtained from different opinionated sites like Amazon, CNet, review centre, bigadda, rediff etc. The third data set consisting of 140 opinionated texts on product is obtained from [3]. These data sets contained a balanced set of positive and negative opinionated texts. The remaining three data sets obtained from [22] contained an unbalanced set of positive and negative opinionated texts. The fourth data set contains 32 opinionated texts, the fifth data set contained 95 opinionated texts and the final data set contained 45 opinionated texts. In our approach we pass an opinionated text to a sentence splitter program. The sentences obtained from the program were input to a part of speech tagger. The tagger used in our approach is Monty Tagger [11]. Extraction patterns are applied to the tagged opinionated sentences to obtain opinionated phrases that are likely to contain user’s opinion. In this paper we use only two phrase extraction patterns. Table 1 shows a few extraction patterns used to obtain opinionated phrases from opinionated sentences. Here, JJ represent adjective and NN/NNS, VB/VBD/VBN/VBG, RB/RBR/RBS represents different forms of noun, verb and adverb. The opinionated phrases are then subjected to Sentiment Product Lexicon (SPL) for capturing only subjective or opinionated phrases. This is necessary as some phrases obtained after application of extraction patterns may be non subjective. Table 1. Extraction patterns Slno. 1 2 3 4 5

First Word JJ RB,RBR or RBS JJ NN or NNS RB,RBR or RBS

Second Word NN or NNS JJ JJ JJ VB,VBD,VBN or VBG

Third Word anything not NN nor NNS not NN nor NNS not NN or NNS anything

Sentiment Product Lexicon is collection of General lexicon and Domain lexicon. General lexicon maintains a list of positive and negative words by collecting opinion words that are positive or negative from sources like General Inquirer [17], Subjective clues [18] and list of adjectives [13]. Domain lexicon maintains a list of positive or negative words from the domain context. We found words like cool, revolutionary etc., appeared in negative list of General lexicon. These words were used to express positive opinion by web user’s. Hence we created a domain lexicon to have opinion words from the domain perspective. The details of construction of General lexicon and Domain lexicon are made available in [4]. In this paper we use these lexicons to identify neutral phrases. The need to identify neutral phrases arises, when extraction patterns yields phrases that are mostly opinionated, but a few non opinionated phrases in some cases. For example, we obtain poor/JJ quality/NN, too/RB grainy/JJ, not/RB recommend/VB, not/RB working/VBG etc., as opinionated phrases. Also a few non opinionated phrases like main/JJ theme/NN, however/RB is/VBZ, not/RB be/VB etc., are obtained. It is important to identify and

490


discard these kind of neutral phrases as they can influence the polarity of opinionated sentence. Sentiment product lexicon can be expressed as SPL = {GLP , GLN , DLP , DLN }

(1)

Where GL GL DL DL

P : Positive words in General lexicon N : Negative words in General lexicon P : Positive words in Domain lexicon N : Negative words in Domain lexicon

For example, consider an opinionated sentence “This is a bad phone.” When the tagger is applied to input sentence, we get the following tagged sentence “This/DT is/VBZ a/DT bad/JJ phone/NN ./.”. Application of extraction patterns from Table 1 will obtain bad/JJ phone/NN as opinion word from the sentence. Sentiment Product Lexicon is used to detect neutral phrases. We consider the extracted phrases or words namely word1 and word2 from an opinionated sentence as neutral if none of the words extracted are found in Sentiment Product Lexicon. From the above example word1 is bad and word2 is phone. We find whether word2 is in positive or negative list of Domain lexicon. If word2 is present in any one of the list in Domain lexicon, polarity of the word will be similar to polarity of list in which it is found. If it is not in positive or negative list of Domain lexicon, then positive and negative list of General lexicon is consulted to find the polarity of a word. If word2 is neither present in Domain lexicon nor in General lexicon, we assume word2 to have neutral polarity, in such a case we use word1 instead of word2, and find polarity of word1 similar to polarity of word2 afore discussed. If polarity is found, then polarity is for the phrase consisting of both word1 and word2. If polarity is not found, we assume both word1 and word2 to be neutral. If a word, either word1 or word2, is present in both Domain lexicon and General lexicon, polarity of word will be similar to polarity of Domain lexicon. If word1 is a negator such as ’not’, the polarity of word2 will be opposite to an earlier obtained polarity of word2. For example the phrase ”not good”, here word1 is ’not’ and word2 is ’good’. The polarity of word2 is positive, since word2 is prefixed by word1 i.e. ’not’. The polarity of phrase is negative. We retain only those phrases that have a polarity and discard phrases that are neutral. We compute the strength of semantic orientation of phrases using the Equation 2 SO(phrase) = log2 [

hits(ws10 (phrase, excellent)).hits(poor) ] hits(ws10 (phrase, poor)).hits(excellent)

(2)

Where SO is the Semantic Orientation. The seed words excellent and poor are part of five star review rating system. SO is computed by finding association of these phrases with seed words from web corpus. It is found by querying the search engine and recording the number of hits returned by search engine for phrases closer to seed words in a window size (ws) of ten words.


491

We used Google search engine to find semantic orientation of the extracted phrases and compute average semantic orientation of opinionated sentences. Our choice to use Google search engine is that, it indexes more number of pages than other search engines. Even though it is dynamic in nature, we believe the semantic orientation of phrases obtained using Google reflects diverse nature of web users in dissemination of information. From the above example, we obtain SO (bad phone) = 4.20. The actual polarity of the phrase is negative, but SO value obtained is positive. We shift the polarity of the phrase in consultation with Sentiment Product Lexicon. We multiply the strength of semantic orientation of phrases by +1, if the phrases are positive, and -1, if phrases are negative. The new value for our example will be SO (bad phone) = - 4.20. We compute the average semantic orientation of all phrases in an opinionated sentence. This is done for all the opinionated sentences in an opinionated text. We obtain opinion from an opinionated text by considering a set of significant opinionated sentences. Edmond [9] used key words, cue words, title words and structural indicators( sentence location) to identify the significant sentence. The significant sentence was used to convey to the reader the substance of the document. One of the methods proposed is called sentence location method. The sentence location is based on the hypothesis that the sentences occurring under certain headings are positively relevant and the topic sentences tend to occur very early or very late in a document. The afore mentioned study highlights the importance of sentence location. We empirically select and use the first opinionated sentence, opinionated sentence with maximum semantic orientation value and last opinionated sentence of the opinionated text as a set of significant opinionated sentences (S OS). SOS = SF + Smax + SL

(3)

An opinionated text is classified as positive, if the semantic orientation of significant opinionated sentences using Equation 3 is greater than a Threshold and negative, when semantic orientation of significant opinionated sentences is less than a Threshold. The Threshold used to classify opinionated sentences as positive or negative is 0.

4 Experiments and Results We use our approach afore discussed to find an opinion from a few significant opinionated sentences. We compare the result of our approach with an approach discussed in [14] as well as with an approach discussed in [6]. We implemented both approaches to find opinion of an user on three data sets. The first approach aims to find opinion from all the sentences of an opinionated text. It makes use of Google search engine to find semantic orientation of phrases, instead of Altavista Search engine as used by [14]. Since Altavista no longer supports proximity search, hence we used Google search engine. The second approach discussed in [6] finds opinion of an user on features of product by considering only adjective as potential opinion words. The polarity of the opinion words is determined from a seed list. Here, we do not find feature of the product, but use the same method to obtain opinion from all sentences that contain adjective in

492


an opinionated text. We use SPL to determine the polarity of opinion words than the original seed list used in [6] Table 2 shows the results of two approaches discussed in [14] and [6] on data set 1, data set 2, data set 3, data set 4, data set 5 and data set 6. We compute the classification accuracy by dividing the sum of true positives and true negatives with total number of items to be classified. Table 2. Results of Approaches Slno. 1 2 3 4 5 6 7 8 9 10 11 12

Approach

Data sets

Accuracy

[14] [14] [14] [14] [14] [14] [6] [6] [6] [6] [6] [6]

Data set 1 Data set 2 Data set 3 Data set 4 Data set 5 Data set 6 Data set 1 Data set 2 Data set 3 Data set 4 Data set 5 Data set 6

71.85% 65.75% 65.42% 68.80% 75.14% 73.80% 74% 76.06% 73.56% 78.16% 80.30% 78%

Table 3. Results of Our Approaches Slno. 1 2 3 4 5 6

Data set

SF

SL

S F+ S L

S max

S OS

Data set 1 Data set 2 Data set 3 Data set 4 Data set 5 Data set 6

44.28% 46.29% 27.54% 65.65% 45.26% 62.22%

47.14% 48.89% 43.52% 65.65% 50.52% 55.55%

45.71% 47.59% 35.53% 65.65% 47.89 % 58.88 %

75.71% 80.87% 71.01% 84.37% 82.10% 82.22%

77.14% 81.24% 69.59% 87.5% 84.21% 86.77%

We conducted a series of experiments to find opinion from sentences of an opinionated text. As already mentioned, our intuition is that, a few users express their actual opinion of the product in the first sentence before elaborating the same in remaining sentences. Also, a few other users express their actual opinion of the product in the last sentence after initially expressing their opinion on different features of the product. The remaining users document their opinion in a sentence apart of first and last sentences. Table 3 shows result of our experiment on different data sets. We started the experiment by finding opinion of the user from first sentence of opinionated text. We obtain an accuracy of 44.28%, 46.29%, 27.54%, 65.65%, 45.26% and 62.22% for data set 1, data set 2, data set 3, data set 4, data set 5 and data set 6. An accuracy of 47.14%, 48.89%, 43.52%, 65.65%, 50.52% and 55.55% were obtained while considering users opinion


493

Fig. 1. Accuracy of Our Approach for Different values of W

only from the last sentence of opinionated texts from different data sets. We achieved an accuracy of 45.71%, 47.59%, 35.53%, 65.65%, 47.89% and 58.88% considering opinion from first and last sentence of opinionated texts from different data sets. Next, we used a sentence in an opinionated text with maximum semantic score to find users opinion. We obtain an accuracy of 75.71%, 80.87%, 71.01%, 84.37%, 82.10% and 82.22% for different data sets. After obtaining the results on first sentence, last sentence and sentence with maximum semantic orientation value, we compute accuracy of users opinion using Equation 3. An accuracy of 77.14%, 81.24%, 69.59%, 87.5%, 84.21% and 86.77% were obtained on different data sets. The first and last sentences of a few opinionated texts in data set 3, inspite of recording the actual users opinion were having a smaller semantic orientation values for opinionated phrases in these sentences. The values of these sentences become insignificant due to presence of a sentence with maximum semantic orientation value. This factor contributed to a loss of accuracy on data set 3 using Equation 3 as compared to accuracy of S max. We reformulated Equation 3 to capture users opinion from first and last sentences with smaller semantic orientation values. SOS = Smax + W (SF + SL )

(4)

We experimented with different values of W. Figure 1 shows the accuracy of our result obtained for different values of W. We observe that, a good accuracy is obtained for W =10 and the accuracy remains consistent for higher values of W. Figures 2, 3, 4,5, 6, 7 show accuracy of our approach using Equation 4 as against the other implemented approaches on different data sets. The results obtained by our approach using Equation 4 are found to be better than the results documented in Table 2 and Table 3. We observe from Figure 4, the result obtained by approach discussed in [6] is better than our approach on data set 3. We obtain an average accuracy of 73.56% with approach discussed in [6], with positive accuracy of 94.28 % and negative accuracy of 52.85%. Our approach provides an average accuracy of 69.59%, with positive accuracy of 67.14% and negative accuracy of 72.05%. Our approach provides a balanced result as compared to a result obtained using the approach discussed in [6] on the same data set. It is also found to be better than the average accuracy of 69.3% recorded in [3] on data set 3.

494


Fig. 2. Accuracy of Three Approaches on Data Set 1







495

496


5 Conclusion We have conducted a set of experiments in our approach to find opinion of web users from a fixed number of significant opinionated sentences in an opinionated text. The results obtained from the experiments are encouraging. It also highlights the importance of sentence position in detection of opinion from an opinionated text. We use Sentiment Product Lexicon to remove neutral phrases and shift polarity of few phrases based on some heuristics. Our proposed approach using Equation 4 provides better results compared to results considering all sentences from an opinionated texts. Our approach provides better results on data sets comprising of both balanced and unbalanced positive and negative opinionated texts.

References 1. Andrea, E., Fabrizio, S.: Determining term subjectivity and term orientation for opinion mining. In: Proceedings of 11th Conference of the European Chapter of the Association for Computational Linguistics, Trento, Italy (2006) 2. Andrea, E., Fabrizio, S.: Determining the semantic orientation of terms through gloss classification. In: Proceedings of 14th ACM International Conference on Information and Knowledge Management, Bremen, Germany, pp. 617–624 (2005) 3. Alistair, K., Diana, I.: Sentiment Classification of Movie and Product Reviews Using Contextual Valence Shifters. In: Proceedings of FINEXIN 2005, Workshop on the Analysis of Informal and Formal Information Exchange during Negotiations, Canada (2005) 4. Anil Kumar, M.K., Suresha: Identifying Subjective Phrases From Opinionated Texts Using Sentiment Product Lexicon. International Journal of Advanced Engineering & Applications 2, 63–271 (2010) 5. Anil Kumar, M.K., Suresha: Detection of Neutral Phrases and Polarity Shifting of Few Phrases for Effective Classification of Opinionated Texts. International Journal of Computational Intelligence Research 6, 43–58 (2010) 6. Bing, L., Minqing, H., Junsheng, C.: Opinion Observer: Analyzing and Comparing Opinions on the Web, Chiba, Japan (2005) 7. Bo, P., Lee, L.: A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In: Proceedings of 42nd Meeting of the Association for Computational Linguistics, Barcelona, Spain, pp. 271–278 (2004) 8. Bo, P., Lee, L., Shivakumar, V.: Thumbs up? sentiment classification using machine learning techniques. In: Proceedings of 7th Conference on Empirical Methods in Natural Language Processing, Philadelphia, US, pp. 79–86 (2002) 9. Edmundson, H.P.: Journal of the Association for Computing Machinery 16(2) (1969) 10. Review centre, http://www.reviewcentre.com/ 11. Hugo: MontyLingua: An end-to-end natural language processor with common sense (2003) 12. Jaap, K., Maarten, M., Robert, J., Mokken, M., De, R.: Using wordnet to measure semantic orientation of adjectives. In: Proceedings of 4th International Conference on Language Resources and Evaluation, Lisbon, Portugal, pp. 1115–1118 (2004) 13. Maite, T., Jack, G.: Analyzing appraisal automatically. In: Proceedings of the AAAI Symposium on Exploring Attitude and Affect in Text: Theories and Applications, California, US (2004) 14. Turney, P.D.: Thumbs up or thumbs down? Semantic orientation applied to unsupervised classification of reviews. In: Proceedings of 40th Annual Meeting of the Association for Computational Linguistics, Philadelphia, US, pp. 417–424 (2002)


497

15. Turney, P.D., Littman, M.L.: Measuring praise and criticism: Inference of semantic orientation from association. ACM Transactions on Information Systems, 315-346 (2003) 16. Owsley, S., Sood, S., Hammond, K.J.: Domain specific affective classification of document. In: Proceedings of AAAI-2006 Spring Symposium on Computational Approaches to Analyzing Weblogs, California, US (2006) 17. Stone, P.J.: Thematic text analysis: New agendas for analyzing text content. In: Roberts, C. (ed.) Text Analysis for the Social Sciences. Lawrence Erlbaum, Mahwah (1997) 18. Theresa, W., Janyce, W., Paul, H.: Recognizing Contextual Polarity in Phrase-Level Sentiment Analysis. In: Proceedings of HLT/EMNLP, Vancouver, Canada (2005) 19. Vasileios, H., McKeown, K.R.: Predicting the semantic orientation of adjectives. In: Proceedings of 35th Annual Meeting of the Association for Computational Linguistics, Madrid, Spain, pp. 174–181 (1997) 20. Wang, A.: Modifying SO-PMI for Japanese Weblog Opinion Mining by Using a Balancing Factor and Detecting Neutral Expressions. In: Proceedings of NAACL HLT 2007, New York, US, pp. 189–192 (2007) 21. Youngho, K., Sung, H.M.: Opinion Analysis based on Lexical Clues and their Expansion. In: Proceedings of NTCIR-6 Workshop Meeting, pp. 308-315. Tokyo, Japan (2007) 22. http://www.cs.uic.edu/~ liub/FBS/sentiment-analysis.html

Extracting Concerns and Reports on Crimes in Blogs Yusuke Abe1 , Takehito Utsuro1 , Yasuhide Kawada2, Tomohiro Fukuhara3 , Noriko Kando4 , Masaharu Yoshioka5 , Hiroshi Nakagawa6, Yoji Kiyota6, and Masatoshi Tsuchiya7 1

University of Tsukuba, Tsukuba, 305-8573, Japan 2 Navix Co., Ltd., Tokyo, 141-0031, Japan 3 National Institute of Advanced Industrial Science and Technology, Tokyo 135-0064, Japan 4 National Institute of Informatics, Tokyo, 101-8430, Japan 5 Hokkaido University, Sapporo, 060-0814, Japan 6 University of Tokyo, Tokyo, 113-0033, Japan 7 Toyohashi University of Technology, Toyohashi, 441-8580, Japan

Abstract. Among other domains and topics on which some issues are frequently argued in the blogosphere, the domain of crime is one of the most seriously discussed by various kinds of bloggers. Such information on crimes in blogs is especially valuable for those who are not familiar with tips for preventing being victimized. This paper proposes a framework of extracting people’s concerns and reports on crimes in their own blogs. In this framework, we solve two tasks. In the first task, we focus on experts in crime domain and address the issue of extracting concerns on crimes such as tips for preventing being victimized. In the evaluation of this first task, we show that we successfully rank expert bloggers high in the results of blog feed retrieval. In the second task, on the other hand, we focus on victims of criminal acts, and address the issue of extracting reports on being victimized. In the evaluation of this second task, we show that we outperform blog post ranking based on the search engine API by incorporating dependency relations for identifying victims’ blog posts.

1

Introduction

Weblogs or blogs are considered to be one of personal journals, market or product commentaries. While traditional search engines continue to discover and index blogs, the blogosphere has produced custom blog search and analysis engines, systems that employ specialized information retrieval techniques. With respect to blog analysis services on the Internet, there are several commercial and noncommercial services such as Technorati, BlogPulse [1], kizasi.jp (in Japanese), and blogWatcher (in Japanese) [2]. With respect to multilingual blog services, Globe of Blogs provides a retrieval function of blog articles across languages. Best Blogs in Asia Directory also provides a retrieval function for Asian language blogs. Blogwise also analyzes multilingual blog articles. A. An et al. (Eds.): AMT 2010, LNCS 6335, pp. 498–509, 2010. c Springer-Verlag Berlin Heidelberg 2010

Extracting Concerns and Reports on Crimes in Blogs

499

Fig. 1. Overall Framework of Extracting Concerns and Reports on Crimes in Blogs

Among other domains and topics on which some issues are frequently argued in the blogosphere, the domain of crime is one of the most seriously discussed by various kinds of bloggers. One type of such bloggers are those who have expert knowledge in crime domain, and keep referring to news posts on criminal acts in their own blogs. Another type of bloggers who have expert knowledge also often post tips for how to prevent certain criminal acts. Furthermore, it is surprising that victims of certain criminal acts post blog articles on their own experiences. Blog posts by such various kinds of bloggers are actually very informative for both who are seeking for information on how to prevent certain criminal acts and who have been already victimized and are seeking for information on how to solve their own cases. Such information on crimes in blogs is especially valuable for those who are not familiar with tips for preventing being victimized. Based on this observation, this paper proposes a framework of extracting people’s concerns and reports on crimes in their own blogs. The overview of the proposed framework is shown in Figure 1. In this paper, we focus on those which represent relatively narrow range of concepts of criminal acts, namely, “fraud” and “Internet crime”. We also extract concerns and reports on crimes from English and Japanese blogs. In this framework, we solve two tasks. In the first task, we focus on experts in crime domain and address the issue of extracting concerns on crimes such as tips for preventing being victimized (to be presented in section 4). Its major component is designed as the blog feed retrieval procedure recently studied in the blog distillation task of TREC 2007 blog track [3]. In this first task, we regard blog feeds as a larger information unit in the blogosphere. We intend to retrieve blog feeds which roughly follow the criterion studied in the blog distillation task, which can be summarized as Find me a blog with a principle, recurring interest

500

Y. Abe et al.

in X. More specifically, in the blog distillation task, for a given target X, systems should suggest feeds that are principally devoted to X over the time span of the feed, and would be recommended to subscribe to as an interesting feed about X. In the evaluation of this first task, we show that we successfully rank expert bloggers high in the results of blog feed retrieval. In the second task, on the other hand, we focus on victims of criminal acts, and address the issue of extracting reports on being victimized (to be presented in section 5). In this second task, we propose a technique which is based on detecting linguistic expressions representing experiences of being a victim of certain fraud, such as “being a victim of ” and “deceived”. In the evaluation of this second task, we show that we outperform blog post ranking based on the search engine API by incorporating dependency relations for identifying victims’ blog posts. We also show that victims of criminal acts such as fraud and Internet crime sometimes post one or two articles to their own blogs just after they are victimized. In most cases, those victims do not keep posting articles related to those crimes, and hence, their blog feeds are not ranked high in the result of the first task of extracting expert bloggers in crime domain.

2

Related Works

There exist several works on studying cross-lingual analysis of sentiment and concerns in multilingual news [4,5,6,7], but not in blogs. [4] studied how to combine reports on epidemic threats from over 1,000 portals in 32 languages. [5] studied how to combine name mentions in news articles of 32 languages. [6] also studied mining comparative differences of concerns in news streams from multiple sources. [7] studied how to analyze sentiment distribution in news articles across 9 languages. Those previous works mainly focus on news streams and documents other than blogs. As another type of related work, [8,9] studied how to collect linguistic expressions which represent trouble situation, where Web documents including writer’s own trouble experiences such as blogs are used for evaluation. Their technique itself as well as the collected expressions representing trouble situation can be applicable to our task of identifying blog posts including blogger’s own experiences of being victimized in certain criminal acts.

3

Topics in the “Fraud / Internet Crime” Domain

In this paper, as topics in the domain of criminal acts, we focus on “fraud” and “Internet crime”. We first refer to Wikipedia (English and Japanese versions1) 1

http://{en,ja}.wikipedia.org/. The underlying motivation of employing Wikipedia is in linking a knowledge base of well known facts and relatively neutral opinions with rather raw, user generated media like blogs, which include less well known facts and much more radical opinions. We regard Wikipedia as a large scale ontological knowledge base for conceptually indexing the blogosphere. It includes about 3,321,00 entries in its English version, and about 682,000 entries in its Japanese version (checked at June, 2010).


501

(Note: “A term tx ” (“a term ty ”) in the nodes above indicates that ty is not listed as a Wikipedia entry, nor extracted from any of Wikipedia entries, but translated from tx by Eijiro. ) Fig. 2. Topics and Related Terms in the “Fraud / Internet Crime” Domain Table 1. Statistics of “Fraud / Internet Crime” # of Cases (sent to the # of Hits in the Blogosphere court in the Year of 2008) (checked at Sept. 2009) ID Topic U.S.A. Japan English Japanese 1 Internet fraud 72,940 N/A 21,300 61,600 2 (Auction fraud) 18,600 1,140 1,760 44,700 3 (Credit card fraud) 6,600 N/A 43,900 8,590 4 (Phishing) N/A 479,000 136,000 5 Bank transfer scam N/A 4,400 30 349,000 6 Counterfeit money N/A 395 16,800 40,500 7 Cyberstalking N/A 20,300 32,100 8 Cyber-bullying N/A 38,900 45,700

and collect entries listed at the categories named as “fraud” and “Internet crime” as well as those listed at categories subordinate to “fraud” and “Internet crime”. Then, we require entry titles to have the number of hits in the blogosphere over 10,000 (at least for one language)2 . At this stage, for the category “fraud”, we 2

We use the search engine“Yahoo!” API (http://www.yahoo.com/) for English, and the Japanese search engine “Yahoo! Japan” API (http://www.yahoo.co.jp/) for Japanese. Blog hosts are limited to 2 for English (blogspot.com,wordpress.com) and 3 for Japanese (FC2.com,goo.ne.jp,Seesaa.net).

502

Y. Abe et al.

have 68 entries for English and 20 for Japanese, and for the category “Internet crime”, we have 15 entries for English and 8 for Japanese. Next, we manually examine all of those categories, and select those which exactly represent certain criminal acts. Then, for the category “fraud”, we have 14 entries for English and 10 for Japanese, and for the category “Internet crime”, we have about 6 entries for English and 5 for Japanese. Out of those entries, Figure 2 shows some sample topics3 . In the figure, the category “Internet fraud” is an immediate descendant of both “fraud” and “Internet crime”, where three entries listed at this category are selected as sample topics. The figure also shows samples of related terms extracted from those selected Wikipedia entries which are to be used for ranking blog feeds/posts in section 4.1. For those selected sample topics, Table 1 shows the number of cases actually sent to the court in U.S.A and in Japan4 . The table also shows the number of hits of those sample entry titles in the blogosphere.

4

Extracting Concerns on Crimes in Blogs by Applying General Framework of Expert Blog Feeds Retrieval

4.1

Retrieving and Ranking Blog Feeds/Posts

Out of the two tasks introduced in section 1, this section describes the first one. In this first task, we simply apply our general framework of expert blog feeds retrieval [10,11] to extracting expert bloggers in crime domain. First, in order to collect candidates of blog feeds for a given query, we use existing Web search engine APIs, which return a ranked list of blog posts given a topic keyword. We use the search engine “Yahoo!” API for English, and the Japanese search engine “Yahoo! Japan” API for Japanese. Blog hosts are limited to those listed in section 3. Then, we employ the following procedure for blog feed ranking: i) Given a topic keyword, a ranked list of blog posts are returned by a Web search engine API. ii) A list of blog feeds is generated from the returned ranked list of blog posts by simply removing duplicated feeds. We next automatically select blog posts that are closely related to the given query, which is a title of a Wikipedia entry. To do this, we first automatically extract terms that are closely related to each Wikipedia entry. More specifically, 3

4

For some of those selected samples, only English or Japanese term is listed as a Wikipedia entry, and the entry in the opposite language is not listed as a Wikipedia entry. In such a case, its translation is taken from an English-Japanese translation lexicon Eijiro (http://www.eijiro.jp/, Ver.79, with 1.6M translation pairs). Statistics are taken from the Internet Crime Complaint Center (IC3), U.S.A. (http://www.ic3.gov/), National Police Agency, Japan (http://www.npa.go.jp/english/index.htm), and NPA Japan Countermeasure against Cybercrime (http://www.npa.go.jp/cyber/english/index.html).


503

Table 2. Statistics of # of topic-related terms extracted from Wikipedia entries, and blog feeds/posts (English/Japanese)

ID

Topic

1 2 3 4 5 6 7 8

Internet fraud (Auction fraud) (Credit card fraud) (Phishing) Bank transfer scam Counterfeit money Cyberstalking Cyber-bullying

# of topic-related terms from # of blog feeds # of blog posts Wikipedia 182 / 76 48 / 60 1576 / 353 24 /36 40 / 38 224 / 121 28 / 181 50 / 31 1086 / 143 172 / 63 49 /118 8982 / 1118 60 / 96 4 / 132 13 / 2617 175 /84 41 / 96 186 / 695 33 / 29 49 / 39 727 / 242 52 / 65 49 / 89 4278 / 613

from the body text of each Wikipedia entry, we extract bold-faced terms, anchor texts of hyperlinks, and the title of a redirect, which is a synonymous term of the title of the target page. Then, blog posts which contain the entry title or at least one of the extracted related terms as well as synonymous terms are automatically selected. For each of the sample topics shown in Figure 2, Table 2 shows the numbers of terms that are closely related to the topic and are extracted from each Wikipedia entry. Then, according to the above procedure, blog posts which contain the topic name or at least one of the extracted related terms (including synonymous terms) are automatically selected. Table 2 also shows the numbers of the selected blog feeds/posts. Finally, we rank the blog feeds/posts in terms of the entry title and the related terms (including synonymous terms) extracted from the Wikipedia entry. – Blog posts are ranked according to the score: weight(type(t)) × f req(t) t

where t is the topic name or one of the extracted related terms (including synonymous terms), and weight(type(t)) is defined as 3 when type(t) is the entry title or the title of a redirect, as 2 when type(t) is a bold-faced term, and as 0.5 when type(t) is an anchor text of a hyperlink to another entry in Wikipedia. Note that those weights are optimized with a development data set. – Blog feeds are ranked according to the total scores for all the blog posts ranked above, where the total score for each blog post is calculated as above, in terms of the entry title and the related terms (including synonymous terms).

504

4.2

Y. Abe et al.

Evaluation Results

This section shows the evaluation results of ranking expert bloggers in crime domain. Out of the sample topics shown in Figure 2, for “auction fraud” and “phishing” (for both English and Japanese), we manually examined top ranked 10 blog feeds. As shown in Table 3, for each topic and for each language, out of the top ranked 10 blog feeds, most are those of expert bloggers in crime domain. Here, as we introduced in section 1, when judging whether a blogger is an expert in crime domain, we employ the criterion studied in the blog distillation task. For the specific topics above, we have not quantitatively compared this result with that of the original rankings returned by search engine APIs. However, we have already reported in [10] that, with our general framework of expert blog feeds retrieval, we achieved improvement over the original rankings returned by “Yahoo!” API and “Yahoo! Japan” API. We then manually categorize those retrieved blog feeds into the following four types: (1) the blogger is a victim or one who personally knows a victim, (2) the blogger warns others with reference to news posts on criminal acts or to some other official sites on the Web, (3) the blogger introduces tips on how to prevent criminal acts, (4) the blogger is closely related to the given query topic, although none of the above three, such as only stating blogger’s own opinion. Here, for (1), we distinguish the two cases: the blogger himself/herself is a victim, or, the Table 3. Evaluation Results of Extracting Expert Bloggers on “Fraud” (English / Japanese) ratio of relevant blog feeds out of top ranked 10 feeds (%) Topic Auction fraud 90 / 90 Phishing 100 / 90 Table 4. Results of Manually Categorizing the Results of Top Ranked 10 Expert Bloggers on “Fraud” (%) (1) closely (2) referring related to to official victims sites blogger (3) introducing blog of personally other Topic news (4) rest prevention a victim knows Web sites tips a victim (a) English Auction fraud 0 0 30 40 70 0 Phishing 0 0 50 60 90 0 (b) Japanese Auction fraud 20 10 20 10 30 10 Phishing 0 0 40 30 70 0


505

blogger personally knows a victim. For (2), we distinguish the two cases: the blogger’s post is with reference to news posts, or, with reference to some other official sites on the Web. The results are shown in Table 4. It is important to note that we can collect many blog posts which refer to official sites such as news sites or which introduce prevention tips.

5

Extracting Reports on Crimes by Identifying Victims’ Blogs

5.1

Detecting Dependency Relations and Bunsetsu Patterns in Victims’ Blogs in Japanese

This section describes the task of extracting reports on being victimized from blogs. In this second task, we propose a technique which is based on detecting linguistic expressions representing experiences of being a victim of certain fraud, such as “being a victim of ” and “deceived”. In this paper, we give the detail of the technique as well as the evaluation results only for Japanese. However, its fundamental methodology can be applicable to any other language by manually examining expressions representing experiences of being a victim of certain fraud. Table 5. Expressions for Detecting Experiences of being a Victim of “Fraud” Type of expressions dependency relation of two bunsetsus single bunsetsu

example weight # of expressions sagi - au 19 (base form) (“be victimized”), + 10 sagi - hikkakaru 84 (conjugated form) (“be scammed”) damasa-reta (“be deceived”) 2 13 higai-todoke (“offense report”) 1 113 keisatsu (“police”) 0.5 17

As shown in Table 5, expressions representing experiences of being a victim of certain fraud can be roughly decomposed into two classes: dependency relations of two bunsetsus and a single bunsetsu5 . A dependency relation of two bunsetsus can be regarded as much stronger evidence of representing experiences of being a victim than a single bunsetsu. Each expression is then assigned a certain weight, where those weights are considered when measuring the score of each blog post. We give each dependency relation of two bunsetsus a weight of 10, while we 5

In Japanese, a bunsetsu corresponds to a phrase in English such as a subject phrase, an object phrase, and a verb phrase. A bunsetsu consists of at least one content word and zero or more functional words. In this paper, we use KNP (http://www-lab25.kuee.kyoto-u.ac.jp/nl-resource/knp-e.html) as a tool for bunsetsu analysis and dependency analysis of Japanese sentences.

506

Y. Abe et al. (a) “auction fraud”

(b) “phishing”

Fig. 3. Evaluation Results of Extracting Reports on Crimes by Identifying Victims’ Blogs

give each single bunsetsu one of the weights 2, 1, and 0.5 based on intuition and empirical tuning. Finally, blog posts are ranked according to the score: weight(type(e)) e∈S weight(type(e)) + # of words in the blog post e∈D

where D is the set of dependency relations of two bunsetsus, S is the set of a single bunsetsu, and e is an expression in D or S. Since single bunsetsus are much weaker evidence and are frequently detected even in posts from bloggers who are not victims. Thus, we normalize the sum of the scores of single bunsetsus in terms of the length of each blog post. For the topics “auction fraud” and “phishing”, we collect 20 and 3 blog posts for training, respectively, and manually extracted 103 dependency relations of bunsetsus as well as 143 single bunsetsus in total (as shown in Table 5).


507

Fig. 4. Topics, Terms in Blogs, and Summaries of Blog Posts: Examples

5.2

Evaluation Results

In order to evaluate the technique presented in the previous section, we first collect 403 test blog posts for the topic “auction fraud” and 466 for the topic “phishing”6 . Those test blog posts are then ranked according to the score presented in the previous section. We manually examined the top ranked 50 blog posts whether the blogger of the post is actually a victim of the certain fraud and plot the changes in precision as shown in Figure 3. Here, as the baseline, we simply show the original rankings returned by “Yahoo! Japan” API with the same search query “topic name t AND higai (victimized)”. 6

We use the Japanese search engine “Yahoo! Japan” API (http://www.yahoo.co. jp/). Blog hosts are limited to 8 (FC2.com,yahoo.co.jp,ameblo.jp,goo.ne.jp, livedoor.jp,Seesaa.net,yaplog.jp,hatena.ne.jp). As the search query, we employ “topic name t AND higai (victimized)”.

508

Y. Abe et al.

As can be clearly seen from these results, the proposed technique drastically outperforms the baseline. For the topic “phishing”, especially, the proposed technique detects just a small number of victims’ blogs, none of which can be ranked high by the baseline. The difference in the performance between the two topics “auction fraud” and “phishing” is estimated to be mainly because the number of bloggers who are actually victims of “phishing” is much less than that for “auction fraud” in the Japanese blogosphere. Furthermore, with the proposed technique, sometimes it can happen that blog posts containing a dependency relation assumed to be an evidence of a victim’s blog are ranked high, even though the blogger is not a victim. We observed that such over-detection occurs when those dependency relations are embedded in adnominal clauses, and the bloggers are actually experts who introduce prevention tips. Both for the first task (presented in section 4) and the second task (presented in this section), Figure 4 shows sample summaries of retrieved blog feeds/posts which are categorized into (1) closely related to victims, (2) referring to official sites, and (3) introducing prevention tips, as in section 4.2. In the figure, samples of related terms extracted from Wikipedia entries (those shown in Figure 2) are marked. Characteristic terms included in the blog posts are also marked.

6

Conclusion

This paper proposed a framework of extracting people’s concerns and reports on crimes in their own blogs. In this framework, we solved two tasks. In the first task, we focused on experts in crime domain and addressed the issue of extracting concerns on crimes such as tips for preventing being victimized. We showed that we successfully ranked expert bloggers high in the results of blog feed retrieval. In the second task, on the other hand, we focused on victims of criminal acts, and addressed the issue of extracting reports on being victimized. We showed that we outperformed blog post ranking based on the search engine API by incorporating dependency relations for identifying victims’ blog posts. Future works include incorporating multilingual sentiment analysis techniques [12,13]. and automatic extraction of reports or experiences of victims of crimes.

References 1. Glance, N., Hurst, M., Tomokiyo, T.: Blogpulse: Automated trend discovery for Weblogs. In: WWW 2004 Workshop on the Weblogging Ecosystem: Aggregation, Analysis and Dynamics (2004) 2. Nanno, T., Fujiki, T., Suzuki, Y., Okumura, M.: Automatically collecting, monitoring, and mining Japanese weblogs. In: WWW Alt. 2004: Proc. 13th WWW Conf. Alternate Track Papers & Posters, pp. 320–321 (2004) 3. Macdonald, C., Ounis, I., Soboroff, I.: Overview of the TREC-2007 Blog Track. In: Proc. TREC-2007 (Notebook), pp. 31–43 (2007) 4. Yangarber, R., Best, C., von Etter, P., Fuart, F., Horby, D., Steinberger, R.: Combining Information about Epidemic Threats from Multiple Sources. In: Proc. Workshop: Multi-source, Multilingual Information Extraction and Summarization, pp. 41–48 (2007)


509

5. Pouliquen, B., Steinberger, R., Belyaeva, J.: Multilingual Multi-document Continuously-updated Social Networks. In: Proc. Workshop: Multi-source, Multilingual Information Extraction and Summarization, pp. 25–32 (2007) 6. Yoshioka, M.: IR Interface for Contrasting Multiple News Sites. In: Prof. 4th AIRS, pp. 516–521 (2008) 7. Bautin, M., Vijayarenu, L., Skiena, S.: International Sentiment Analysis for News and Blogs. In: Proc. ICWSM, pp.19–26 (2008) 8. De Saeger, S., Torisawa, K., Kazama, J.: Looking for Trouble. In: Proc. 22nd COLING, pp. 185–192 (2008) 9. Torisawa, K., De Saeger, S., Kakizawa, Y., Kazama, J., Murata, M., Noguchi, D., Sumida, A.: TORISHIKI-KAI, an Autogenerated Web Search Directory. In: Proc. 2nd ISUC, pp.179–186 (2008) 10. Kawaba, M., Nakasaki, H., Utsuro, T., Fukuhara, T.: Cross-Lingual Blog Analysis based on Multilingual Blog Distillation from Multilingual Wikipedia Entries. In: Proc. ICWSM, pp. 200–201 (2008) 11. Nakasaki, H., Kawaba, M., Yamazaki, S., Utsuro, T., Fukuhara, T.: Visualizing Cross-Lingual/Cross-Cultural Differences in Concerns in Multilingual Blogs. In: Proc. ICWSM, pp. 270–273 (2009) 12. Evans, D.K., Ku, L.W., Seki, Y., Chen, H.H., Kando, N.: Opinion Analysis across Languages: An Overview of and Observations from the NTCIR6 Opinion Analysis Pilot Task. In: Proc. 3rd Inter. Cross-Language Information Processing Workshop (CLIP 2007), pp. 456–463 (2007) 13. Wiebe, J., Wilson, T., Cardie, C.: Annotating Expressions of Opinions and Emotions in Language. Language Resources and Evaluation 39, 165–210 (2005)

Automatically Extracting Web Data Records Dheerendranath Mundluru, Vijay V. Raghavan, and Zonghuan Wu IMshopping Inc., Santa Clara, USA University of Louisiana at Lafayette, Lafayette, USA Huawei Technologies Corp., Santa Clara, USA [email protected], [email protected], [email protected]

Abstract. It is essential for Web applications such as e-commerce portals to enrich their existing content offerings by aggregating relevant structured data (e.g., product reviews) from external Web resources. To meet this goal, in this paper, we present an algorithm for automatically extracting data records from Web pages. The algorithm uses a robust string matching technique for accurately identifying the records in the Webpage. Our experiments on diverse datasets (including datasets from third-party research projects) show that the proposed algorithm is highly effective and performs considerably better than two other state-of-the-art automatic data extraction systems. We made the proposed system publicly accessible in order for the readers to evaluate it. Keywords: Structured data extraction, Web content mining.

1 Introduction It is often important for applications such as e-commerce portals and local search engines to enrich their existing content by aggregating relevant data displayed on external Websites. Such data to be aggregated is usually displayed as regularly structured data records. Fig. 1, for example, shows a sample Web page from USPS.com that displays a list of post office locations as regularly structured records (in dashed boxes). Each record displays attributes such as address, phone, and business hours. The records are displayed in one particular region of the Web page and are also formatted regularly i.e., HTML tags that makeup these records are similar in syntax thus making similar attributes to be displayed in similar positions. Though some large Websites expose their data through APIs, data from most other sources will have to be extracted programmatically. Extracting structured data programmatically is a challenging task as [1]: (a) HTML tags only convey presentation details rather than the meaning of the Web page content, (b) different Web sources display information in different formats, (c) Web pages also include other content such as navigation links and sponsored results that need to be filtered out, (d) attributes present in one record may be missing from another record in the same Web page, (e) few attributes may have multiple values and the number of such values for an attribute may also vary across records in the same Web page. A. An et al. (Eds.): AMT 2010, LNCS 6335, pp. 510–521, 2010. © Springer-Verlag Berlin Heidelberg 2010

Automatically Extracting Web Data Records

511

Fig. 1. Webpage displaying structured data

The area of research that specifically deals with extracting structured data from Web pages has been very well studied and is referred to as structured data extraction [1]. Structured data extraction algorithms are based on the assumption that structured data is rendered regularly usually in the form of records as shown in Fig. 1. Such algorithms typically build data extraction rules called wrappers for extracting structured data. Wrappers can be constructed manually, semi-automatically, or automatically. Manual approaches are tedious and are not scalable when there are a large number of sources to extract data from. Semi-automated methods are very popular and are referred to as wrapper induction techniques [2]. In such methods, a human first labels the target data to be extracted in a few training Web pages collected from the target resource. A machine learning algorithm then uses these training pages to learn a wrapper. The learned wrapper can then be used to extract target data from similar, unseen Web pages returned by the target resource. A limitation of this approach is the time involved in labeling the training pages, which may be performed only by trained users. Scalability can still be an issue if there are thousands of sources to handle. Nevertheless, based on our experience, wrapper induction techniques are still very effective as they can build robust wrappers with reasonable effort [3]. Finally, automatic methods can avoid the limitations of wrapper induction methods by constructing wrappers and extracting structured data using completely automated heuristic techniques [4][5][7][9]. These methods use a pattern discovery algorithm in combination with other heuristics such as HTML tags, visual cues and regular expressions for building wrappers and subsequently extracting target data. Though current automated approaches are very effective, we believe that still more significant improvements are needed for them to achieve near perfect accuracy when dealing with large number of

512

D. Mundluru, V.V. Raghavan, and Z. Wu

data sources. However, using such algorithms as a component in wrapper induction systems can provide the benefit of reducing the labeling time [3]. In this paper, we propose an algorithm called Path-based Information Extractor (PIE) for automatically extracting regularly structured data records from Web pages. PIE uses a robust approximate tree matching technique for accurately identifying the target records present in a Webpage. PIE filters out content present in other nonrelevant sections such as navigation links and sponsored results. Due to space constraints we only present the algorithm that identifies target records in a Web page and do not discuss the component that generates wrappers for subsequent fast extraction of data. Though the discussed algorithm is a prerequisite for constructing wrappers, it may still be used independently for data extraction tasks. The rest of the paper is organized as follows: In section 2, we will briefly discuss the related work. In section 3, we discuss the proposed algorithm in detail. In Section 4, we will present our experiment results. Finally, we conclude in section 5.

2 Related Work Automatic structured data extraction systems have been well studied in the past several years [4][5][7][9]. Of all the automatic extraction systems, MDR is most similar to ours [4]. It uses an approximate string matching algorithm for discovering records. However, as we will show in section 4, MDR performed poorly on all the datasets that we used in our experiments. It was not effective when the degree of regularity across the records was not very high. MDR was also not effective when the Web page contains several regions (e.g., advertisements) in addition to target region. It also makes a strong assumption about the regularity in the structure of records in a region. Specifically, MDR views each Web page as a DOM tree (referred to as tag-tree) and defines a region as two or more generalized nodes having the same parent node. A generalized node is in turn a combination of one or more adjacent nodes in the tree. The authors, in [4], make the strong assumption that all the generalized nodes, which collectively comprise a region have the same length i.e., each of them is composed of the same number of adjacent nodes. However, in many Web pages, generalized nodes forming a region may not have the same length. As will be discussed later, we address this issue by relaxing the definition for regularity in the structure of records. Through this work, we propose several important improvements over MDR. In section 4, we also compare PIE to another automatic data extraction system called ViNTs [5]. ViNTs relies on both visual cues (as appearing in a Web browser) and HTML tag analysis in identifying records and building wrappers. For example, it uses a visual cue-based concept called content line for identifying candidate records. A content line is defined as a group of characters that visually form a horizontal line in the same region on the rendered page. A blank line is an example of a content line. Though ViNTs is very effective on result pages of search engines, we found in our experiments that its accuracy reduces significantly when handling Web pages in vertical portals such as content review sites.


513

3 Proposed Algorithm 3.1 Observations and Overview PIE is based on the following three important observations. The first two observations have been used by most prior research projects and we present them here from [4]. Observation 3 can be considered as an extension to Observation 1. Observation 1: “A group of data records that contains descriptions of a set of similar objects are typically presented in a contiguous region of a page and are formatted using similar HTML tags. Such a region is called a data record region.” Observation 2: “A group of similar data records being placed in a specific region is reflected in the tag-tree by the fact that they are under one parent node.” Observation 3: If a region consists of n records displayed contiguously, then the left most paths of the sub-trees corresponding to all n records under the parent are identical. First observation is straightforward and has been discussed through Fig. 1 in section 1. Since HTML source of every Web page is inherently a tree, according to Observation 2, records displayed in a region will also be present under the same parent in the tree. For example, the tag-tree in Fig. 5 displays five records R1-R5 (in dashed boxes), which are present under the same parent P. Similarly, as specified in Observation 3, the left most paths of the sub-trees of these records are also identical.

Fig. 2. Three-steps in the PIE algorithm

As depicted in Fig. 2, the PIE algorithm involves three steps for extracting records. The input to the algorithm is a Web page and the output is the target records extracted from the Web page. The first step (Parent Discoverer) is based on the second observation. It identifies a set of candidate parent nodes present in the tag-tree of the input Web page. One of the identified candidate parent nodes is expected to be the parent node of the target records. The second step (Record Discoverer) takes candidate parent nodes as the input and outputs a list of candidate regions along with the records

514


discovered in those regions. The Record Discoverer is based on the first and third observations. The third and final step (Target Region Selector) takes the different candidate regions and their records as input and identifies exactly one region as the target region. Records present in this region form the final output. A constraint for the record discovery process is that the input Web page should have at least K (2 in our case) target records. 3.2 Three-Step Record Discovery Process In this section, we discuss in detail the three steps in the PIE algorithm. Parent Discoverer. Parent Discoverer is based on Observation 2. It takes the input Web page and outputs a list of candidate parent nodes. One of the candidate parent nodes is expected to contain the target records. Parent Discoverer first builds a tagtree of the input Web page. An example tag-tree is shown in Fig. 3. The root of the tag-tree is and each tag node can be located by following a path from the root to the node. The system finds the candidate parent nodes by analyzing such paths. Two types of paths, called Relaxed Path (rpath) and Indexed Path (ipath), are used for very effectively identifying the candidate parent nodes. rpath and ipath are defined as follows. Definition 1 (Relaxed Path): If n1, n2… np-1, np are p tag nodes in a tag-tree T containing N tag nodes (p ≤ N), where n1 is the root of T and also parent of n2, n2 is parent of n3 and so on, then the relaxed path of np is defined as rpath(np) = n1.n2…np-1.np Definition 2 (Indexed Path): If n1, n2, n3 are 3 tag nodes in a tag-tree T containing N tag nodes (3 ≤ N), where n1 is the root of T, n2 is the ith immediate child of n1 and n3 is the jth immediate child of n2, then the indexed path of n3 is defined as ipath(n3) = n1.n2[i].n3[j] Fig. 4 shows the algorithm for discovering candidate parent nodes while the tagtree in Fig. 3 is used to illustrate the algorithm. Some of the nodes in the tree have been indexed for clarity. Let sub-trees of UL0, TABLE0 and TABLE1 in Fig. 3 represent three different regions in the Web page. After constructing a tag-tree of the input Web page, the algorithm constructs all unique rpaths and ipaths that lead to leaf nodes with textual content (line 2). This generates two unique rpaths and seven ipaths: rpath1: HTML.BODY.UL.LI.A rpath2: HTML.BODY.TABLE.TR.TD ipath1: HTML.BODY.UL0.LI0.A0 ipath2: HTML.BODY.UL0.LI1.A0 ipath3: HTML.BODY.TABLE0.TR0.TD0 ipath4: HTML.BODY.TABLE0.TR1.TD0 ipath5: HTML.BODY.TABLE0.TR2.TD0 ipath6: HTML.BODY.TABLE1.TR0.TD0 ipath7: HTML.BODY.TABLE1.TR1.TD0 In line 3, we map ipaths to rpaths, i.e., we group all ipaths having the same rpath. In our example, ipath3-ipath7 are mapped to rpath2 as their leaf nodes have the same rpath HTML.BODY.TABLE.TR.TD. Similarly, ipath1-ipath2 are mapped to rpath1. Assumption here is that due to regularity in the rendered structured data, rpaths of similar attributes in different records of a region are identical, but their ipaths differ


515

due to presence of index numbers. Assuming that each text leaf node corresponds to an attribute in a record, by grouping ipaths of these attributes with identical rpaths, we aim to conceptually group the records in a region. In line 4, we discard rpaths with less than K associated ipaths as the constraint to the algorithm is for the Web page to have at least K records. For each remaining rpath, we perform a pair-wise comparison of all its associated ipaths to generate a longest common prefix from each comparison (line 6). In our example, for rpath1, the longest common prefix generated by comparing ipath1 and ipath2 is HTML.BODY.UL0. Similarly, the longest common prefixes generated for rpath2 are: HTML.BODY, HTML.BODY.TABLE0 and HTML.BODY.TABLE1. A map of each longest common prefix and the list of ipaths that lead to its generation is maintained (line 7). In line 8, we discard longest common prefixes generated by less than K ipaths. Finally, trailing tags of the remaining longest common prefixes are returned as candidate parents.

Fig. 3. Sample tag-tree with three regions

Procedure: parentDiscoverer(Web page P) 1: construct tag-tree T of P 2: find all unique rpaths & ipaths in T leading to leaf nodes with text 3: map ipaths to rpaths 4: discard rpaths with less than K associated ipaths 5: for each rpath r do 6: find longest-common-prefixes by doing pair-wise comparison of all ipaths of r 7: M = map longest-common-prefix to ipath-list 8: discard longest-common-prefixes generated by less than K ipaths 9: return trailing tag of longest-common-prefixes Fig. 4. Parent discovery process

Record discoverer. Record discoverer takes candidate parent nodes as input and outputs a list of candidate regions along with records discovered in those regions. Sometimes, a parent may have more than one region in its sub-tree. For Web pages that display only one record in each row (e.g., Fig. 1), we define a record as follows. Definition 3 (Record): A record in a region is represented wholly by one or more immediate contiguous child nodes (of a parent) and their corresponding sub-trees.

516


Record discoverer is based on Observations 1 and 3. Due to Observation 3, the goal is to discover identical left most paths of sub-trees corresponding to records under the parent. Such left most paths are nothing but rpaths with parent node as their first node in the rpath. Once such paths are identified for a region, extracting records from that region is trivial as a record exists between two successive paths. We refer to such paths as Record Identifying Paths (RIP) as they can separately identify the records in a region. A robust string matching algorithm based on edit distance [6] is used in PIE to identify such RIPs. Fig. 5 is used to illustrate the record discovery process. It displays a tag-tree with parent P and its children labeled 1-8. For clarity, numbers are used instead of HTML tag names. The tag-tree has eleven candidate RIPs and five records R1-R5 that are to be discovered. A candidate RIP is the left most path of every sub-tree under P.

Fig. 5. Record discoverer illustration

We begin our search with the first candidate RIP P.1.2, left most path under P. We check if it appears at least K times as the algorithm requires the input page to have at least K records i.e., if there are K records, then we also have at least K identical RIPs. Since P.1.2 appears less than 2 times (since K=2), we discard it and consider the next candidate RIP P.3.4.5. We consider P.3.4.5 for further processing as it appears 5 times. P.3.4.5 appearing in the last sub-tree under P is not considered as a candidate as it is not the left most path in its sub-tree. We next construct a tag-string using the tags between the first and second appearances of P.3.4.5 (shown in dashed arrows). If this tag-string is represented as an ordered pair (1,2), then (1,2) would be P.3.4.5.6.7. Tags appearing in the second appearance of P.3.4.5 are excluded in the tag-string. Since goal is to identify contiguous records, we construct another tag-string using the tags between second and third appearances of P.3.4.5, which is represented using (2,3). The pair (2,3) corresponds to the tag-string P.3.4.5.6.7. Since from Observation 1, contiguous records in a region are formatted using similar HTML tags, we use edit distance (ED) to capture the similarity between the two tag-strings by computing ED((1,2),(2,3)). Here, ED is defined as the smallest number of operations (insertion, deletion and substitution of tags) needed to convert one tag-string into another. If ED is less than or equal to a certain threshold, then the two tag-strings are considered similar and thus correspond to two contiguous records. ED threshold is calculated automatically depending on the


517

characteristics of the sub-tree under the parent [1]. Specifically, if the different instances of the candidate RIP appear at regular positions under the parent, then a small value (1 in our case) is used as ED threshold. Otherwise, a higher value (7 in our case) is used as threshold. We found that using a static threshold for all Web pages, like in MDR, reduces the extraction accuracy. Therefore, setting the ED threshold based on sub-tree characteristics under the parent gives us more flexibility. For the current example, let’s assume an ED threshold of 7. Since ED((1,2),(2,3)) is equal to 0, we consider the corresponding records as being contiguous. We save the left ordered pair (1,2) as the first discovered record. We next compute ED((2,3),(3,4)), which is equal to 1 as the ordered pair (3,4) had one extra node 8. Since the newly computed ED is less than ED threshold, we save (2,3) as second record. We next save (3,4) as ED((3,4),(4,5)) is equal to 1. At this point, since we do not have any more appearances of P.3.4.5, we extract the tag-string starting at the last instance of P.3.4.5. Choosing the ending path to construct this last tag-string is a characteristic of the number of paths between the previous two appearances of P.3.4.5. Since there is only one path (P.6.7) between (4,5), there should also be only one path between last appearance of the candidate RIP P.3.4.5 and the ending path. If ending path happens to be part of a sub-tree ‘s’ that is different from the sub-tree of the starting path (P.3.4.5), then we use the left most path in ‘s’ as the ending path. But, if the ending path falls outside the sub-tree under P, then we set the right most path under P as the ending path. In our case, the ending path (P.3.4.5) is part of a different sub-tree and since it is not the left most path in its sub-tree, we set the ending path as P.3.7. Due to this the last tag-string will be P.3.4.5. ED between (4,5) and the last tag-string is 2. Therefore, both (4,5) and the last tag-string are saved as newly discovered records. Since the number of records discovered so far is greater than or equal to K, we consider that a region has been found. It can be seen that all saved ordered pairs reflect the target records R1 through R5. If the number of records discovered was less than K, we would have discarded all the records discovered so far and would have repeated the process with the next immediate candidate RIP, P.6.7. Similarly, if there were more candidate RIPs appearing after the discovered region, then we would continue with the above process searching for new regions. Virtual-tree Based Edit Distance (VED). The above record discovery process would be effective on Web pages whose records display high regularity in their tag structures. It would however be ineffective when records display less regularity. For example, it failed on the Web page displayed in Fig. 1 as the tag structures of the records were very dissimilar due to the absence of “Business Hours” attribute (and its sub-tree) from the second record. To handle such complex cases, we designed a robust string matching function called, the virtual-tree based edit distance. VED considers the tree structure of one of the tag-strings to more effectively identify the similarity between the two tag-strings. After integrating VED, if ED of two tagstrings is greater than threshold, then we invoke VED with the tree structures of the two tag-strings. VED returns true if it considers the two input tree structures as contiguous records. Otherwise, it returns false. VED algorithm is given in Fig. 6. Input to VED includes the tree structures corresponding to the two tag-strings along with the threshold (same as ED threshold). The larger of the two trees (in terms of the number of tag nodes) will be big-tree while the other will be small-tree. The algorithm traverses through each node of the big-tree and at each node it creates a

518


new virtual tree (line 5) from the big-tree by deleting the node and its sub-tree. The tree traversal is done until either a newly created virtual tree is very similar to the small-tree (lines 6-7) or all the nodes in the big-tree are traversed. Tree traversal is done from left-most path to right-most path (lines 1 and 9) and within each path we traverse from leaf node upwards (lines 3 and 8) until we reach a node that was traversed earlier or until we reach the root (line 4). If n, m represent the total number of tags in big-tree and small-tree, then complexity of VED is O(n2m). Though the complexity of the algorithm has increased, it should be noted that usually record discovery process is only performed as part of wrapper creation process where some delay in extraction is not an issue. Procedure: VED(big-tree, small-tree, threshold) 1: curr-path = getNextPath(big-tree) 2: while curr-path != NULL do 3: curr-node = leaf(curr-path) 4: while curr-node != root of big-tree && curr-node NOT YET traversed do 5: virtual-tree = createNewTree(big-tree,curr-node) 6: if ED(virtual-tree, small-tree) EDthreshold and VED returns false), then instead of stopping the record discovery process for the current candidate RIP, we still try other combinations for the pair (i,j). Fig. 7 shows how different combinations, for a particular RIP, are tried when ED-VED of a particular combination results in a success S (ED ≤ ED-threshold or VED returns true) or a failure F (ED > ED-threshold and VED returns false). This combination generation makes record discovery process even more effective.


519

Target region selector. Target region selector takes the input candidate regions and their records and selects only one of them as the target region whose records form the final output. It first selects three best candidate regions from all input candidate regions and subjects them to additional processing to select the final target region. Total number of characters appearing in all the records in each candidate region is used as a heuristic to select the three best candidate regions. Usually the target region has the highest number of characters compared to all other regions in a Web page though sometimes there may be cases where other regions might have more characters. Moreover, sometimes record discoverer might incorrectly construct a region which might have more characters than the target region. Four HTML features are further used in selecting the final target region from the three selected candidate regions. The features used are: (a) anchor tag, (b) image tag, (c) bold text and (d) presence of any text outside anchor tags. The system looks for these features in each record of the region. If a feature appears in at least half of the records in a region, then we consider it as a region feature. Among the three candidate regions, the candidate region that has the highest number of region features is selected as the target region. If there is more than one candidate region with the same number of region features, then the region having the maximum number of characters is selected as the target region. The motivation behind using feature count is that usually the target region has the maximum number of the above features in comparison to all other regions.

4 Experiments We conducted experiments on three datasets to evaluate PIE and compare it with MDR and ViNTs [5][6]. The first two datasets were randomly collected by us from the Web while the third was taken from third-party data extraction research projects. As MDR returns all the regions in a Web page, its performance on only the target region is considered for evaluation. The PIE prototype system and the datasets used in this experiment are publicly accessible [10]. Evaluation Measures. Recall and precision defined below are used as the evaluation measures [5]: recall = Ec/Nt and precision = Ec/Et where Ec is the total number of correctly extracted target records from all Web pages in the dataset, Nt is the total number of target records that appear in all Web pages in the dataset, Et is the total number of target records extracted. Et includes target records that are both correctly and incorrectly extracted. An incorrect extraction involves partial extraction of a record, merging one or more records into a single record and extracting content that appears outside the target region. Experiment on Dataset 1. The first dataset includes 60 Web pages taken from 60 Web sources that mostly include general and special purpose search engines. Each source was given one query and the returned result page was included in our dataset. There were a total of 879 target records across all 60 Web pages. Results of the experiment on this dataset are summarized in Table 1. Detailed results of the three systems on all the Web pages used in this and the remaining two datasets are available at [10]. As we can see from Table 1, the performance of PIE is significantly better than MDR and marginally better than ViNTs. Most of the Web pages on which MDR

520

D. Mundluru, V.V. Raghavan, and Z. Wu Table 1. Summary of experiment on dataset 1

PIE ViNTs MDR

#Correct 810 748 592

#Incorrect 45 54 176

Precision 94.73% 93% 77.08%

Recall 92.15% 85.09% 67.34%

failed completely (extracted zero records) were result pages of general-purpose search engines that displayed multiple regions. Interestingly, MDR was either failing completely or was perfectly extracting all the records in the Web page. Experiment on Dataset 2. While dataset 1 included mostly search engine result pages (normally seen in metasearch domain), dataset 2 includes 45 Web pages taken randomly from review sites and local search engines. Compared to dataset 1, Web pages in this dataset usually have distinct characteristics such as displaying records with varying lengths and displaying several non-target regions. Using such diverse datasets has allowed us to more effectively evaluate the three systems. Results on this dataset that has a total of 652 records are summarized in Table 2. As we can see PIE significantly outperformed both ViNTs and MDR. Unlike ViNTs, PIE performed consistently across both datasets 1 and 2 thus showing that it is effective in handling diverse Web pages. ViNTs and MDR performed almost equally. ViNTs precision was affected mainly as it wrongly extracted 889 records from one Web page. Table 2. Summary of experiment on dataset 2

PIE ViNTs MDR

#Correct 607 469 463


Precision 96.65% 31.35% 73.14%

Recall 93.09% 71.93% 71.01%

Experiment on Dataset 3. Dataset 3 includes a total of 58 Web pages taken from several prior third-party data extraction projects. This dataset was chosen to avoid any bias in our experiments. Like dataset 1, Web pages in this dataset are also mostly search engine result pages. Of the 58 Web pages, 40 were taken from Omini [7], 10 from RISE [8], and 8 from RoadRunner [9]. Results on this dataset that had a total of 1623 records are summarized in Table 3. As we can see, PIE once again outperformed MDR. In terms of precision, PIE performed considerably better than ViNTs. Though PIE had a higher recall than ViNTs, they both performed almost equally. The difference in recall was mainly due to failure of ViNTs on one particular Web page that had large number of records. Table 3. Summary of experiment on dataset 3

PIE ViNTs MDR

#Correct 1467 1338 928


Precision 95.57% 80.94% 85.21%

Recall 90.38% 82.43% 57.17%


521

5 Conclusion and Future Work In this paper, we proposed an algorithm for automatically extracting data records from Web pages. The algorithm is based on three important observations about the regularity in displaying records and uses a robust string matching technique to accurately identify the records. Specifically, we propose an approximate tree matching algorithm to effectively handle Web pages containing records with greater structural variation. Experiments on diverse datasets showed that the proposed system, PIE, is highly effective and performed considerably better than two other state-of-the-art automatic record extraction systems. In future, we would like to extend this work by: (1) further improving the algorithm to more effectively and efficiently extract records, (2) automatically extracting attributes from each record.

References [1] Mundluru, D.: Automatically Constructing Wrappers for Effective and Efficient Web Information Extraction. PhD thesis, University of Louisiana at Lafayette (2008) [2] Muslea, I., Minton, S., Knoblock, C.: A Hierarchical Approach to Wrapper. In: Proceedings of the 3rd International Conference on Autonomous Agents, Seattle, pp. 190–197 (1999) [3] Mundluru, D., Xia, S.: Experiences in Crawling Deep Web in the Context of Local Search. In: Proceedings of the 5th Workshop on Geographical Information Retrieval, Napa Valley (2008) [4] Liu, B., Grossman, R., Zhai, Y.: Mining Data Records in Web Pages. In: Proceedings of the ACM International Conference on Knowledge Discovery & Data Mining, Washington, D.C, pp. 601–606 (2003) [5] Zhao, H., Meng, W., Wu, Z., Raghavan, V., Yu, C.: Fully Automatic Wrapper Generation for Search Engines. In: Proceedings of the 14th International World Wide Web Conference, Chiba, pp. 66–75 (2005), http://www.data.binghamton.edu:8080/vints/ [6] Hall, P., Dowling, G.: Approximate String Matching. ACM Computing Surveys 12(4), 381–402 (1980) [7] Buttler, D., Liu, L., Pu, C.: A Fully Automated Extraction System for the World Wide Web. In: Proceedings of the International Conference on Distributed Computing Systems, Phoenix, pp. 361–370 (2001) [8] ISE. A Repository of Online Information Sources Used in Information Extraction Tasks, University of Southern California (1998), http://www.isi.edu/info-agents/ RISE/index.html [9] Crescenzi, V., Mecca, G., Merialdo, P.: RoadRunner: Towards Automatic Data Extraction from Large Web Sites. In: Proceedings of the 27th International Conference on Very Large Data Bases, Rome, pp. 109–118 (2001) [10] PIE Demo System, http://www.fatneuron.com/pie/

Web User Browse Behavior Characteristic Analysis Based on a BC Tree Dingrong Yuan1,2 and Shichao Zhang2 1

The International WIC Institute, Beijing University of Technology Beijing 100022, China 2 College of Computer Science, Guangxi Normal University Guilin, 541004 China {dryuan,Zhangsc}@mailbox.gxnu.edu.cn

Abstract. Analysis of Web user browser behavior characteristics is a key technology in the domains, such as Initiative Web Information Retrieval and Information Recommendation. Taking into account the layout of Web pages, in this paper we constructed a user browse behavior characteristic (BC) tree based on the browsing history of Web users, and then established a new approach for analyzing Web user BC trees. This method delivers us the interesting topics of a user, the extent and depth of these topics, as well as the explanation of the frequency of accessing hierarchic block paths on web pages. We illustrated the efficiency with experiments, and demonstrated that the proposed approach is promising in Initiative Web Information Retrieval and Information Recommendation.

1

Introduction

Behavior characteristic analysis of user browsing is an important topic in Web intelligence [12,11]. For example, merchants recommend products to clients according to their purchasing behavior habits [2,5,6]. Recommendation system provides information according to user browsing behavior [7]. Web intelligence behavior has been studied well with brain informatics [4,13]. Enterprise and companies identify potential clients by user accessing behavior patterns [14]. Consequently, understanding the behavior characteristics of user is crucial in many areas, such as document pre-sending, information recommendation system, and potential clients recognizing. From existent techniques, initiatively retrieving the information required by users, called as initiative information retrieval, has become a hopeful and hot research topic in information retrieval system. Initiative information retrieval is essentially different from traditional information retrieval because such a retrieval system initiatively forms the multidimensional retrieval conditions according to behavior characteristics of a user. However, traditional retrieval systems acquire the required characteristics dependent on session information which is included in user logs located on a server. These characteristics actually tend to detract from generality, or specified only in a special site. In particular, the session information is unstructured. This means, it is often difficult in real applications. A. An et al. (Eds.): AMT 2010, LNCS 6335, pp. 522–529, 2010. c Springer-Verlag Berlin Heidelberg 2010

Web User Browse Behavior Characteristic Analysis Based on a BC Tree

523

This paper is focused on the issue of initiative information retrieval. The needed hierarchical information is extracted from the layout of a Web page, so as to construct a BC (behavior characteristic) tree. To identify potentially useful information from the BC tree, a new approach is designed for analyzing Web user behavior. The rest of this paper is organized as follows: Section 2 briefly recalls main related work. Section 3 describes the construction of a behavior characteristic tree based on the browsing history of Web users. Section 4 provides methods for analyzing browsing behavior and acquiring behavior characteristics of web users. Section 5 evaluates the efficiency with experiments. Finally this paper is concluded in Section 6.

2

Related Work

Related work includes browse behavior modeling and analyzing, Web page analyzing and hierarchical information extracting, and Web user access behavior pattern discovering. Behavior is a representation of intelligence, recorded by data. Extracting out useful pattern or knowledge from this kind of behavior data needs to use data mining technique. However, most existing data mining algorithms and tools only stop at discovered models, which require human experts to do some post-process. Yang and Cao have upgraded data mining to behavior analysis [1,9]. They found a basic frame of behavior analysis, constructed a behavior model including plan, object and action, and analyzed the behavior model to obtain the direction of next behavior. Yang suggested customers to change from an undesired status to a desired one to maximize the objective function [8]. Cao developed a technique of exploring frequent impact-oriented activity patterns, impact-contrasted activity sequential pattern and impact-reversed sequential activity pattern [1]. A Web page can be divided into different blocks by its layout. Each block holds a topic. In fact, a block is an information class on one page, and its class name is the topic of the block. Information items on a Web page belong to one block or another, but usually goes with useless information called garbage or junk. Therefore, Web information preprocessing gradually becomes more important than before. For example, Wang et al erected a technique to analyze the layout characteristics of Web pages [7]. Song et al designed an algorithm to extract topics of blocks on a Web page and version comparison etc [6]. All these works are to structure Web page information and to make it more suitable to access and process. Web user browse behavior analysis is to find out behavior characteristics from user browse records. Present techniques mainly mine access patterns from user logs and discover behavior characteristics on the basis of vector space model and Markov model. For example, Zhou et al mine access patterns in the light of EOEM model to discover corresponding potential clients [14]. Zhu et al pre-send a document according to time sequence relevant document requirement and a user session model [15]. Zhao obtained anonymous user browse behavior pattern by session feature space which is generated from the session information [10]. Present techniques of behavior characteristic analysis mainly consider session message, neglect of layout characteristics of information on Web pages.

524

D. Yuan and S. Zhang

Therefore, we will propose a novel technique so as to analyze Web user browse behavior from history records of Web users. The technique makes use of the layout characteristics of information items on a Web page.

3

Constructing a User Browse Characteristic (BC) Tree

There is a hierarchical structure in the layout of Web pages. A Web page can be parsed as a Web tree. Figure 1 gives a sample Web tree, in which there are four blocks in the page M ypage, and block Sports includes three sub-blocks. Some definitions are given as follows:

Fig. 1. A Web page tree

Fig. 2. A BC tree

Definition 1. Provided a user access to information in the block W orldcup, the corresponding path in the Web tree is M ypage → Sports → F ootball → W orldcup → Drawresult. The path is called user-browse-behavior path. Definition 2. Set P− set = {p1 , p2 , . . . , pn }, pi (i = 1, 2, . . . , n) is a user behavior path. A tree is constructed based on pi as follow. The root of the tree is labeled as user. The node is defined as < topic, f requency, link >, topic is the theme of a block, f requency is the visiting count of a block, link is a pointer link to the next node. ∀pi , if there is no prefix in pi and branch of the tree, then pi should be inserted into the tree as a branch, else merge the prefix into the branch of the tree, different parts of pi should be linked as a sub-branch of the prefix. Such a tree is called browse behavior character tree (or BC tree in short). According to the behavior paths as shown in Table 1, A BC tree can be constructed as shown in Fig. 2. Algorithm 1 is used to construct such a BC tree. Table 1. User behavior path table PID

Path

1 a→c→g 2 a→c→m→k

PID

Path

PID

Path

3 c→k 5 c→f →k→t 4 c→f →k→t 6 a→c→g


525

Algorithm 1. Constructing a BC tree Input: a path set P SET Output: a BC tree Procedure: 1. Open a node and label it as BC 2. From 1 to |P SET | 2.1 p = read P SET 2.2 if p is a branch of BC or there is a prefix between p and a branch of BC Then f requency + + for all nodes in the path or prefix of the path. Link last parts of the path into the branch of the prefix. Else link p into as a sub-branch of BC 2.3 read the next path until NULL in P SET 3. Output a BC tree. Definition 3. Set N = {node1 , node2 , . . . , noden }, nodei , i ∈ {1, 2, . . . , m} is a node of BC tree, ∀ε ≥ 0, if Sup(node1 ) =

nodei .f requent support(nodei ) = n ≥ε n support(nodei ) nodei .f requent i=1

i=1

then the nodei is called f requent node. Definition 4. Set T ∈ BC, P ∈ T and P = node1 − > node2 − > . . . − > noden , ∀ε ≥ 0, ∀nodei (i ∈ {1, 2, . . . , m}) is a node of P , if support(nodei ) ≥ ε P is called a f requent path of T . Definition 5. Set tree = {tree1 , tree2 , . . . , treem } is a BC tree, treei is a branch of the tree, hi is height of the treei , di denotes width of the treei . Therefore, 1 hi m i=1 n

h=

1 di m i=1 n

d=

where h denotes the height balance factor, d denotes the width balance factor. ∀ε ≥ 0 ∀i ∈ {1, 2, . . . , m}, if |hi − h| ≤ ε, the tree is called balance tree in height, if |di − d| ≤ ε, the tree is called balance tree in width, if a tree satisfies the two conditions, we call the tree as a balance BC tree. Definition 6. Set tree = {tree1 , tree2 , . . . , treem } is a BC tree, treei is a branch of the tree, hi is height of the treei , di denotes width of the treei , h and d denote height and width balance factors of the tree, respectively. Therefore, 1 (hi − h)2 m i=1 m

Sh =

1 (di − d)2 m i=1 m

Sd =

where Sh denotes the height deflection factor of the tree, Sd denotes the width deflection factor of the tree. ∀ε ≥ 0 ∃i ∈ {1, 2, . . . , m}, if |hi − Sh | ≥ ε, the

526


tree is called the deflection tree in height; if |di − Sd | ≥ ε, the tree is called the deflection tree in width; and if a tree satisfies the two conditions, the tree is called a def lection BC tree. Theorem 1. Set topici is an interesting topic of a user, nodei is a node of T , then topici ⇔ nodei . set T = {topic1 , topic2 , . . . , topicn }, topici (i = 1, 2, . . . , n) is an interesting topic of a user, I = {I1 , I2 , . . . , In }, Ii (i = 1, 2, . . . , n) is an information block on a page. N = {node1 , node2 , . . . , noden }, nodei (i = 1, 2, . . . , n) is a node of T . According to the principle of browsing Web page, ∃Ii = τ (toici ). According to the principle of constructing BC tree, ∃nodei = ζ(Ii ). So we have topici ⇒ nodei . On the other hand, since both τ and ζ are reversible, ∃τ − and ∃ζ − saitisfy topici = τ − (Ii ) = τ − 1(ζ − (nodei )). So we have nodei ⇒ topici . Therefore topici ⇔ nodei Theorem 2. Set p− treei (i = 1, 2, . . . , n) is a branch path of BC, l− topici is an interesting topic hierarchical path, then p− treei ⇔ l− topici . Set Layer− block = {l− block1 , l− block2 , . . . , l− blockn }, is a hierarchical path of a block in a Web page. Lay− topic = {l− topic1 , l− topic2 , . . . , l− topicn }, is a hierarchical path of an interesting topic for a user. According to the principle of browsing Web page, ∃l− blocki = ϕ(l− topici ). According to our strategy of constructing a BC tree, ∃p− treei = ψ(l− blocki ). So we have l− topici ⇒ p− treei . On the other hand, since both ϕ and ψ are reversible, ∃ϕ− and ∃ψ − satisfy l− topici = ϕ− (l− blocki ) = ϕ− (ψ − (p− treei )). So we have p− treei ⇒ l− topici . Therefore p− treei ⇔ l− topici . Theorem 3. The deflection branch of a BC tree is a user interesting information preference. Set tree is a BC tree, treei is the deflection branch of the tree, the width and height of the tree are dtreei , htreei , and the width and height deflection factors are Sd and Sh , respectively. According to Definition 6, we have |dtreei − Sd | ≥ ε, |htreei − Sh | ≥ ε. According to Theorems 1 and 2, we known that tree1 is the Web user interesting information preference.

4

Discovering User Behavior Characteristics

On the basis of the above discussion, we designed three algorithms to mine user behavior characteristics on the BC Tree, such as frequent paths, interesting topics, interesting preferences etc. Algorithm 2 is used to find frequent paths. For example, if the frequency of the path sports → worldcup → f ootball satisfies the condition of supporting, we take the path as frequent paths. Algorithm 2. Mining a frequent path Input: a BC tree, ξ Output: a frequent path Procedure: 1. calculate support of all son-nodes of the BC and cut out all branches of BC tree, which satisfies support(son− node) ≤ ξ 2. repeat step 1 until no branch being cut out; 3. go through the BCT ree and output all F requentP aths.


527

Algorithm 3 is used to discover the user behavior characteristics of an interesting topic. For example, a user is interested in the national and international news. He/She would browse the topic about news on the corresponding website, probably on the BBC website or Yahoo. Anyway, so long as a page includes some news, the user will be interested in the page. This algorithm can help a user to find his/her interested website. Algorithm 3. Mining an interesting topic Input: a BC tree, ξ Output: an interesting topic set Procedure: 1. for every node in BC, calculate support(node); 2. merger the same name node; 3. for every node, if support(node) > ξ then insert the node into InterestingT opicSet; 4. output InterestingT opicSet. Algorithm 4 is used to discover the preference of interesting of a user. This is an interesting result in our work. Usually, there are many sub-topics in one topic. For example, local news, national news and international news are the sub-topics of news. Furthermore, international news includes news about Middle East, America, Korea and Japan; Korean news includes military affairs, politics etc. Two Web users are interested in this news, but the one is more interested in the news of Korea military affairs, the other one is only interested in the news in general. Algorithm 4. Mining interesting preference from the deflexed BC tree Input: a deflection BC tree, ξ Output: an interesting preference Procedure: 1. calculate d, h of every branch of a BC tree 2. calculate Sd , Sh of the BC 3. for every branch BC tree if |d − Sd | ≥ ξ if |h − Sh | ≥ ξ, then output the def lection branch in d and h, else output the def lection branch only in d else if |h − Sh | ≥ ξ, then output the def lection branch only in h, else output the branch is not a def lection tree. Next branch BC. end.

5

Experiments

In the experiments, we took Matlab as a tool, and synthesized 3000 browse paths of a user. The experiments run in the DELL PC machine, 2G main memory, 2.6GHz CPU. It took only two seconds to construct a BC tree, and all other tasks,

528


which obtain the behavior characteristics, such as interesting topic, interesting information path and preference of the topic, could be finished in one second. All results of our experiments are listed in Tables 2, 3, 4 and 5. Table 2 shows the frequent supporting of an access path. Table 3 shows the supports of an interesting topic. Tables 4 and 5 show the width and height deflections. Table 2. Frequent path supports

Table 3. Interesting topic supports

Table 4. Width deflections

Table 5. Height deflections

Based on the above discussion, we could acquire characteristic information of Web user browser behavior instantly, such as interesting topic, frequent behavior path and interesting preference of a Web user. Furthermore, we could answer questions, such as what the extent and depth of an interesting topic are according to interesting preference. The behavior characteristics of Web users could not be discovered by other techniques. The frequent path table could tell us what frequent behavior paths of a web user on Web page are. The path is a hierarchy of layout of a page, not a link path in the net. It, in fact, is a hierarchic classification. A node in the path is a class in the logic hierarchy. Therefore, our technique is practicable and valuable for the initiative retrieval and information recommendation.

6

Conclusion

Taking into account the layout of Web pages, we have constructed a BC tree from the browsing history of Web users. And then we proved that the BC tree is


529

equivalent to user browse behavior. The BC tree is mined to discover user browsing behavior characteristics. The technique is valuable for Web document presending, link recommendation, and personalized information retrieval. It could answer questions, such as “what is the extent of an interesting topic?” “how much is the depth of an interesting topic?” and “what are frequent paths on the Web page?” and so on. While previous techniques only tell us the user characteristics on some specific site, and the characteristics acquired by present techniques are only about link related characteristics, not about the characteristics of a page layout. The layout characteristics are classification of information item on the page, but it is neglected by other techniques.

References 1. Cao, L., Zhao, Y., Zhang, H., Luo, D., Zhang, C.: Mining impact-targeted activity atterns in imbalanced data. IEEE Transactions on Knowledge and Data Engineering 20(8), 1053–1066 (2008) 2. Choa, Y.B., Chob, Y.H., Kimc, S.H.: Mining changes in customer buying behavior for collaborative recommendations. Expert Systems with Applications 28, 359–369 (2005) 3. Kobayashi, I., Saito, S.: A study on an information recommendation system that provides topical information related to user’s inquiry for information retrieval. New Generation Computing 26(1), 39–48 (2008) 4. Ohshima, M., Zhong, N., Yao, Y.Y., Liu, C.: Relational peculiarity oriented mining. Data Mining and Knowledge Discovery 15(2), 249–273 (2007) 5. Park, Y.-J., Chang, K.-N.: Individual and group behavior-based customer profile model for personalized prodBCt recommendation. Expert Systems with Applications 36, 1932–1939 (2009) 6. Song, Q., Shepperd, M.: Mining Web browsing patterns for E-commerce. Computers in Industry 57, 622–630 (2006) 7. Wang, L., Meinel, C.: Behaviour recovery and complicated pattern definition in Web usage mining. In: Koch, N., Fraternali, P., Wirsing, M. (eds.) ICWE 2004. LNCS, vol. 3140, pp. 531–543. Springer, Heidelberg (2004) 8. Yang, Q., Yin, J., Ling, C., Pan, R.: Extracting actionable knowledge from decision trees. IEEE Transactions on Knowledge and Data Engineering 19(1), 43–56 (2007) 9. Zhang, H., Zhao, Y., Cao, L., Zhang, C., Bohlscheid, H.: Customer activity sequence classification for debt revention in social security. J. Comput. Sci. and Technol. (2009) 10. Zhao, L., Zhang, S., Fan, X.: Anonymous user network browser feature mining. Computer Research and Development 39(12), 1758–1764 (2002) 11. Zhong, N., Liu, J., Yao, Y.Y., Wu, J., Lu, S., Li, K. (eds.): Web Intelligence Meets Brain Informatics. LNCS (LNAI), vol. 4845. Springer, Heidelberg (2007) 12. Zhong, N., Liu, J., Yao, Y.Y.: Envisioning Intelligent Information Technologies through the Prism of Web Intelligence. CACM 50(3), 89–94 (2007) 13. Zhong, N., Liu, J. (eds.): Intelligent Technologies for Information Analysis. Springer, Heidelberg (2004) 14. Zhou, B., Wu, Q., Gao, H.: On model and algorithms for mining user access patterns. Computer Research and Development 36(7), 870–875 (1999) 15. Zhu, P., Lu, X., Zhou, X.: Based on customer behavior patterns Web document present. Journal of Software 10(11), 1142–1148 (1999)

Clustering Web Users Based on Browsing Behavior Tingshao Zhu School of Information Science and Engineering Graduate University of Chinese Academy of Sciences Beijing 100190, P.R. China

Abstract. It is critical to acquire web users behavior model in E-commerce community. In this paper, we propose to train the web user’s browsing behavior, and clustering web users based on their browsing behavior. In particular, our method uses page-content information extracted from the user’s click stream, then trains a behavior model that describes how web user locates useful information in the Internet. The classifier is trained on the data which describes how the user treats the information that she has visited, that is, browsing behavior. We find that some user groups based on the browsing behavior can get much higher accuracy for prediction.

1

Introduction

While the World Wide Web contains a vast amount of information, people spend more and more time on browsing to find useful information, such as web pages that contain information that they are interested in, or some products that they are willing to buy. Meanwhile, because of the massive information in the Internet, it is very often difficult for web users to find the particular pieces of information that they are looking for. This has led to the development of a number of recommendation systems, which typically observe a user’s navigation through a sequence of pages, and then suggest pages that may provide relevant information (see, e.g., [7], [11]). There are also some research focusing on web science [1] [4], to find how people interact with the web, thus to understand more about web users. In this paper, we propose to cluster web users based on their browsing behaviors. Intuitively, the group-based behavior model has a better performance than population model [17] which is expected to produce Information-Content(IC) pages which are those pages that user must examine to complete her task.

2

Related Work

There are many ways to generate recommendations for web users. Collaborative Filtering [13] is the first attempt using AI technology for personalization [9], but it is unrealistic to ask the user to rank all the pages that explored, and it is very A. An et al. (Eds.): AMT 2010, LNCS 6335, pp. 530–537, 2010. c Springer-Verlag Berlin Heidelberg 2010

Clustering Web Users Based on Browsing Behavior

531

difficult to get enough manually labeled web pages in real world. Some frequency based methods such as[7], [11] can also be used to predict specific URLs, but they are not capable in some cases, especially across web sites. Since very few web pages would be accessed very frequently, so very few or even no clusters, rules or sequential patterns can be obtained, thus the recommendation system will keep in silence almost all the time. Most of all, these systems are trained on specific web pages or one particular web site, that is, they cannot give any recommendations if applied in a new environment. In our research we want to acquire web users behavior model describing how user finds the information that she really wants, and the model is trained based on generalize click streams. Since such model is not based on specific URLs or web pages, it can be used even in a totally new web environment. And we also propose that some web users may have very similar browsing behavior, since some users may have similar interesting, background, and browsing preference. Each such group should present very strong behavior model, thus it is much easier to acquire group’s behavior, and also recommendation system adaptive to the group may be real useful for the group member. Web user behavior model extraction is also a critical task for E-commerce community, and much work have been done here. Lynch and Ariely [6] show that if the customers can be presented with useful product-related information, it will increase their satisfaction with the merchandize they buy. In other word, if we can infer the goal of web users, and we not only can retrieve related information, but also can help them dramatically. Randolph and Sismeiro [2] developed a model to describe within-site browsing behavior: the visitor’s decisions to stay or exit and the duration of each page view. But their research is at individual level, and nothing to do with what kind of information users want. Park and Fader [10] incorporate observable outcomes and the latent propensities across web sites to build a web browsing behavior model. Moe et. al, [8] use a Bayesian Tree Model to present online purchasing model. Johnson, Moe, Fader, Bellman, and Lohse [5] propose a model of the users search behavior, but it only give us brief description, not explicit enough to infer what users want. However, the research in this area is limited, either does not take into account the content of pages viewed, or the model is too general to be used in real application. And the search behavior model is always proposed by expert after examining the recorded log data which may lose some important aspects of the real web user model. In our research, we propose to acquire user information seeking model by using machine learning, based on the content of her observed click stream. In particular, our method uses page-content information extracted from the user’s click stream to learn a classifier to obtain the model that describes how web user locates useful information in the Internet. The classifier is not trained on specific words or URLs, but generalized information to indicate how the use treats the information that she has visited, that is, browsing behavior. And also we cluster web users based on the browsing behavior, that is, not on what kind of information they want, but how they find useful information.

532

T. Zhu

This paper describes our research on learning web user browsing behavior, we are not supposed to find explicit rules or model, but to train machine learning algorithm to let it obtain the patterns of the web browsing, and then clustering web users based on such browsing behavior. Section 3 describes how we collected such data. Section 3.2 shows how we used the collected information to learn a classifier to obtain the web users’ browsing behavior, and in Section 4, we introduce our algorithm to find user groups based on browsing behavior, and also report the results of a first test of the performance of this approach, based on the data collected in our study.

3 3.1

Data Preprocessing User Study

To learn, and later evaluate, we collected a set of annotated web logs; a sequence of web pages that a user visits, where each page is labeled with a bit that indicates whether this page is ”IC” - i.e., essential to achieving the user’s specific task goal. We collected these annotated web logs in a laboratory study. A total of 128 subjects participated in the study. Each participant was asked to perform a specific task: 1. Identify 3 novel vacation destinations (i.e., places you have never visited) that you are interested in. 2. Prepare a detailed plan for a vacation at each destination (including specific travel dates, flight numbers, accommodation, activities, etc.). They were given access to our browsing tool (AIE - Annotation Internet Explorer; described below) [16], which recorded their specific web logs, and required them to provide the IC annotation. Each participant also had to produce a short report summarizing her specific vacation plans ; AIE was engineered to help the user remember these citations and insert them into her travel plan. To help motivate participants to take this exercise seriously, they were informed that two (randomly selected) participants would win $500 to help pay for the specific vacation they had planned. 3.2

Learning Problem

In the dataset, we know which pages are IC pages and the page sequence proceeded. Our goal is a classifier that can predict which words will appear in the IC page. We do not train the classifier based on specific words, but some features to show how user treats these words, because we think that based on the observation of the user’s action to the words, we can predict what information she wants. To train such a classifier, we first gather all the words from the observed page sequence, then compute certain features for each word, based on how that word appeared within the sequence. The label for each word is whether it appears in the IC page that terminates this session (Thanks to AIE, we know these IC labels.).


3.3

533

IC-Session Identification

To make it easy for our learner, we divide the whole page sequence of each subject into several IC-Sessions. An ”IC Session” is a consecutive sequence of pages that ends with an IC page, or the end of the user’s traversal sequence. In our case, since the browsing is task driven, we terminate a session on reaching an IC page. However, it is not clear that the next session should begin on the subsequent page. For example, imagine reaching an index page I after visiting a sequence of pages A → B → C → I, and moreover, I contains a number of useful links, say I → P1 and I → P2 , where both P1 and P2 were ICs. Here, each IC session should contain the sequence before the index page since they also contribute to locating each of the IC pages - i.e., given the browsing sequence A → B → C → I → P1 → I → P2 , we would produce the two ICsessions A → B → C → I → P1 and A → B → C → I → P2 . We use some heuristics to identify these IC-session, including the idea that sessions end with search engine queries, since it is very common that when one task is done, people will go to a search engine to begin the next task. 3.4

Feature Extraction

We consider all words that appear in all pages, removing stop words and stemming [12], and calculate a weight for each of these remaining words. The word’s frequency in the page is its initial weight; we add more weight according to its location and layout in the web page, such as, in the title, bold, strong size, etc. [14]. We next compute the following attributes, for each word, from the IC-session, to get a detailed description of these features, please refer to [15]. Search Query Category. As our data set includes many requests to search engines, we include several attributes to relate to the words in the search result pages. Each search engine will generate a list of results according to the query, but the content of each result may differ for different search engines. For example, one result from Google contains Description and Category, but in Altavista’s search result, there is no such information. In our research, we only considered information produced by every search engine: the title (i.e., the first line of the result) and the snippet (i.e., the text below the title). We can tag each title-snippet pair in each search result page as one of: Skipped, Chosen, and Untouched. If the user follows a link, the state of its title and snippet will be “Chosen”. All links in the list that the user did not follow, before the last chosen one, will be deemed “Skipped”, and all results after the last chosen link in the list will be “Untouched”. Sequential Attributes. All the following measures are extracted from pages in an IC session except the search result pages and the last destination page. If the URL refers to a frame page, then we will calculate all the following measures

534

T. Zhu

based on the page view. We say a hyperlink (in page P) is backed if the user followed that link to other page, but went back to page P later. A page is backward if that page has been visited before; otherwise we say a page is forward. 3.5

Classifier Training

After preparing the data, we learned NaiveBayes (NB) classifier. Recall that NB is a simple belief net structure which assumes that the attributes are independent of one another, conditioned on the class label [3]. Since the dataset we have collected is very imbalanced, that is, the number of N egative(Non-IC Words) is far greater than the number of P ositive(IC Words) ones. To generate the training and testing data, we randomly selected positive and negative instances of equal size as testing data, then diminishing negative samples by randomly select to get the equal number of positive and negative training samples. For each IC session, let wseq denote all the words in the sequence except the last page, which is an IC page, wdest denotes the words in that final IC page, and |wseq wdest| . coverage = wdest To better understand this, we computed precision and recall values. The ”Precision” for each IC page is TruePositive/ PredictedAsPositive and the ”Recall” for IC words is TruePositive/AllRealPositive. Similarly, we define Precision and Recall for non-IC words as TrueNegative/ PredictedAsNegative and TrueNegative/AllRealNegative, respectively. For each trial, we built 10-fold training/testing datasets, and computed the median value of these 10 results’ precision/recall as the final precision/recall. (We used medians because they are less sensitive to outliers than means.) For positive and negative prediction, we compute F-Measure: recision∗Recall F − M easure = 2∗P P recision+Recall and egativeF −M easure as the final prediction AF −M easure = P ositiveF −Measure+N 2 accuracy of this trial.

4

Greedy Web User Clustering

We propose that there exists general model of goal-directed information search on the web, and some users that share similar behavior can be clustered as one user group. The generic method for clustering web users is to represent each user with a feature vector, and then apply clustering algorithm(K-means, etc.) to find some user groups. But for web user, it is very difficult to define the exact features, since they may have very different interesting. Sometimes it is impossible to obtain users’ profile for traditional clustering analysis. Recall that the features that we extract for each word can be considered as the user’s browsing preference, thus we propose to find user clusters based on such feature data.


Algorithm W ebU serClustering : The subjectsU (ui , i = 1, 2, 3, . . . , n) CandidateL: Queue; stores the candidate groups. BEGIN Clear CandidateL Forany 2-user group {ui , uj }, i, j = 1 . . . n and i =j if its AF − M easure ≥ threshold, put it into CandiateL; while CandiateL is not Empty begin Removehead of CandidateL as maxGroup For each group(checkGroup) remaining in CandiateL begin if checkGroup ⊆ maxGroup, remove checkGroup; else if checkGroup maxGroup not Empty begin Merge them together, if the new group’s AF − M easure ≥ threshold take it as maxGroup, remove checkGroup; end end Output maxGroup end END

Fig. 1. Testing Result for Web User Clustering

535

536

T. Zhu

In Figure 1, we compare the AF −M easure of our greedy algorithm with other two Na¨ıve methods. The first is just to put all users in one big group, and the second one is take each user as a group, and calculate the average AF-Measure of all such 1-user groups. From Fig. 1, it is easy to see that the prediction increased dramatically for greedy group. This also support our proposal that there exist strong regularities among web user groups, and within such groups, we can acquire more accurate browsing model, and can provide better services for group member.

5


Our ultimate goal is to get web users where they want to go more quickly. The first step towards this is to identify web user’s behavior model from the click stream. In particularly, we acquire the user model by training a classifier based on generalized click stream. The training data for the classifier is not just some specific words or URLs, but the way how web user treats the observable information. By clustering based on such generalized data, we can find web user groups, and for each group, we can get fair high accuracy. We are currently investigating more efficient ways to improve the accuracy of the prediction, and ways to further increase the recall for the positive prediction. We also plan to explore the potential of content and structure mining, as well as tools for learning from imbalanced datasets to aid us in this endeavor.

References 1. Berners-Lee, T., Hall, W., Hendler, J., Shadbolt, N., Weitzner, D.J.: Creating a science of the web. IEEE Transactions on Systems, Man and Cybernetics 36 (2006) 2. Bucklin, R., Sismeiro, C.: A model of web site browsing behavior estimated on clickstream data 3. Duda, R., Hart, P.: Pattern Classification and Scene Analysis. Wiley, New York (1973) 4. Hendler, J., Shadbolt, N., Hall, W., Berners-Lee, T., Weitzner, D.: Web science: An interdisciplinary approach to understanding the web. Communications of the ACM 51(7) (2008) 5. Johnson, E., Moe, W., Fader, P., Bellman, S., Lohse, J.: On the depth and dynamics of world wide web shopping behavior. Management Science 50(3), 299–308 (2004) 6. Lynch, J., Ariely, D.: Wine online: Search costs and competition on price, quality, and distribution. Marketing Science 19(1), 83–103 (2000) 7. Mobasher, B., Cooley, R., Srivastava, J.: Automatic personalization through web usage mining. Technical Report TR99-010, Department of Computer Science, Depaul University (1999) 8. Moe, W., Chipman, H., George, E., McCulloch, R.: A bayesian treed model of online purchasing behavior using in-store navigational clickstream 9. Mulvenna, M., Anand, S., B¨ uchner, A.: Personalization on the net using web mining: introduction. Communications of ACM 43(8), 122–125 (2000) 10. Park, Y.-H., Fader, P.: Modeling browsing behavior at multiple websites


537

11. Perkowitz, M., Etzioni, O.: Adaptive sites: Automatically learning from user access patterns. Technical Report UW-CSE-97-03-01, University of Washington (1997) 12. Porter, M.: An algorithm for suffix stripping. Program 14(3), 130–137 (1980) 13. Resnick, P., Iacovou, N., Suchak, M., Bergstorm, P., Riedl, J.: Grouplens: An open architecture for collaborative filtering of netnews. In: Proceedings of ACM 1994 Conference on Computer Supported Cooperative Work, Chapel Hill, North Carolina, pp. 175–186. ACM, New York (1994) 14. W3C. Html 4.01 specification 15. Zhu, T.: Goal-Directed Complete-Web Recommendation. PhD thesis, University of Alberta, Edmonton AB, Canada (2006) 16. Zhu, T., Greiner, R., H¨ aubl, G.: Learning a model of a web user’s interests. In: Brusilovsky, P., Corbett, A.T., de Rosis, F. (eds.) UM 2003. LNCS, vol. 2702, Springer, Heidelberg (2003) 17. Zhu, T., Greiner, R., H¨ aubl, G., Jewell, K., Price, B.: Goal-directed siteindependent recommendations from passive observations. In: Proceedings of the Twentieth National Conference on Artificial Intelligence (AAAI 2005), Pittsburgh, Pennsylvania, pp. 549–557 (July 2005)

Privacy Preserving in Personalized Mobile Marketing Yuqing Sun and Guangjun Ji School of Computer Science and Technology, Shandong University sun [email protected], jgj [email protected]

Abstract. With the popularity of portable smart devices and advances in wireless technologies, mobile marketing increases quickly. Among various methods, short message is regarded as the most efficient mode. While mobile advertising enhances communication with consumers, the messages without a required permission from users cause privacy violations. So, how to simultaneously support personalization and privacy preserving in mobile marketing is a challenging problem. In this paper, we investigate this problem and propose a Privacy Preserving Model for Personalized Mobile Marketing (P 2 PMM ), that can protect both users’ preferences and location privacy while supporting personalization in mobile marketing. We propose an efficient coding method to manage hierarchical categories and users’ preferences. We also investigate how such a model can be applied into practice and a prototype is implemented.

1 Introduction Mobile Marketing is a set of practices that enables organizations to communicate and engage with their audience in an interactive and relevant manner through any mobile device or network [1]. With the advances in wireless technologies and the popularity of portable smart devices, mobile marketing increases quickly. Among various methods, short message (SMS) is regarded as the most efficient mode [2] when businesses start to collect mobile phone numbers and send off wanted (or unwanted) content to users. Another trend on mobile marketing is the type of location-based services (LBS) that can offer messages to users based on their current location. A cell phone service provider gets the location from a GPS (Global Positioning System) chip built into a phone, or by radiolocation and trilateration based on the signal-strength of the closest cell-phone towers. While mobile marketing enhances communication with users, it may causes privacy violations when the sent messages are without a required permission from consumers. A number of concerns, such as mobile spam, personal identification, location information and wireless security etc., mainly stem from the fact that mobile devices are intimately personal and are always with the user[3]. Experts cited fear of spam as the strongest negative influence on consumer attitudes towards SMS advertising [4]. Actually, no matter how well advertising messages are designed, if consumers do not have confidence that their privacy will be protected, this will hinder their widespread deployment[5]. To support personalization, messages should be appropriately tailored before sending to consumers. Solutions have been deployed to personalize text messages based on a consumer’s local time, location, and preferences [6], e.g. directions to the nearest vegetarian restaurant open at the time of request. A. An et al. (Eds.): AMT 2010, LNCS 6335, pp. 538–545, 2010. c Springer-Verlag Berlin Heidelberg 2010

Privacy Preserving in Personalized Mobile Marketing

539

However, personalization in mobile marketing means collecting and storing information about a particular person, such as monitoring of user behavior, which causes other privacy concerns [7]. To address such problems, different techniques have been proposed that are based on two main approaches: location cloaking, under which a suitable large region is returned to the service provider instead of the precise user location [8]; location k-anonymization, under which the location of an individual is returned to the service provider only if it is indistinguishable with respect to the location of other k-1 individuals [9,10]. Some recent works discuss the preferences on privacy, such as the Privacy-preserving Obfuscation Environment system (PROBE)[11]. But these works are not suitable for privacy preserving in mobile marketing since they do not consider the customization requirement of message contents. In this paper, we investigate the problem on how to protect users’ privacy while supporting personalization. In the proposed Privacy Preserving Model for Personalized Mobile Marketing (P 2 PMM ), users can customize their preferences for messages without any privacy leakage to information provider. A trusted third party collects the marketing messages and makes classification according to predefined categories. Users thus have their options on location, time and categories etc. We investigate how such a model can be realized in the GPS and cellular network systems. The prototype system is designed and implemented. The remainder of the paper is organized as follows. Section 2 presents the main components of the proposed model. Section 3 investigates the problem of efficient information organization and query processing in the model. Section 4 discusses the system architecture and describes the details on implementation of the prototype. Section 5 concludes the paper and outlines future research directions.

2 The Privacy Preserving Model for Personalized Mobile Marketing In this section we introduce the Privacy Preserving Model for Personalized Mobile Marketing (P 2 PMM for short), depicted as Figure 1. There are four entities in this model. The functionalities of each party are narrated as follows. – Mobile network operator (M N O): is a telephone company that provides communication services for mobile phone subscribers. – Intermediary services provider(ISP ): is a trusted third party for users that is independent to Merchants. It provides platform for merchants to manage their advertisements, as well as for users to subscribe their interested messages and maintain private individual data. Sometimes, it can be integrated with M N O if required. – Users: are the cell phone subscribers. After registering on ISP , they are allowed to option their preferred messages from ISP based on their location, time or interested topics. – Merchants: represent the organizations who want to advertise their business messages. After registering on ISP , they are allowed to publish their advertisements to interested users. There are three distinct characteristics from other mobile marketing models. Firstly, it is active marketing. The P 2 PMM model is in a “PULL” schema rather than a traditional

540

Y. Sun and G. Ji

Fig. 1. The Privacy Preserving Model for Personalized Mobile Marketing

“PUSH” way such that all the messages sent to users are what they want. Second is that users’ preferences privacy is preserved. The sensitive information of an individual, such as user profile, user preferences in each query and current location etc., are stored in ISP . This avoids the case that every merchant has a copy of user profile. Thirdly, users’ location privacy is also preserved. ISP is allowed to acquire the approximate location square without any awareness of the exact real-time position of a user. Now we formalize the basic notions of our model. Definition 1 (Position). A geographic position Loc = [lngt, lat] denotes a point in a map, where lngt and lat are the longitude and latitude value of this point. Let P OS denote the class of positions. U and M respectively denote the set of users and the set of merchants. We assume that every merchant is associated with an exact position. When a merchant registers on ISP , its position can be acquired by some position technologies like GoogleM ap. Similarly, every mobile phone user has an exact position at any time, which can be acquired by GPS or wireless positioning technologies. We introduce two predicates here LocU (u ∈ U ) : P OS and LocM (m ∈ M ) : P OS to calculate the location of a user and a merchant, respectively. Definition 2 (Message). Let ISSUE denote the set of all issues considered in the model. A message msg is specified as a tuple msg=< ID, T XT, Issues, T W, Loc >, where ID is the unique identifier of msg, T XT represents the message content, Issues ⊆ ISSUE is a subset of ISSUE denoting the issues correlated with msg, T W is in form of [t1 , t2 ] representing the time window when msg is effective, and Loc ∈ P OS is in form of [lngt, lat] denoting the merchant’s position who launches msg. For example, the department store M acy ∗ s in West Lafayette, IN wants to make an advertisement for sales promotion. It launches the following message to ISP : msg=

Active Media Technology: 6th International Conference, AMT 2010, Toronto, Canada, August 28-30, 2010, Proceedings (Lecture Notes in Computer Science ... Applications, incl. Internet Web, and HCI)

Active Media Technology: 5th International Conference, AMT 2009, Beijing, China, October 22-24, 2009, Proceedings (Lecture Notes in Computer Science ... Applications, incl. Internet Web, and HCI)

Web Information Systems and Mining: International Conference, WISM 2010, Sanya, China, October 23-24, 2010, Proceedings (Lecture Notes in Computer ... Applications, incl. Internet Web, and HCI)

Pervasive Computing: 8th International Conference, Pervasive 2010, Helsinki, Finland, May 17-20, 2010, Proceedings (Lecture Notes in Computer Science ... Applications, incl. Internet Web, and HCI)

Persuasive Technology: 5th International Conference, PERSUASIVE 2010, Copenhagen, Denmark, June 7-10, 2010, Proceedings (Lecture Notes in Computer ... Applications, incl. Internet Web, and HCI)

Web-Age Information Management: 11th International Conference, WAIM 2010, Jiuzhaigou, China, July 15-17, 2010, Proceedings (Lecture Notes in Computer ... Applications, incl. Internet Web, and HCI)

Geographic Information Science: 6th International Conference, GIScience 2010, Zurich, Switzerland, September 14-17, 2010. Proceedings (Lecture Notes ... Applications, incl. Internet Web, and HCI)

Brain Informatics: International Conference, BI 2010, Toronto, Canada, August 28-30, 2010, Proceedings (Lecture Notes in Computer Science Lecture Notes in Artificial Intelligence)

Pervasive Computing: 5th International Conference, PERVASIVE 2007, Toronto, Canada, May 13-16, 2007, Proceedings (Lecture Notes in Computer Science Information ... Applications, incl. Internet Web, and HCI)

Social Informatics: Second International Conference, SocInfo 2010, Laxenburg, Austria, October 27-29, 2010, Proceedings (Lecture Notes in Computer ... Applications, incl. Internet Web, and HCI)

Business Process Management: 8th International Conference, BPM 2010, Hoboken, NJ, USA, September 13-16, 2010, Proceedings (Lecture Notes in Computer ... Applications, incl. Internet Web, and HCI)

Ambient Intelligence: First International Joint Conference, AmI 2010, Málaga, Spain, November 10-12, 2010, Proceedings (Lecture Notes in Computer ... Applications, incl. Internet Web, and HCI)

Distributed Computing and Internet Technology: 6th International Conference, ICDCIT 2010, Bhubaneswar, India, February 15-17, 2010, Proceedings ... Applications, incl. Internet Web, and HCI)

Algorithmic Game Theory: Third International Symposium, SAGT 2010, Athens, Greece, October 18-20, 2010, Proceedings (Lecture Notes in Computer Science ... Applications, incl. Internet Web, and HCI)

Advances in Web Based Learning - ICWL 2009: 8th International Conference, Aachen, Germany, August 19-21, 2009, Proceedings (Lecture Notes in Computer Science ... Applications, incl. Internet Web, and HCI)

Algorithmic Aspects in Information and Management: 6th International Conference, AAIM 2010, Weihai, China, July 19-21, 2010. Proceedings (Lecture ... Applications, incl. Internet Web, and HCI)

Secure Data Management: 7th VLDB Workshop, SDM 2010, Singapore, September 17, 2010, Proceedings (Lecture Notes in Computer Science Information Systems and Applications, incl. Internet Web, and HCI)

Advanced Information Systems Engineering: 22nd International Conference, CAiSE 2010, Hammamet, Tunisia, June 7-9, 2010, Proceedings (Lecture Notes in ... Applications, incl. Internet Web, and HCI)

Secure Data Management: 7th VLDB Workshop, SDM 2010, Singapore, September 17, 2010, Proceedings (Lecture Notes in Computer Science Information Systems and Applications, incl. Internet Web, and HCI)

Conceptual Modeling - ER 2010: 29th International Conference on Conceptual Modeling, Vancouver, BC, Canada, November 1-4, 2010, Proceedings (Lecture ... Applications, incl. Internet Web, and HCI)

Advances in Web Based Learning - ICWL 2009: 8th International Conference, Aachen, Germany, August 19-21, 2009, Proceedings (Lecture Notes in Computer ... Applications, incl. Internet Web, and HCI)

Active Media Technology: 6th International Computer Science Conference, AMT 2001, Hong Kong, China, December 18-20, 2001. Proceedings

Advances in Intelligent Data Analysis IX: 9th International Symposium, IDA 2010, Tucson, AZ, USA, May 19-21, 2010, Proceedings (Lecture Notes in ... Applications, incl. Internet Web, and HCI)

Secure Data Management: 6th VLDB Workshop, SDM 2009, Lyon, France, August 28, 2009, Proceedings (Lecture Notes in Computer Science Information Systems and Applications, incl. Internet Web, and HCI)

Haptic and Audio Interaction Design: 5th International Workshop, HAID 2010, Copenhagen, Denmark, September 16-17, 2010, Proceedings (Lecture Notes in ... Applications, incl. Internet Web, and HCI)

Information, Technology in Bio- and Medical Informatics, ITBAM 2010: First International Conference, Bilbao, Spain, September 1-2, 2010, Proceedings ... Applications, incl. Internet Web, and HCI)

Web Engineering: 9th International Conference, ICWE 2009 San Sebastián, Spain, June 24-26 2009 Proceedings (Lecture Notes in Computer Science ... Applications, incl. Internet Web, and HCI)

Advances in Computer Science and Information Technology: AST UCMA ISA ACN 2010 Conferences, Miyazaki, Japan, June 23-25, 2010. Joint Proceedings ... Applications, incl. Internet Web, and HCI)

Research and Advanced Technology for Digital Libraries: 14th European Conference, ECDL 2010, Glasgow, UK, September 6-10, 2010, Proceedings (Lecture ... Applications, incl. Internet Web, and HCI)

User Modeling, Adaptation, and Personalization: 18th International Conference, UMAP 2010, Big Island, HI, USA, June 20-24, 2010, Proceedings (Lecture ... Applications, incl. Internet Web, and HCI)

Active Media Technology: 6th International Conference, AMT 2010, Toronto, Canada, August 28-30, 2010, Proceedings (Lecture Notes in Computer Science ... Applications, incl. Internet Web, and HCI)

Active Media Technology: 5th International Conference, AMT 2009, Beijing, China, October 22-24, 2009, Proceedings (Lecture Notes in Computer Science ... Applications, incl. Internet Web, and HCI)

Web Information Systems and Mining: International Conference, WISM 2010, Sanya, China, October 23-24, 2010, Proceedings (Lecture Notes in Computer ... Applications, incl. Internet Web, and HCI)

Pervasive Computing: 8th International Conference, Pervasive 2010, Helsinki, Finland, May 17-20, 2010, Proceedings (Lecture Notes in Computer Science ... Applications, incl. Internet Web, and HCI)

Persuasive Technology: 5th International Conference, PERSUASIVE 2010, Copenhagen, Denmark, June 7-10, 2010, Proceedings (Lecture Notes in Computer ... Applications, incl. Internet Web, and HCI)

Web-Age Information Management: 11th International Conference, WAIM 2010, Jiuzhaigou, China, July 15-17, 2010, Proceedings (Lecture Notes in Computer ... Applications, incl. Internet Web, and HCI)

Geographic Information Science: 6th International Conference, GIScience 2010, Zurich, Switzerland, September 14-17, 2010. Proceedings (Lecture Notes ... Applications, incl. Internet Web, and HCI)

Brain Informatics: International Conference, BI 2010, Toronto, Canada, August 28-30, 2010, Proceedings (Lecture Notes in Computer Science Lecture Notes in Artificial Intelligence)

Pervasive Computing: 5th International Conference, PERVASIVE 2007, Toronto, Canada, May 13-16, 2007, Proceedings (Lecture Notes in Computer Science Information ... Applications, incl. Internet Web, and HCI)

Social Informatics: Second International Conference, SocInfo 2010, Laxenburg, Austria, October 27-29, 2010, Proceedings (Lecture Notes in Computer ... Applications, incl. Internet Web, and HCI)

Business Process Management: 8th International Conference, BPM 2010, Hoboken, NJ, USA, September 13-16, 2010, Proceedings (Lecture Notes in Computer ... Applications, incl. Internet Web, and HCI)

Ambient Intelligence: First International Joint Conference, AmI 2010, Málaga, Spain, November 10-12, 2010, Proceedings (Lecture Notes in Computer ... Applications, incl. Internet Web, and HCI)

Distributed Computing and Internet Technology: 6th International Conference, ICDCIT 2010, Bhubaneswar, India, February 15-17, 2010, Proceedings ... Applications, incl. Internet Web, and HCI)

Algorithmic Game Theory: Third International Symposium, SAGT 2010, Athens, Greece, October 18-20, 2010, Proceedings (Lecture Notes in Computer Science ... Applications, incl. Internet Web, and HCI)

Advances in Web Based Learning - ICWL 2009: 8th International Conference, Aachen, Germany, August 19-21, 2009, Proceedings (Lecture Notes in Computer Science ... Applications, incl. Internet Web, and HCI)

Algorithmic Aspects in Information and Management: 6th International Conference, AAIM 2010, Weihai, China, July 19-21, 2010. Proceedings (Lecture ... Applications, incl. Internet Web, and HCI)

Secure Data Management: 7th VLDB Workshop, SDM 2010, Singapore, September 17, 2010, Proceedings (Lecture Notes in Computer Science Information Systems and Applications, incl. Internet Web, and HCI)

Advanced Information Systems Engineering: 22nd International Conference, CAiSE 2010, Hammamet, Tunisia, June 7-9, 2010, Proceedings (Lecture Notes in ... Applications, incl. Internet Web, and HCI)

Secure Data Management: 7th VLDB Workshop, SDM 2010, Singapore, September 17, 2010, Proceedings (Lecture Notes in Computer Science Information Systems and Applications, incl. Internet Web, and HCI)

Conceptual Modeling - ER 2010: 29th International Conference on Conceptual Modeling, Vancouver, BC, Canada, November 1-4, 2010, Proceedings (Lecture ... Applications, incl. Internet Web, and HCI)

Advances in Web Based Learning - ICWL 2009: 8th International Conference, Aachen, Germany, August 19-21, 2009, Proceedings (Lecture Notes in Computer ... Applications, incl. Internet Web, and HCI)

Active Media Technology: 6th International Computer Science Conference, AMT 2001, Hong Kong, China, December 18-20, 2001. Proceedings

Advances in Intelligent Data Analysis IX: 9th International Symposium, IDA 2010, Tucson, AZ, USA, May 19-21, 2010, Proceedings (Lecture Notes in ... Applications, incl. Internet Web, and HCI)

Secure Data Management: 6th VLDB Workshop, SDM 2009, Lyon, France, August 28, 2009, Proceedings (Lecture Notes in Computer Science Information Systems and Applications, incl. Internet Web, and HCI)

Haptic and Audio Interaction Design: 5th International Workshop, HAID 2010, Copenhagen, Denmark, September 16-17, 2010, Proceedings (Lecture Notes in ... Applications, incl. Internet Web, and HCI)

Information, Technology in Bio- and Medical Informatics, ITBAM 2010: First International Conference, Bilbao, Spain, September 1-2, 2010, Proceedings ... Applications, incl. Internet Web, and HCI)

Web Engineering: 9th International Conference, ICWE 2009 San Sebastián, Spain, June 24-26 2009 Proceedings (Lecture Notes in Computer Science ... Applications, incl. Internet Web, and HCI)

Advances in Computer Science and Information Technology: AST UCMA ISA ACN 2010 Conferences, Miyazaki, Japan, June 23-25, 2010. Joint Proceedings ... Applications, incl. Internet Web, and HCI)

Research and Advanced Technology for Digital Libraries: 14th European Conference, ECDL 2010, Glasgow, UK, September 6-10, 2010, Proceedings (Lecture ... Applications, incl. Internet Web, and HCI)

User Modeling, Adaptation, and Personalization: 18th International Conference, UMAP 2010, Big Island, HI, USA, June 20-24, 2010, Proceedings (Lecture ... Applications, incl. Internet Web, and HCI)

Recommend Documents