Semantic Multimedia: 4th International Conference on Semantic and Digital Media Technologies, SAMT 2009 Graz, Austria, December 2-4, 2009 Proceedings ... Applications, incl. Internet Web, and HCI)

Lecture Notes in Computer Science Commenced Publication in 1973 Founding and Former Series Editors: Gerhard Goos, Juris Hartmanis, and Jan van Leeuwen

Editorial Board David Hutchison Lancaster University, UK Takeo Kanade Carnegie Mellon University, Pittsburgh, PA, USA Josef Kittler University of Surrey, Guildford, UK Jon M. Kleinberg Cornell University, Ithaca, NY, USA Alfred Kobsa University of California, Irvine, CA, USA Friedemann Mattern ETH Zurich, Switzerland John C. Mitchell Stanford University, CA, USA Moni Naor Weizmann Institute of Science, Rehovot, Israel Oscar Nierstrasz University of Bern, Switzerland C. Pandu Rangan Indian Institute of Technology, Madras, India Bernhard Steffen TU Dortmund University, Germany Madhu Sudan Microsoft Research, Cambridge, MA, USA Demetri Terzopoulos University of California, Los Angeles, CA, USA Doug Tygar University of California, Berkeley, CA, USA Gerhard Weikum Max-Planck Institute of Computer Science, Saarbruecken, Germany

5887

Tat-Seng Chua Yiannis Kompatsiaris Bernard Mérialdo Werner Haas Georg Thallinger Werner Bailer (Eds.)

Semantic Multimedia 4th International Conference on Semantic and Digital Media Technologies, SAMT 2009 Graz, Austria, December 2-4, 2009 Proceedings

13

Volume Editors Tat-Seng Chua National University of Singapore 3 Science Drive, Singapore 117543, Singapore E-mail: [email protected] Yiannis Kompatsiaris Informatics and Telematics Institute Centre for Research and Technology–Hellas 6th km Charilaou-Thermi Road, 57001 Thermi-Thessaloniki, Greece E-mail: [email protected] Bernard Mérialdo Institut Eurécom Département Communications Multimédia 2229, route des Crêtes, 06904 Sophia-Antipolis CEDEX, France E-mail: [email protected] Werner Haas Georg Thallinger Werner Bailer JOANNEUM RESEARCH Forschungsgesellschaft mbH Institute of Information Systems Steyrergasse 17, 8010 Graz, Austria E-mail: {werner.haas, georg.thallinger, werner.bailer}@joanneum.at Library of Congress Control Number: 2009939151 CR Subject Classification (1998): H.5.1, H.4, I.7, I.4, H.5, H.3.5 LNCS Sublibrary: SL 3 – Information Systems and Application, incl. Internet/Web and HCI ISSN ISBN-10 ISBN-13

0302-9743 3-642-10542-4 Springer Berlin Heidelberg New York 978-3-642-10542-5 Springer Berlin Heidelberg New York

This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, re-use of illustrations, recitation, broadcasting, reproduction on microfilms or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable to prosecution under the German Copyright Law. springer.com © Springer-Verlag Berlin Heidelberg 2009 Printed in Germany Typesetting: Camera-ready by author, data conversion by Scientific Publishing Services, Chennai, India Printed on acid-free paper SPIN: 12798667 06/3180 543210

Preface

This volume contains the full and short papers of SAMT 2009, the 4th International Conference on Semantic and Digital Media Technologies 2009 held in Graz, Austria. SAMT brings together researchers dealing with a broad range of research topics related to semantic multimedia and a great diversity of application areas. The current research shows that adding and using semantics of multimedia content is broadening its scope from search and retrieval to the complete media life cycle, from content creation to distribution and consumption, thus leveraging new possibilities in creating, sharing and reusing multimedia content. While some of the contributions present improvements in automatic analysis and annotation methods, there is increasingly more work dealing with visualization, user interaction and collaboration. We can also observe ongoing standardization activities related to semantic multimedia in both W3C and MPEG, forming a solid basis for a wide adoption. The conference received 41 submissions this year, of which the Program Committee selected 13 full papers for oral presentation and 8 short papers for poster presentation. In addition to the scientific papers, the conference program included two invited talks by Ricardo Baeza-Yates and Stefan R¨ uger and a demo session showing results from three European projects. The day before the main conference offered an industry day with presentations and demos that showed the growing importance of semantic technologies in real-world applications as well as the research challenges coming from them. From the submitted proposals, the Workshop and Tutorial Chairs selected two full day workshops, namely: – Semantic Multimedia Database Technologies – Learning the Semantics of Audio Signals In addition, there were three half-day tutorials, namely: – A Semantic Multimedia Web: Create, Annotate, Present and Share Your Media – MPEG Metadata for Context-Aware Multimedia Applications – Web of Data in the Context of Multimedia The workshops complement the conference by providing a forum for discussion about emerging fields in the scope of SAMT and the tutorials are an opportunity for the participants to get a condensed introduction in one of the many areas related to semantic multimedia. This conference would not have been possible without the tremendous support of many people. We would like to thank the Workshop and Tutorial Chairs, Josep Blat, Noel O’Connor and Klaus Tochtermann, the Industry Day Chairs

VI

Preface

Wessel Kraaij and Alberto Messina, as well as Wolfgang Halb, Helen Hasenauer and Karin Rehatschek, who did a great job in organizing this event. We would like to thank the Program Committee members for the thorough review of the submissions, the invited speakers, the workshop organizers and tutors, and all contributors and participants. We are grateful for the support provided by the consortia of the SALERO and VidiVideo projects, the Young European Associated Researchers (YEAR) network, the City of Graz, the Province of Styria and the Austrian Federal Ministry of Science and Research. December 2009

Tat-Seng Chua Yiannis Kompatsiaris Bernard M´erialdo Werner Haas Georg Thallinger Werner Bailer

Conference Organization

General and Local Chairs Werner Haas Georg Thallinger Werner Bailer

JOANNEUM RESEARCH, Austria JOANNEUM RESEARCH, Austria JOANNEUM RESEARCH, Austria

Program Chairs Tat-Seng Chua Yiannis Kompatsiaris Bernard M´erialdo

National University of Singapore, Singapore ITI, Greece Eurecom, France

Program Committee Riccardo Albertoni Yannis Avrithis Bruno Bachimont Wolf-Tilo Balke Mauro Barbieri Jenny Benois-Pineau Stefano Bocconi Susanne Boll Nozha Boujemaa Tobias B¨ urger Chiara Catalano Oscar Celma Lekha Chaisorn Stavros Christodoulakis Philipp Cimiano Matthew Cooper Charlie Cullen Thierry Declerck Mark van Doorn Touradj Ebrahimi Alun Evans Bianca Falcidieno Christophe Garcia

IMATI-GE/CNR, Italy NTUA, Greece INA, France University of Hannover, Germany Philips Research, The Netherlands University of Bordeaux, France University of Trento / VUA, Italy/ The Netherlands University of Oldenburg, Germany INRIA, France STI Innsbruck, Austria University of Genova, Italy Universitat Pompeu Fabra, Spain I2R, Singapore Technical University of Crete, Greece Uni Karlsruhe, Germany FXPAL, USA Dublin Institute of Technology, Ireland DFKI, Germany Philips Research, The Netherlands Swiss Federal Institute of Technology, Switzerland Barcelona Media, Spain IMATI-GE/CNR, Italy France Telecom R&D, France

VIII

Organization

Joost Geurts Michael Granitzer William Grosky Siegfried Handschuh Michael Hausenblas Willemijn Heeren Winston Hsu Ichiro Ide Ignasi Iriondo Antoine Isaac Ebroul Izquierdo Alejandro Jaimes Joemon Jose Mohan Kankanhalli Brigitte Kerherv´e Stefanos Kollias Harald Kosch Hyowon Lee Jean Claude Leon Paul Lewis Craig Lindley Suzanne Little Vincenzo Lombardo Mathias Lux Erik Mannens Stephane Marchand-Maillet Simone Marini Jose M. Martinez Mark Maybury Oscar Mayor Vasileios Mezaris Carlos Monzo Michela Mortara Frank Nack Chong-Wah Ngo Zeljko Obrenovic Jacco van Ossenbruggen Jeff Z. Pan Thrasyvoulos Pappas Ewald Quak Lloyd Rutledge Mark Sandler Simone Santini Shin’ichi Satoh Ansgar Scherp

CWI, The Netherlands Know Center, Austria University of Michigan, USA DERI, Ireland DERI, Ireland University of Twente, The Netherlands NTU, Taiwan Nagoya University / NII, Japan Universitat Ramon Llull, Spain VUA, The Netherlands QMUL, UK Telefonica R&D, Spain University of Glasgow, UK NUS, Singapore Universit´e du Qu´ebec `a Montr´eal, Canada NTUA, Greece University of Passau, Germany Dublin City University, Ireland INPG, France University of Southampton, UK Blekinge Tekniska H¨ogskola, Sweden Open University, UK Universit`a di Torino, Italy University of Klagenfurt, Austria Ghent University, Belgium University of Geneva, Switzerland IMATI-GE / CNR, Italy GTI-UAM, Spain MITRE, USA Universitat Pompeu Fabra, Spain ITI, Greece Universitat Ramon Llull, Spain IMATI-GE/CNR, Italy CWI, The Netherlands City University of Hong Kong, Hong Kong TU Eindhoven, The Netherlands VUA, The Netherlands University of Aberdeen, UK Northwestern University, USA Tallinn University of Technology, Estonia Open Universiteit Nederland, The Netherlands Queen Mary, UK Universidad Autonoma de Madrid, Spain NII, Japan University of Koblenz-Landau, Germany

Organization

Nicu Sebe Elena Simperl Alan Smeaton Cees Snoek Michela Spagnuolo Steffen Staab Vojtech Svatek Nadja Thalmann Raphael Troncy Giovanni Tummarello Vassilis Tzouvaras Remco Veltkamp Paulo Villegas Doug Williams Marcel Worring Li-Qun Xu Rong Yan

University of Amsterdam, The Netherlands STI Innsbruck, Austria Dublin City University, Ireland University of Amsterdam, The Netherlands IMATI-GE/CNR, Italy University of Koblenz-Landau, Germany University of Economics Prague, Czech Republic University of Geneva, Switzerland CWI, The Netherlands DERI, Ireland NTUA, Greece Utrecht University, The Netherlands Telefonica R&D, Spain BT, UK University of Amsterdam, The Netherlands British Telecom, UK IBM, USA

Additional Reviewers Rabeeh Ayaz Abbasi Jinman Kim Francesco Robbiano Jinhui Tang Xiao Wu

Organizing Institution

University of Koblenz-Landau, Germany University of Geneva, Switzerland IMATI Institute at CNR, Italy National University of Singapore, Singapore City University of Hong Kong, Hong Kong

IX

X

Organization

Supporting Organizations and Projects

Table of Contents

Keynote Talk: Mining the Web 2.0 for Improved Image Search . . . . . . . . . Ricardo Baeza-Yates

1

Keynote Talk: More than a Thousand Words . . . . . . . . . . . . . . . . . . . . . . . . Stefan R¨ uger

2

Content Organization and Browsing A Simulated User Study of Image Browsing Using High-Level Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Teerapong Leelanupab, Yue Feng, Vassilios Stathopoulos, and Joemon M. Jose Exploring Relationships between Annotated Images with the ChainGraph Visualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Steffen Lohmann, Philipp Heim, Lena Tetzlaff, Thomas Ertl, and J¨ urgen Ziegler On the Co¨ operative Creation of Multimedia Meaning . . . . . . . . . . . . . . . . . Claudio Cusano, Simone Santini, and Raimondo Schettini

3

16

28

Annotation and Tagging I On the Feasibility of a Tag-Based Approach for Deciding Which Objects a Picture Shows: An Empirical Study . . . . . . . . . . . . . . . . . . . . . . . Viktoria Pammer, Barbara Kump, and Stefanie Lindstaedt Statement-Based Semantic Annotation of Media Resources . . . . . . . . . . . . Wolfgang Weiss, Tobias B¨ urger, Robert Villa, Punitha P., and Wolfgang Halb Large Scale Tag Recommendation Using Different Image Representations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Rabeeh Abbasi, Marcin Grzegorzek, and Steffen Staab Interoperable Multimedia Metadata through Similarity-Based Semantic Web Service Discovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Stefan Dietze, Neil Benn, John Domingue, Alex Conconi, and Fabio Cattaneo

40 52

65

77

XII

Table of Contents

Content Distribution and Delivery Semantic Expression and Execution of B2B Contracts on Multimedia Content . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . V´ıctor Rodr´ıguez-Doncel and Jaime Delgado

89

A Conceptual Model for Publishing Multimedia Content on the Semantic Web . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Tobias B¨ urger and Elena Simperl

101

CAIN-21: An Extensible and Metadata-Driven Multimedia Adaptation Engine in the MPEG-21 Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fernando L´ opez, Jos´e M. Mart´ınez, and Narciso Garc´ıa

114

Annotation and Tagging II Shot Boundary Detection Based on Eigen Coefficients and Small Eigen Value . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Punitha P. and Joemon M. Jose

126

Shape-Based Autotagging of 3D Models for Retrieval . . . . . . . . . . . . . . . . . Ryutarou Ohbuchi and Shun Kawamura

137

PixGeo: Geographically Grounding Touristic Personal Photographs . . . . . Rodrigo F. Carvalho and Fabio Ciravegna

149

Short Papers Method for Identifying Task Hardships by Analyzing Operational Logs of Instruction Videos . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Junzo Kamahara, Takashi Nagamatsu, Yuki Fukuhara, Yohei Kaieda, and Yutaka Ishii Multimodal Semantic Analysis of Public Transport Movements . . . . . . . . Wolfgang Halb and Helmut Neuschmied

161

165

CorpVis: An Online Emotional Speech Corpora Visualisation Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Charlie Cullen, Brian Vaughan, John McAuley, and Evin McCarthy

169

Incremental Context Creation and Its Effects on Semantic Query Precision . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Alexandra Dumitrescu and Simone Santini

173

OntoFilm: A Core Ontology for Film Production . . . . . . . . . . . . . . . . . . . . . Ajay Chakravarthy, Richard Beales, Nikos Matskanis, and Xiaoyu Yang

177

Table of Contents

RelFinder: Revealing Relationships in RDF Knowledge Bases . . . . . . . . . . Philipp Heim, Sebastian Hellmann, Jens Lehmann, Steffen Lohmann, and Timo Stegemann Image Annotation Refinement Using Web-Based Keyword Correlation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ainhoa Llorente, Enrico Motta, and Stefan R¨ uger

XIII

182

188

Automatic Rating and Selection of Digital Photographs . . . . . . . . . . . . . . . Daniel Kormann, Peter Dunker, and Ronny Paduschek

192

Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

197

Keynote Talk: Mining the Web 2.0 for Improved Image Search Ricardo Baeza-Yates Yahoo! Research Barcelona http://research.yahoo.com

There are several semantic sources that can be found in the Web that are either explicit, e.g. Wikipedia, or implicit, e.g. derived from Web usage data. Most of them are related to user generated content (UGC) or what is called today the Web 2.0. In this talk we show how to use these sources of evidence in Flickr, such as tags, visual annotations or clicks, which represent the the wisdom of crowds behind UGC, to improve image search. These results are the work of the multimedia retrieval team at Yahoo! Research Barcelona and they are already being used in Yahoo! image search. This work is part of a larger effort to produce a virtuous data feedback circuit based on the right combination many different technologies to leverage the Web itself.

T.-S. Chua et al. (Eds.): SAMT 2009, LNCS 5887, p. 1, 2009. c Springer-Verlag Berlin Heidelberg 2009 

Keynote Talk: More than a Thousand Words Stefan R¨ uger Knowledge Media Institute The Open University http://kmi.open.ac.uk/mmis

This talk will examine the challenges and opportunities of Multimedia Search, i.e., finding multimedia by fragments, examples and excerpts. What is the stateof-the-art in finding known items in a huge database of images? Can your mobile phone take a picture of a statue and tell you about its artist and significance? What is the importance of geography as local context of queries? To which extent can automated image annotation from pixels help the retrieval process? Does external knowledge in terms of ontologies or other resources help the process along?

T.-S. Chua et al. (Eds.): SAMT 2009, LNCS 5887, p. 2, 2009. c Springer-Verlag Berlin Heidelberg 2009 

A Simulated User Study of Image Browsing Using High-Level Classification Teerapong Leelanupab, Yue Feng, Vassilios Stathopoulos, and Joemon M. Jose University of Glasgow, Glasgow, G12 8RZ, United Kingdom {kimm,yuefeng,stathv,jj}@dcs.gla.ac.uk

Abstract. In this paper, we present a study of adaptive image browsing, based on high-level classification. The underlying hypothesis is that the performance of a browsing model can be improved by integrating highlevel semantic concepts. We introduce a multi-label classification model designed to alleviate a binary classification problem in image classification. The effectiveness of this approach is evaluated by using a simulated user evaluation methodology. The results show that the classification assists users to narrow down the search domain and to retrieve more relevant results with respect to less amount of browsing effort.

1

Introduction

The accumulation of large volumes of multimedia data, such as images and videos, has led researchers to investigate indexing and search methods for such media in order to render them accessible for future use. Early, Content Based Image Retrieval (CBIR) systems were solely based on low-level features extracted from images inspired by developments in image processing and computer vision [3]. Nevertheless, due to the “semantic gap” [13] problem, using just low-level descriptors will not lead to an effective image retrieval solution. Recent research in multimedia indexing has investigated automatic annotation methods to index multimedia data with keywords which convey the semantic content of media. Those keywords cannot however represent all aspects of image content due to inherent complexity of multimedia data. In both of the above cases in multimedia retrieval, the search paradigm is similar and inspired by traditional information retrieval systems. The searcher poses a query to the system, which can be a rough sketch: a predicate query such as “images with at least 80% blue” or a textual query, and then the system returns a ranked list of potentially relevant images. An alternative search paradigm that suits better to the nature of multimedia data, and especially images, is browsing. A well studied browsing approach is to visualize retrieved images as a graph where nodes are images and paths are relationships between them based on some underlying similarity. Browsing is facilitated by allowing users to browse the collection by following paths in this graph (e.g. [5,6]). In this approach, relevance feedback and the Ostensive Model of developing information needs can be easily integrated [15]. T.-S. Chua et al. (Eds.): SAMT 2009, LNCS 5887, pp. 3–15, 2009. c Springer-Verlag Berlin Heidelberg 2009 

4

T. Leelanupab et al.

Browsing models are inherently different than traditional image retrieval systems since the focus is not on user queries but in the user’s browsing path where implicit feedback is provided. It is therefore difficult to see how automatically extracted keywords can be utilized to improve image browsing. Although query by keywords has been shown to be able to improve performance over low-level similarity in image retrieval especially when both methods are combined [10], no study has been performed on integrating high-level classification applied to annotate images on browsing models to the best of our knowledge. The underlying assumption of this integration is that the browsing effectiveness will be enhanced. Motivated by this we aim at answering the following research questions: – How to integrate high-level semantic concepts to a browsing model where user queries are limited and search is based on user’s implicit feedback? – Can we improve the response of a browsing model by using high-level classification? That is, can we reduce the number of clicks a user followed, and consequently the time spend browsing, in order to find relevant images? Browsing systems are interactive search systems that require user intervention and therefore user experiments are required to evaluate such methods. However, there are several strategies for integrating high-level classification to the browsing model and each has to be evaluated separately; hence, evaluating them requires a large scale user experiment which is expensive and time consuming. In this paper we propose an evaluation methodology based on simulating user actions by exploiting log files of user interactions with the system from previous user experiments [9]. Simulated evaluation can be used as the first step before performing an actual user study while ensuring a fair comparison between different methods. Once an appropriate methodology is found to perform reasonably well using this simulated methodology, a user experiment can be carried out so as to validate the validity of approach. The main contributions of our paper are: – We integrate high-level semantic concepts using multi-label classification to our browsing model. – We evaluate the effect of high-level semantics by using a simulated evaluation methodology by exploiting user logs. – We show that semantic similarity can improve browsing performance by reducing the time spent by a user browsing in order to find relevant images. The rest of this paper is structured as follows. In Section 2, we give a short survey of related works in current image browsing systems and present their inadequacies. Section 3 introduces an approach of integrating high-level concepts to a browsing model. Section 4 presents the experimental design and measures of our study. The result of our experiments is detailed in Section 5. Finally, we conclude our work in Section 6.

2

Image Browsing

A graph-based representation of retrieved images has been well studied to assist users in accessing their image collection. Heesch [4] surveys related work

A Simulated User Study of Image Browsing Using High-Level Classification

5

Fig. 1. Screenshot of Image Browsing Interface

in several browsing models in content based image retrieval. For instance, NNk networks introduced in [5] provides a graph-based structure by ranking images under the metric parametrised in terms of feature-specific weights. Torres et al. [14] propose the Spiral and Concentric Rings technique to visualise query results by placing the query image in the centre and fills a spiral line with the similar retrieved images. Similar work was carried out by Urban et al. [15] where the intentionality of user’s information needs is represented as nodes in a graph. In this approach, users browse through an image collection via retrieved images visualised in the graph. User’s interactions on the images, such as clicks, is considered as relevance feedback, which is then used to expand a search query. This approach employs the Ostensive Model of developing information needs [2] to adaptively tailor the search query to retrieve other similar images related to the leave of the graph users click on. The Ostensive Model reformulates a temporal dimension of interacted information objects to the notion of probabilistic relevance using different ostensive profiles. We adopt Urban’s approach by implementing the multi-aspect based image browsing system as introduced by Leelanupab et al. [9]. This system is employed as a baseline system to investigate our assumption on integrating high-level semantic concepts in our study. Figure 1 shows the screenshot of the interface, which can be divided into two main vertical panels. The left panel consists of: Full View tab (A) and Relevant Result tab (B). Dragging and dropping images to the Full View (A) tab will display a full size visualisation of the image, accompanied with its textual descriptions. In our user evaluation, users are expected to browse an image collection to find relevant images for a given search scenario and store them into the result panel (B). The right panel is the Browsing Panel (C) containing independent browsing sessions, visualised as tabs. It is hypothesised that each tab/session represents different aspects of search topics given by user’s interests to support complex

6

T. Leelanupab et al.

search tasks as suggested by Villa et al. [18]. On the browsing panel, a user, for instance, selects an image (1) considered as a node of this graph. Similar images will be shown as leaves of this node. Selecting one of these leaves (i.e. image (2)) will implicitly provide relevance feedback, with which the system incorporates the Ostensive Model to expand the search query. This model considers the iteration when feedback was provided by decaying the relevance of feature extracted from objects interesting to a user against the time of interacting with him. It is suggested that lower weighting should be given to earlier iterations since the user has most likely narrowed down his search interest in later few iterations [15]. As a result, the subsequent information in the path is assumed to be more relevant to the user. Here, browsing sessions can be initiated by selecting images from a keyword search or other browsing sessions. At the top right of the frame, a Switching Mode button (D) is provided in order to offer the user the option to change search methods between traditional keyword search and adaptive browsing. Although this approach has been proved to retrieve more relevant information [15], it still relies on retrieval performance which is based on features extracted from objects.

3

Incorporating High-Level Semantics

Current existing image browsing systems are mostly driven by low-level visual cues (e.g. [5,15,17]); however, there is the semantic gap problem, which is basically the disparity between low-level features and high-level semantics. Use of low level features cannot give satisfactory retrieval results in many cases, especially when the high level concepts in the user’s mind are not easily expressible in terms of low level features. Thus, the extraction of visual concepts can be regarded as a bridge between the low level features and high level semantic concepts to improve the retrieval performance. In order to fill this gap, a considerable amount of research into classification methods [16] has been done in particular since the classification can be assigned as a translator to bridge the low-level features with the semantic concepts by classifying images into different categories based on its similarity with each category. Furthermore, most of the classification methods are focusing on binary classifiers, which only classifies the data into one of two classes. One of the most famous methods in binary classifier is the Support Vector Machine (SVM). The key advantage of the SVM is that it seeks to fit an optimal hyperplane between classes and may require only a small training sample. However, using the binary classifier on image data may bring the problems associated when an image may belong to more than one class in semantics. For example, an image of natural scene taken during a trip to the Highlands might belong to outdoor, and nature classes rather than either of them. Motivated by the existing needs in CBIR, we have optimised the merits of the above retrieval models to build our retrieval framework. We apply a classification method on a browsing model to support explorative search tasks by semantics.

A Simulated User Study of Image Browsing Using High-Level Classification

3.1

7

High-Level Classification

A multi-label classification technique is employed to alleviate a binary classification problem on SVM. We defined a small set of six generic concepts paired for three classifiers that create three class labels for each image since our underlying idea is to define classes which will be suitable for all images instead of specific ones. Using specific classes might cause the difficulties in classification since the system needs a large number of classes to describe an image collection. In addition, using large number of specific classes will result in the degradation in the accuracy and efficiency. The concept groups are defined based on the nature of the database. A SVM based image classification method is employed to learn the visual concepts from the training set, and then applied on the testing images to give the concept labels. Spatial Features for Concept Detection – A number of existing works [12] have stated that the most efficient way for human beings to identify an image is from coarse to fine. Thus, different images can be classified into different scene concept groups based on their coarse scene information [11]. For instance, images of man-made scenes are characterised by geometry of vertical and horizontal structure: urban outdoor scenes will have more vertical edges, with less in indoor scenes. Considering the possibility of extracting concept information via scene characters, we develop a concept based image classification by using scene characteristic features. The scene characteristic features are computed in the frequency domain using the Gabor filter [7]. SVM-based Concept Detector Training –The original SVM is designed for binary classification. In our case, we have six pre-defined image classes, resulted in a multi-class problem. We use the following method to reduce a set of binary problems. First, a set of binary classifiers, each of which was trained to separate one class from the rest, is built. In another word, n hyperplanes are constructed, where n is the number of pre-defined classes. Each hyperplane separates one class from the others. In this experiment, three pairs of classifiers were defined in order to classify three pairs of generic concepts, such as indoor/outdoor, nature/man-made, and portrait/crowd, where different concepts can overlap in an image. As a result, a total combination of those classifiers can form eight (23 ) different categories. For instance, the classification result for one image can be represented as 011, where 0 means that the image is classified into the 1st category of each pair and 1 means into the 2nd category. It is our intention to measure the effectiveness of integrating such high-level classes for image browsing models. 3.2

Using Semantic Concepts on Image Browsing System

In the browsing system, the classification will work as follows. First, the classification is applied to compute the category of images in the experimental collection

8

T. Leelanupab et al.

so that the raw image data in term of low-level features can be transferred into high level concept. Next, given images in a browsing path selected by the user, the retrieval algorithm takes these browsing images as a query and searches for similar images only within the same category as labelled in the collection. Note that every image in the path belongs to only one of eight categories based on three pairs of pre-defined classes with the underlying assumption that high-level concepts will increasingly improve retrieved results in each browsing iteration. This approach exploits user’s feedback of selecting images as query to specifically browse an image collection within the same category he or she is interested in. The user will browse an image collection based not only on low-level features, but also on semantic concepts.

4

Experiment

In this section, we detail our experimental setup. First, we outline two browsing systems used in the evaluation. A detail of a data collection is described in the next section. We present our method to obtain user interactions to simulate users acting on systems, followed by discussing task information which affects the experimental results. We finally describe a strategy which simulates users’ browsing behaviour. 4.1

System Description

There are two image browsing systems used in this evaluation: a baseline system that can enhance a simple search query using adaptive browsing, and a proposed system that extends the standard browsing system with high-level classification. Both systems have the same interface as shown in Figure 1 and share the same retrieval back–end, which uses textual and visual features as well as an “Ostensive Model” [2] as adaptive retrieval model. The Terrier IR system1 was used for removing stop-word, stemming, indexing in textual retrieval. Okapi BM 25 was used to rank retrieval results. Importantly, to support visual queries, three MPEG7 image features have been extracted for image dataset: Colour Layout, Edge Histogram, and Homogeneous Texture. The weight of visual and textual features are equally balanced in retrieval. The proposed system employed highlevel classification, which classifies the image collection into sub-categories using multi-label classification. 4.2

Data Collection

Our aim of this study is to assume the role of a real user, browsing his/her collection. Therefore we employed a real user collection, called CoPhIR2 , for SVM training and experiment. The current collection contains 106 million images derived from the Flickr3 archive. For training classification, we asked three 1 2 3

http://ir.dcs.gla.ac.uk/terrier/ http://cophir.isti.cnr.it/ http://www.flickr.com/

A Simulated User Study of Image Browsing Using High-Level Classification

9

multimedia information retrieval experts to manually classify 200 sample images for each class according to six pre-defined concepts. We selected the other subset of estimated 20000 images taken by unique users between 1 October 2005 and 31 March 2006 as an experimental collection. This time period was selected since it covers the highest density of images from unique users. The text used for keyword search is derived from titles, descriptions, and tags given by Flickr users. 4.3

Mining User Interactions

There has been no precedent set of the methods to best evaluate a IR system. A system-oriented evaluation based on the Cranfield model is unsuitable to evaluate interactive search systems due to its adaptive, cognitive, and behavioural features of the environment where interactive systems perform. Borlund [1] proposed an alternative approach, called user-centred evaluation to evaluate interactive systems. This approach is very helpful to obtain valuable data on the behaviour of users and interactive search systems. Nevertheless, such methodology is inadequate to benchmark various underlying adaptive retrieval algorithms because it is expensive in terms of time and repeatability. Another alternative means of evaluating such systems is the use of simulated user evaluation to assume a role of users to trigger search queries and browse retrieved results. There are two possible ways to run a simulated user study. One is to use a test collection, similar to the Cranfield method, to mimic user’s query formulation, browsing and relevance assessment [19]. However, this method requires ground truth data, which are hard to generate for a large collection and has a limitation of diversity in queries per topic. The other one is to create a pool of user’s interactions derived from an actual user study to generate a range of search strategies [8]. Since we have no ground truth data and browsing is a complex activity based on individual users, we therefore adopted the simulated user study by using the log-files of a prior user experiment [9]. If such a user was available, he or she would similarly do a set of actions that, in their opinion, would increase the chance of retrieving more relevant documents for given search topics. Our objective is to find out whether the effectiveness of the browsing system modelled using high-level classification would have improved. We mined 4 types of user’s interactions based on the nature of task exploration reflecting their judgement on retrieved images as shown in Table 1: (1) Table 1. 24 Users’ interaction statistics Interactions Topic # Queries # Browses # T1 151 258 T2 316 274 T3 180 231 T4 153 351

Results Sessions # Total # Two or More % 104 397 115 29.0 114 215 83 38.6 95 254 76 29.9 125 377 104 27.6

10

T. Leelanupab et al.

“# Queries”, a list of textual queries executed to get a potential set of images to start browsing such as “wild animals”, “endangered birds” etc. (2) “# Browses”, a list of clicked images to be used for browsing further, called “Browse Images” in this paper; (3) “# Sessions”, a list of chosen images to start new browsing sessions, referred to as “Session Images”; and (4) “Results”, a list of relevant images added to a relevance list in total and when two or more users selected with its percentage, named as “Total” and “Two or More” respectively in the sub-column. Our underlying assumption of each user interaction is that users clicked on browse images from a set of retrieved images when they found them at most closely relevant and capable of leading them to see more relevant images. Users selected session images when they found them relevant and show different aspects of search topics. Users added images into a result list when they found them relevant to their information needs. 4.4

Search Task Information

For this evaluation we aimed at simulating browsing patterns and search strategies of 24 users performing four tasks in the prior user study [9]. All fours tasks were explorative search topics which gave users broad indicative requests and asked them to discover different aspects of images used in many various simulated situations as suggested by Borlund [1]. Tasks 1 – 4 entitled “Find different aspects of wild living creatures, “Find different aspects of vehicles”, “Find different aspects of natural water”, and “Find different aspects of open scenery” respectively. After completing each task, the users were asked to describe their experiences related to tasks and a system in questionnaires. The questionnaires disclosed that user perceived Task 2 and 3 as the most difficult tasks, followed by Task 4 and 1. Accordingly, the total number of relevant images retrieved from Task 2 and 3 is lower than for the other tasks as shown in Table 1. Another reason supporting this is the task complexity for given topics resulted in the nature of the collection. Table 1 illustrates the level of task complexity using the number of user agreement on selecting relevant images. As Table 1 shows, the percentage of relevant images selected by two or more users is higher in Task 2 and 3, which indicates Task 2 and 3 may be “narrower” than Task 1 and 4. Assuming that there will be less agreement amongst users for broader tasks, which require a greater extent of interpretation. According to the questionnaires, the retrieval results found, and the level of specification, T2 and T3 might be more complicated and difficult than T1 and T4, which requires more interpretation. Consequently, this factor may influence the results of our simulation which will be discussed later. 4.5

A Browsing Strategy

We devised a browsing strategy to repeat user interactions to answer our research questions based on four lists of interaction types mined from the previous study: textual query, browsing, session, and result/relevance lists. The strategy was aimed at following user interactions to decide what the next action will be performed and then updating relevance results. We use a hypothetical component called the Agent or simulated user who controls the flow of interactions

A Simulated User Study of Image Browsing Using High-Level Classification

11

with two browsing systems. We recorded all the results and actions performed by the agent. Our simulation procedure uses the following steps. First, the agent submits a textual search query by randomly selecting a textual query in a textual query list and then the systems return a list of top nine images. The agent interacts with these images by matching images appearing in a session list to start new browsing sessions. Note that each interaction data is used only once for each task. If there are two or more session images found, they will be put into a session queue according to their ranking in that query. The session queue follows a First-In, First-Out (FIFO) pattern which serves the image found first to start a new browsing session. In a next step, the agent selects one of the retrieved images to further browsing. There are two options to select the images. The agent chooses the images existing either in a relevance list or in a browsing list. One difference is that if the agent found any images on a relevance list, they will also be added to a relevance results. At this step, the agent can only select one image in order to simplify our browsing strategy. If there were two or more images found on any of two lists, the agent will take the image in the relevance list first, indicating higher relevance based on user judgement. In case of two or more images found from the same list, an agent will select the one ranked higher. During browsing, the systems retrieved six candidates to the agent in each search iteration. Whenever the agent cannot match retrieved images with any images in the three lists, he will start to create a new session from the first image in the session queue or to re-enter a new query in a textual query list when the queue is empty. Moreover, if the agent found images existing in a session list during browsing, those images will be added to the session queue. The browsing simulation will end up whenever the agent completely found all relevant images in the relevance list or performed all search queries in the query list. Following the given browsing strategy, we separately performed simulation runs on all four tasks over baseline and proposed systems.

5

Results

This section presents the results of our experiments based on the research questions stated in Section 1. Table 2 shows the comparison of the experimental results. We used a total of 4 search tasks per system in our analysis. We denote “A” by the baseline system, whereas “B” stands for the proposed system. The “# Iterations” means the total number of iteration that an agent stops browsing. ¯ and “SD” show the mean and standard deviation of relevant images, re“X” trieved in each iteration. To measure the statistical significance of the results, we applied T-Test (parametric) to the difference between the baseline and proposed systems. All tests were paired to carry out a one-tailed test and critical value (p-value) was set to 0.05, unless otherwise stated. “A>B” represents the number of browsing iterations, where System “A” retrieved more relevant images than a system “B” in each iteration, followed by its percentage. “A which are triples where a concept can be set in relation to other concepts. Using statements with semantic elements is a compromise in complexity between loose and fully semantically described annotations. [Fig. 2] illustrates statements with an example Image from the tiny planets3 universe. To create such statements, three different input options are available as shown in [Fig. 3]: (1) combining concepts via drag-and-drop, (2) selecting concepts consecutively and (3) using the text box as a command line interface in the spirit of [8] with auto-completion. Input option three is optimally suited for frequent users and input options one and two are ideal for users who rarely create annotations. 3

http://www.mytinyplanets.com/

56

W. Weiss et al.

Fig. 2. Example of Annotation Statements

Fig. 3. Creation of Annotation Statements

Fig. 4. Creation of relationships

Statement-Based Semantic Annotation of Media Resources

57

An additional possibility to annotate the content of media resources in the IMAS system is to relate them to each other. Hereby, we can describe that one media resource is, for instance, a part, a revision, or a transformation of another media resource. This allows us to use statements of the related media resources, to keep track of revisions of the media resources or to suggest alternatives in the search result. The relationship ontology with its properties is described subsequently in the section 3.2. To create relationships (see also [Fig. 4]) of selected media resources (1), the user drags a source media resource from the file browser (2) and drops it on the desired relationship (e.g. the revision) of the relationship panel (3). This action creates the following annotation (4): < SourceResource is a revision of TargetResource >.

3.2

Ontologies

To represent the semantic annotations we developed two ontologies according our needs. The first one is the SALERO annotation ontology (c.f. [Fig. 5]), for describing media resources, authors of media resources, projects in which they are used, and annotations. The property annotea:related, derived from the

Fig. 5. The SALERO annotation ontology

58

W. Weiss et al.

Annotea Annotation Schema4 , is used to describe the annotation statements. The subclasses of Annotation and AnnotationConcept form the domain specific parts of the ontology, which have to be adapted if someone wants to use the annotation tool in a different domain. The subclasses of Annotation describe media specific annotation types such as annotations of audio, image or video. The subclasses of AnnotationConcept represent the domain-specific part of the annotation ontologies and are used to describe the content of media resources. They currently include the following classes: – Character: The actors of a movie, e.g. Bing and Bong. – CharacterParts: Parts of the body of the characters, e.g. Hand, Nose, or Toe. – Expression: Includes verbs and adjectives to describe the behaviour of characters, e.g. smiling, dancing, open, or wet. – Location: A geographical or imaginary location, e.g. Barcelona or Outer Space. – Object: A tangible or visible entity, e.g. Balloon, Umbrella, or Cake. The scope of the second ontology is to describe relationships between media resources and how they are most probably derived or based on each other. The relationships of the Functional Requirements for Bibliographic Records (FRBR)5 model provide a solid ground to describe the relationships of media resources in the media production domain in general and in the SALERO project in particular. The relationships are supposed to enhance the browsing experience of media professionals in their media collections. The relationships of our ontology were chosen based on a general analysis of the domain based on related work, an analysis of images from the MyTinyPlanets collection, and a set of expert interviews. It contains a subset of the FRBR core ontology6 and two additional relationships, which were not covered by the FRBR core ontology (frbr: http://purl.org/vocab/frbr/core#) (c.f. [9,10]): – frbr:adaptation. A media resource, which is based on another media resource, exchanges parts of the original content, e.g. in a new image a tree is exchanged by a space ship. – frbr:alternate. An alternative (file) format of the same content, e.g. jpg instead of png. – frbr:imitation. A real world scene is imitated in a cartoon, e.g. a scene of “Star Wars” is imitated by Bing & Bong in “Tiny Planets”. – frbr:part. A media resource contains a part of another media resource, e.g. the hands of the character Bing are a part of the final animation of Bing. Supposing that the hands are modelled in a different file. – frbr:reconfiguration. A rearrangement of the content of a media resource, e.g. a new scene is based on an existing scene with the same content such as trees, space ships, characters. In the new scene the content is locally rearranged. 4 5 6

http://www.w3.org/2000/10/annotation-ns http://www.ifla.org/VII/s13/wgfrbr/index.htm http://vocab.org/frbr/core.html

Statement-Based Semantic Annotation of Media Resources

59

– frbr:revision. A newer or other version of a media resource, e.g. a new version of a space ship, which is based on an existing space ship, uses a different texture for its surface. – frbr:transformation. From a sketch to a (3-d) model, e.g. the first step in creating a 3-d model is to draw a sketch on a sheet of paper. The next step is to create a 3-d model in the computer, which is a transformation of the sketch. – frbr:translation. A translation into a different language, e.g. the embedded text and audio of a clip are translated from English to French. – duplicate. For duplicated media resources, e.g. the same file is stored at a different location. – version. The super property for frbr:adaption, frbr:reconfiguraiton, frbr:revision and frbr:transformation. The property relationships is the super property for all relationship properties, its range and domain is MediaResource. 3.3

Usability Test

An initial evaluation of the annotation aspect of the IMAS is already done. Our aim was (i) to find major obstacles in the annotation user interface as well as in the annotation process and (ii) to compare the semantic statement-based approach with other existing approaches. Evaluation Methodology: We recruited 9 participants of our institute who were not involved in the project or in the realm of semantic annotation. The subjects ranged in age from 25 to 40 and all are software developers. We created two groups. The first group of five participants compared the annotation process of the IMAS application with the desktop application of Google Picasa7. In the second group we did chronometries of creating annotations with (i) IMAS, (ii) a free text tagging approach, similar to flickr8 and (iii) creating fully semantic annotations with PhotoStuff [11]. For later analysis we did screen captures of the user actions and conspicuous behaviour was noted by the observer. Furthermore, the usability test consisted of a short user questionnaire where the participants had to answer following questions: – What are the positive and negative impressions of each tool? – What was the most difficult task? – Which tool would you use, if you have to annotate 1000 images? Before the test began, each subject got an introduction in the usability test and into the different annotation applications. Then the users had to annotate the test data with each application. The test data consisted of ten images from the tiny planets9 universe together with descriptive text for each image. For example, 7 8 9

http://picasa.google.com/ http://www.flickr.com/ http://www.mytinyplanets.com/

60

W. Weiss et al.

the descriptive text for [Fig. 2] is: “Bing has a bag on his shoulder and a book in his hands. Bong and alien are smiling.” Results: The success rate to complete the annotation tasks with IMAS is 93%. The reasons why not all participants where able to successfully complete all tasks are (i) user interface and design problems and (ii) that the semantic statementbased approach was not clear for every participant and thus produced wrong or incomplete annotations. A successful example, done by a participant with IMAS, includes following statements for the image [Fig. 2]: – – – –

Bing is related to: Shoulder bag, Shoulder Bing is related to: Book, Hand Bong is related to: smiling Alien is related to: smiling

and following tags created with Google Picasa: “alien, bing, bong, book, hand, shoulder bag on his shoulder, smiling”. These examples describe the content according the descriptive text of the image well and fulfils our quality requirements for a successful annotation.

Task Completion Times

180

160

140

time (s)

120

100

80

60

40

20

0 Tag Tool

IMAS

PhotoStuff

Fig. 6. Comparison of task completion times

[Fig. 6] shows the time measurements for completing the tasks where the statement-based approach is compared to a free text tagging approach and to creating fully semantic annotations with PhotoStuff. Creating annotations with PhotoStuff was the most time consuming approach (median 60s; mean 69.9s for

Statement-Based Semantic Annotation of Media Resources

61

creating the annotations of a single media resource). The subjects complained that they have to do long-winded recurrent tasks, such as selecting concepts and manually creating instances. The fastest approach was to use the simple free text tagging approach (median 19s; mean 18.6s), although the subjects claimed to use a system with auto completion feature. Task completion time using IMAS with the statement-based approach ranks between the two other systems with a median of 30s and a mean of 36.3s. We observed that most subjects firstly used the concept tables (see also [Fig. 3] and the section 3.1) and after annotating approximately three media resources the subjects tended to use only the command line interface with auto completion. The questionnaire revealed following facts: The users liked the auto completion feature which demonstrates following user statement: “I highly appreciate that the system suggests concepts to me I can use.” In the subjects opinion this feature helps efficiently to create specific suitable annotations. Furthermore, this was a crucial reason why for 8 out of 9 participants IMAS is the first choice for annotating 1000 media resources. One participant prefers to use a simple free text tagging approach. On the other hand, the participants also revealed new suggestions to improve the annotation process: “A copy and paste functionality of concepts and statements would be fine.” “Existing annotations should be editable.” The users also complained that the text box does not work as expected in some situations, e.g. when pressing the space bar to select a concept.

4

Related Work

The organisation, classification, and retrieval of media objects is an ongoing challenge in games and media production. Semantic technologies have been identified as a viable solution to overcome some of the problems in this area [2]. A wide range of multimedia annotation tools [12,13] already offer functionality to attach ontological annotations to (parts) of the multimedia content and some offer reasoning services on top of them to semi-automatically create annotations based on existing annotations. The K-Space Annotation Tool [14] provides a framework around the Core Ontology for Multimedia for efficient and rich semantic annotations of multimedia content. PhotoStuff [11] allows to use any ontology for the annotation of images and is available as a standalone desktop application. A Web-based demonstrator for browsing and searching with very limited functionality is also available. Imagenotion [15] already provides an integrated environment for the collaborative semantic annotation of images and image parts. User tests showed that the use of standard ontologies and tools is not generally suitable, which led to the development of a method where ontologies consist of imagenotions that graphically represent a semantic notion through an image.

62

W. Weiss et al.

A first step in closing the semantic gap, between low level signal processing and high level semantic descriptions, is to use a multimedia ontology infrastructure. The Core Ontology on Multimedia (COMM) [16] aims to provide a high quality multimedia ontology that is compatible with existing (semantic) Web technologies. The MPEG-7 based multimedia ontology enables to describe algorithms and digital data, decompositions, region locations, content and media annotations, semantic annotations and example annotations. The objective of the AIM@SHAPE [17] ontology for virtual humans is to provide a semantic layer to reconstruct, stock, retrieve and reuse content and knowledge of graphical representations of humans. The ontology allows to represent, amongst other things, shapes, parts of bodies, animations and emotions. The aim of the W3C Annotea [18] project is to enhance collaborations via shared metadata based on Web annotations, bookmarks, and their combinations. The project encourages users to create Annotea objects, including annotations, replies, bookmarks and topics. Therefore, a RDF schema can be used which defines all necessary properties.

5

Conclusion and Outlook

In this paper we have presented a semantic statement-based approach for fast and easy annotation of media resources in the realm of media productions. This concept is implemented in the Intelligent Media Annotation & Search10 application which also allows to create relationships of media resources. Creating annotations using statements with semantic elements is a compromise in complexity and expressiveness between loose and full semantical descriptions. We have developed two ontologies to store annotations and relationships. The system integrates semantic and content-based search to provide a fall-back and an alternative retrieval system. The initial usability test has shown that the approach of semantic statement-based annotation is not as fast as free text tagging but much faster than creating full semantic annotations with PhotoStuff. It is planned to explore a more sophisticated fusion mechanism for the search system. A thematic browsing through the content, based on the ontologies, would also be feasible. An important item is a further evaluation of IMAS, especially the search system in combination with the annotations, to determine the limitations of this approach.

Acknowledgements The research leading to this paper was partially supported by the European Commission under contract “IST-FP6-027122” (SALERO). Images from “My Tiny Planets” are by courtesy of Peppers Ghost Productions11 . 10 11

http://salero.joanneum.at/imas/ http://www.peppersghost.com

Statement-Based Semantic Annotation of Media Resources

63

The authors would like to thank their colleagues who contributed to the implementation of the components described here, especially Georg Thallinger, Gert Kienast, Philip Hofmair, Roland M¨ orzinger and Christian Ammendola.

References 1. B¨ urger, T., Ammendola, C.: A user centered annotation methodology for multimedia content. In: Poster Proceedings of ESWC 2008 (2008) 2. B¨ urger, T.: The need for formalizing media semantics in the games and entertainment industry. Journal for Universal Computer Science, JUCS (June 2008) 3. Ounis, I., Amati, G., Plachouras, V., He, B., Macdonald, C., Lioma, C.: Terrier: A High Performance and Scalable Information Retrieval Platform. In: Proc. of ACM SIGIR 2006 Workshop on Open Source Information Retrieval, OSIR (2006) 4. O’Connor, N., Cooke, E., le Borgne, H., Blighe, M., Adamek, T.: The ace-toolbox: low-level audiovisual feature extraction for retrieval and classification. In: 2nd IEE European Workshop on the Integration of Knowledge, Semantic and Digital Media Technologies, pp. 55–60 (2005) 5. Punitha, P., Jose, J.M.: Topic prerogative feature selection using multiple query examples for automatic video retrieval. In: Proceeding of the 32nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR). ACM, Boston (in press, 2009) 6. Punitha, P., Jose, J.M.: How effective are low-level features for video retrieval (submitted for review) (2009) 7. Weiss, W., B¨ urger, T., Villa, R., Swamy, P., Halb, W.: SALERO Intelligent Media Annotation & Search. In: Proceedings of I-SEMANTICS 2009 - International Conference on Semantic Systems, Graz, Austria (2009) 8. Raskin, A.: The Linguistic Command Line. In: ACM Interactions, January/February 2008, pp. 19–22 (2008) 9. IFLA Study Group: Functional requirements for bibliographic records, final report (February 2009), http://www.ifla.org/files/cataloguing/frbr/frbr_2008.pdf 10. Riva, P.: Introducing the functional requirements for bibliographic records and related ifla developments. In: ASIS&T The American Society for Information Science and Technology (August/September 2007) 11. Halaschek-Wiener, C., Golbeck, J., Schain, A., Grove, M., Parsia, B., Hendler, J.A.: PhotoStuff - An Image Annotation Tool for the Semantic Web. In: Poster Proceedings of the 4th International Semantic Web Conference (2005) 12. Obrenovic, Z., B¨ urger, T., Popolizio, P., Troncy, R.: Multimedia semantics: Overview of relevant tools and resources (September 2007), http://www.w3.org/2005/Incubator/mmsem/wiki/Tools_and_Resources 13. Simperl, E., Tempich, C., B¨ urger, T.: Methodologies for the creation of semantic data. In: Sicilia, M.A. (ed.) Handbook of Metadata, Semantics and Ontologies. World Scientific Publishing Co., Singapore (2009) 14. Saathoff, C., Schenk, S., Scherp, A.: KAT: The K-Space Annotation Tool. In: Proccedings of the SAMT 2008 Demo and Poster Session (2008) 15. Walter, A., Nagyp´ al, G.: IMAGENOTION - Collaborative Semantic Annotation of Images and Image Parts and Work Integrated Creation of Ontologies. In: Proceedings of 1st Conference on Social Semantic Web (CSSW), pp. 161–166 (2007)

64

W. Weiss et al.

16. Arndt, R., Troncy, R., Staab, S., Hardman, L., Vacura, M.: COMM: Designing a Well-Founded Multimedia Ontology for the Web. In: Aberer, K., Choi, K.-S., Noy, N., Allemang, D., Lee, K.-I., Nixon, L.J.B., Golbeck, J., Mika, P., Maynard, D., Mizoguchi, R., Schreiber, G., Cudr´e-Mauroux, P. (eds.) ASWC 2007 and ISWC 2007. LNCS, vol. 4825, pp. 30–43. Springer, Heidelberg (2007) 17. Garcia-Rojas, A., Vexo, F., Thalmann, D., Raouzaiou, A., Karpouzis, K., Kollias, S.: Emotional body expression parameters in virtual human ontology. In: Proceedings of 1st Int. Workshop on Shapes and Semantics, Matsushima, Japan, June 2006, pp. 63–70 (2006) 18. Koivunen, M.R.: Annotea and semantic web supported collaboration. In: Proceedings of UserSWeb: Workshop on End User Aspects of the Semantic Web. Heraklion, Crete (2005)

Large Scale Tag Recommendation Using Different Image Representations Rabeeh Abbasi, Marcin Grzegorzek, and Steffen Staab ISWeb - Information Systems and Semantic Web, University of Koblenz-Landau Universit¨ atsstrasse 1, 56070 - Koblenz, Germany {abbasi,marcin,staab}@uni-koblenz.de

Abstract. Nowadays, geographical coordinates (geo-tags), social annotations (tags), and low-level features are available in large image datasets. In our paper, we exploit these three kinds of image descriptions to suggest possible annotations for new images uploaded to a social tagging system. In order to compare the benefits each of these description types brings to a tag recommender system on its own, we investigate them independently of each other. First, the existing data collection is clustered separately for the geographical coordinates, tags, and low-level features. Additionally, random clustering is performed in order to provide a baseline for experimental results. Once a new image has been uploaded to the system, it is assigned to one of the clusters using either its geographical or low-level representation. Finally, the most representative tags for the resulting cluster are suggested to the user for annotation of the new image. Large-scale experiments performed for more than 400,000 images compare the different image representation techniques in terms of precision and recall in tag recommendation.

1

Introduction

With the explosive growth of Web and the recent development in digital media technology, the number of images on the Web has grown tremendously. Online photo services such as Flickr and Zooomr allow users to share their pictures with family, friends, and the online community at large. An important functionality of these services is that users manually annotate their pictures using so called tags, which describe their contents or provide additional contextual and semantical information. Tags are used for navigation, finding, and browsing resources and thus provide an immediate benefit for users. In practice, however, users often tag their pictures fully manually which is very time-consuming and therefore very inconvenient and expensive. For this reason, it is very important to automatize this process by developing the so called Tag Recommender Systems assisting users in the tagging phase. Although this research area has been now active for a couple of years [1,2,6,7], the existing recommendation strategies are preliminary and their performance for generic scenarios rather moderate. The basic idea of recommending tags for a new image is to reuse tags assigned to similar images which have been stored in T.-S. Chua et al. (Eds.): SAMT 2009, LNCS 5887, pp. 65–76, 2009. c Springer-Verlag Berlin Heidelberg 2009 

66

R. Abbasi, M. Grzegorzek, and S. Staab

the data collection before. One of the most challenging problems here is to find those similar images in a large-scale photo collection. Early approaches aimed at solving this image retrieval problem by using exclusively low-level features [13] which turned out to be almost impossible in generic environments at large-scale. Their performance was acceptable only for certain domain specific applications such as the content-based medical image retrieval [12]. Nowadays, the state-ofthe-art imaging devices provide pictures together with the geographical coordinates (geo-tags) stating precisely where they have been acquired. Therefore, more and more researchers make use of the additional information designing tag recommender systems with quite promising results [4,8]. The most of recent tag recommendation approaches combine different image description types (geo-tags, tags, low-level features) in order to achieve reasonable results [9,11]. However, one can observe a lack of research activities comparing the benefits each of these description types brings to a tag recommender system on its own. In our paper, we exploit these three kinds of image descriptions to suggest possible annotations for new images uploaded to a collaborative tagging system independently of each other. First, we cluster the existing largescale data collection separately for the geo-tags, tags, and low-level features. Additionally, we perform random clustering in order to provide a baseline for experimental results. Once a new image has been uploaded to the system, it is assigned to one of the clusters using either its geographical or low-level representation. Finally, the most representative tags for the resulting cluster are suggested to the user for annotation of the new image. Large-scale experiments performed for as many as 413, 848 images compare the different image representation techniques in terms of precision and recall in tag recommendation. The paper is structured as follows. Section 2 gives an overview about our tag recommendation system. In Section 3 the content description methods (features) used in social media are shortly explained, especially those used in our framework. Section 4 explains how generating image annotations works in our framework. In Section 5, we describe the dataset used for experiments and evaluate tag recommender systems of the architecture proposed at large-scale. The tests compare different image representation techniques in terms of precision and recall in tag recommendation. Section 6 concludes our investigations and their results presented in this paper.

2

System Overview

We split the overall system for tag recommendation into two parts: training and tag recommendation. The system is trained based on the image features available in social media, once the system is trained, it is used for recommending tags for new images. Following is the brief description of training and tag recommendation phases: Training: In the training phase images are clustered based on their features. A cluster contains homogeneous images depending upon the type of features

Large Scale Tag Recommendation Using Different Image Representations

67

used for clustering. For this research work, we considered geographical coordinates, low-level image features and tags as image features. As an example, a cluster based on geographical coordinates might represent the images taken in a particular location, a cluster based on low-level features might contain images showing buildings or a beach, and a cluster based on tagging data might represent concepts like concert or river. Clustering process used in this research work is described in section 4.1. Representative tags of a set of homogeneous images (i.e. images in a cluster) are used to annotate new images. The method of identifying representative tags is described in the section 4.2. Tag Recommendation: For recommending tags to a new image, we map the image to its closest cluster and assign the representative tags of the cluster to the new image. The method of classifying an image to its closest cluster and recommending tags are described in section 4.3. In the following section, we describe the features that we have used in our experiments and are also available in Folksonomies on a large scale.

3

Features in Social Media

To analyze the effect of different type of features on the performance of tag recommendation, we use three different image features in our experiments, namely Geographical Coordinates (G), Low-level image features (L), and Tags (T ). Following are the details of the features used in this research work. Geographical Coordinates: With the advancement in camera and mobile technologies, nowadays many devices are available in market that are able to capture the location of the image using a built-in or external device. In addition to the possibility of capturing location of an image using a GPS device, some folksonomies like Flickr facilitate the users to add geographical coordinates to their images by providing a map interface where users can place their images on the map. Due to this easiness, there are many images in Flickr which are enriched with geographical information. In the CoPhIR dataset [3], around 4 Million out of 54 Million images are annotated with geographical coordinates. The number of geographically annotated images is supposed to increase in future as more devices will be able to capture the geographical coordinates. We represent the geographical coordinates of the images in a two dimensional vector space G ∈ 2 . Each row vector gi of the feature space G represents the geographical coordinates of the image i. Low-level Image Features: There are five different types of low-level MPEG7 features available in the CoPhIR dataset for 54M images. Table 1 shows the properties and dimensions of the low-level features available in CoPhIR dataset. Based on initial experimental results, we consider two low-level features for evaluation, the MPEG-7 Edge Histogram Descriptor (EHD) and Color Layout (CL), which outperformed other available low-level image features. EHD represents the local edge distribution and CL represents the color and spatial information in

68

R. Abbasi, M. Grzegorzek, and S. Staab

Table 1. Properties and dimensions of low-level features available in CoPhIR dataset Low-level Feature Scalable Color Color Structure Color Layout Edge Histogram Homogeneous Texture

Properties Color histogram Localized color distributions Color and spatial information Local-edge distribution Texture

Dims 64 64 12 80 62

the images. We represent the low-level image features based on EHD and CL in 80 and 12-dimensional feature spaces LE ∈ 80 and LC ∈ 12 respectively. A row vector i of the feature space LE or LC represents the edge histograms or color layout of the image i respectively. Tags: Tags are freely chosen keywords associated with the images. There is no restriction in selecting a tag for an image. A tag might represent a concept in an image, describe the image itself or it might also represent the context of the image (e.g. location, event, time etc.). On average there are only a few tags associated with the images. In 54M images of the CoPhIR dataset, each images has on average 3.1 tags. We represent the tags of the images as a nt dimensional vector space T ∈ nt , where nt is the number of tags in the dataset. A row vector ti∗ of the vector space T represents a resource whose non-zero values represent the tags associated with the resource i. A column vector t∗j represents a tag vector whose non-zero values represent the resources associated with the tag j. A value tij represents the number of times resource i is associated with the tag j. The images in all feature spaces are indexed in the same order. For an image i, the row vector gi represents its geographical coordinates, li represents its low-level image features, and ti represents the tags associated with the same image i.

4

Tags Recommendation

This section explains the proposed tag recommendation system in detail. In the training phase of tag recommendation, the resources are first clustered (Sec. 4.1), then for each cluster, its representative tags are identified (Sec. 4.2). In the tag recommendation phase, a new resource is mapped to its closest cluster and the representative tags of the closest cluster are recommended for the new image (Sec. 4.3). 4.1

Clustering

Although many sophisticated clustering algorithms exist in literature, but the literature is still sparse for clustering high dimensional and large datasets. We use K-Means clustering algorithm in our experiments. K-Means is capable of

Large Scale Tag Recommendation Using Different Image Representations

69

Input: Feature space F ∈ {G, L, T, D}, Number of clusters k Output: A set of k clusters Method: 1. 2. 3. 4.

Randomly select k images from feature space F as the initial cluster centroids Assign each image to the closest cluster Update cluster centroids If cluster centroids are changed, then repeat step 2-3 Fig. 1. K-Means clustering algorithm

clustering very large and high dimensional datasets. Of course, other clustering methods can also be employed in the framework, when one desires to fine tune the performances or improve the results. The K-Means algorithm we used is described in figure 1. In the following, we describe in detail how do we set different parameters for using K-Means. Number of clusters: There is no generally accepted rule for setting the number of clusters for using K-Means. For our experiments we use the number of clusters as suggested by Mardia et al [10, page 365]. We define the number of clusters for n images as follows:  n k= (1) 2 By using k as defined in the above equation, we get same number of clusters for each feature space. Initial Cluster Centroids: In K-Means clustering, the quality of clustering also depends on the selection of initial cluster centroids. For our experiments, k images are randomly selected. The same set of randomly selected images are used as cluster centroids for each feature space. Selecting the same set of images for different feature spaces avoids accidental improvement of one feature space over an other based on the initial centroids. Computing distance/similarity between resources: During the clustering process, each image is assigned to its closest cluster (fig 1, step 2). We need a distance measure to compute the distance between an image and its closest centroid. The most popular distance measure used is Euclidean Distance [5, page 388]. Euclidean Distance between two m-dimensional vectors f and c is defined as follows:  m  euclidean(f , c) =  (fi − ci )2 (2) i=1

We use euclidean distance for non-text feature spaces (i.e. geographical, low-level, and random feature spaces). For text (or tags) based feature spaces it is common

70

R. Abbasi, M. Grzegorzek, and S. Staab

to use Cosine Similarity [5, page 397]. We use cosine similarity to compute similarity between image tags (in feature space T ) and cluster centroids. Cosine similarity between two m-dimensional vectors f and c is defined as follows: cosine(f , c) =

f T· c ||f ||||c||

(3)

Experimental results show that cosine similarity for tag/text based features performs significantly better than euclidean distance. For comparison between different distance measures, we also evaluated the results on Manhattan distance for non-text based features. There was no significant improvement in results if we use Manhattan distance. Manhattan distance between two vectors f and c is defined as follows manhattan(f , c) =

m 

|fi − ci |

(4)

i=1

4.2

Identifying Representative Tags

After clustering images into k clusters, we identify the representative tags for each cluster. The most representative tags of a cluster are recommended for the new image. To identify the representative tags of each cluster, we rank the tags by user frequency in descending order. The rank of a tag is higher if more users have used it and vice versa. We associate the top s tags to the cluster c, and represent the set of most representative top s tags associated with a cluster c as cT . 4.3

Classification and Tag Recommendation

Once we have clustered the images and identified representative tags of these clusters, we can recommend representative tags of the closest cluster from a new image. The image is mapped to the cluster, whose centroid is at minimum distance from the image. Most representative tags associated with the mapped cluster are assigned to the new image. We assume that we have the geographical coordinates and low-level features of the new image, but we do not have tags associated with the new image. In the case of clusters based on geographical or low-level feature space, we can directly measure the distance between the geographical or low-level features of the new image and the centroids of the clusters. But for tag based clustering, we do not have tags for the new image. Therefore we have to compute the centroids of tag based clusters in terms of either geographical or low-level features. For clusters based on geographical coordinates, we classify the new image to one of the clusters whose centroid is at minimum geographical distance from the new image. For low-level clusters, we classify the new image based on the distance between its low-level features and cluster centroids. For tag based clusters, as we do not have any tags for the new image, we classify the new image based on the distance between its geographical

Large Scale Tag Recommendation Using Different Image Representations

71

coordinates and the mean of geographical coordinates of the tag based clusters. The mismatch between feature spaces used for tag based clustering and the new image negatively effects the results of tags based clustering. To sum up the tag recommendation process, we list down the recommendation processes in three steps as follows: 1. Find closest cluster centroid c to the image f (use geographical mean as cluster centroid for geographical (G) and tag (T ) based clusters; and lowlevel mean as cluster centroids for clusters based low-level (L) features) 2. Recommend the tags cT associated with the cluster c to the new image

5

Experiments and Results

In this section the experiments and results are presented. The image dataset is briefly described in Section 5.1, the distinction between the training and the test data comes in Section 5.2, which is followed by the evaluation method in Section 5.3. Section 5.4 presents the comprehensive results achieved in our work. 5.1

Image Dataset

CoPhIR dataset [3] consists of images uploaded to Flickr by hundreds of thousands of different users, which makes the dataset very heterogeneous. One can find images of very different types like portraits, landscapes, people, architecture, screen shots etc. To perform an evaluation on different types of features (geo-tags, tags, low-level) on a reasonably large scale, we created a subset of the original CoPhIR dataset. We selected the images taken in national capitals1 of all the world countries. For this purpose, we considered all the images with Euclidean distance (in terms of latitude and longitude) from center of a capital city not higher than 0.1. We ignored the capital cities which had less than 1, 000 images; this resulted into a set of 58 cities. To keep the experiments scalable, we randomly selected 30, 000 images for cities which had more than 30, 000 images. There were only three such cities Paris, London, and Washington DC. In the end, we had images of 58 capital cities, ranging from 1, 000 to 30, 000 images with an average of 8, 000 images per city. Total number of images in our evaluation dataset was 413, 848. For scalability, particularly for low-level image features, images are trained and evaluated separately for each city. Base Line: In order to compare the effectiveness of different image features, we created a random feature space for the images. We assign a random value between 0 and 1 to each of the image in dataset as its random feature. We consider the random features as the baseline for comparison. Same clustering methods are applied on the random features as on the other features. Random feature space is uni-dimensional and is represented as D ∈ . 1

http://en.wikipedia.org/wiki/National_capitals

72

5.2

R. Abbasi, M. Grzegorzek, and S. Staab

Training and Test data

It is important to carefully select the training and test datasets, because when a user uploads images to Flickr, he can perform batch operations on the set of images. For example, he can assign same tags or geographical coordinates to all the images in a batch. It is also possible that the images have similar low-level features, e.g. if the images belong to a beach or a concert. If we randomly split the images into test and training datasets, there is a chance that some images belonging to a user are used for training, while other images of the same user are used in test dataset for evaluation. Such random split may effect the final results because a test image might be mapped to a cluster containing images from the same user, having similar features as the test image. It is very likely that the test image gets annotated with perfect tags, as tags of both test and training images were provided by the same user. To make the evaluation transparent, instead of randomly splitting the resources into training and test dataset, we split the users. For each city, we use resources of 75% users for training and resources of 25% users as test dataset. No image in the test dataset is annotated by a user who has also annotated images in the training dataset. After splitting the users into training and test datasets, we use 310, 590 images for training the system and 103, 258 images used as ground truth for evaluating the system. Another aspect of fair evaluation is the quality of the tags. There are some tags which are very common in both test and training datasets. These tags mostly represent city or country names, which can be suggested by looking into a geographical database. Some common tags might not be very specific, e. g., the tags geotagged, 2007, travel etc. Very common tags also effect the evaluation results, as they are abundant in both test and training datasets, and are almost suggested for every test image. This results into higher precision and recall values. To make the evaluation more transparent, we do not consider the ten most frequent tags for each city and we also ignore the frequent tags geotagged and geotag, because all the images in our dataset are geo-tagged and most of the images have these two tags. For each city, we also remove the very rare tags which might be incorrectly spelled tags or tags specific to a particular user. For this reason, for each city, we ignore those tags which are used by less than three users. 5.3

Evaluation

We consider the tags associated with the 103, 258 test images as ground truth. The images in the ground truth are tagged by different users and as there is no restriction on the selection of tags for a resource, therefore the tags in ground truth are very noisy. The noise in the data leads to inferior results, but the overall results show the comparative analysis of different feature spaces. We evaluate the methods using standard evaluation methods used in information retrieval: Precision P , Recall R, and F-Measure F . The evaluation measures are defined as follows:

Large Scale Tag Recommendation Using Different Image Representations

P = R= F =

Number of correctly suggested tags Number of suggested tags Number of correctly suggested tags Number of expected tags

2×P ×R P +R

73

(5) (6) (7)

In addition to the standard precision and recall measures, we also computed the macro precision Pm , macro recall Rm , and macro F-Measure Fm over tags as follows:  Pm =

# of times t correctly suggested

t∈Tags Suggested

# of tags suggested

 Rm =

Fm =

5.4

# of times t suggested

(8)

# of times t correctly suggested

t∈Tags Expected

# of times t expected

# of tags expected

2 × Pm × Rm Pm + Rm

(9)

(10)

Results

The results presented in this section give a comparative view of tag recommendation based on different types of features. The automated evaluation on one hand provides the possibility to do evaluation on a large scale, but on the other hand the ground truth (test data) might contain invalid tags. We try to make the evaluation transparent and more meaningful by filtering certain types of tags (see Section 5.2). By removing very common tags, there is a certain decrease in evaluation results, but we believe that filtering make the evaluation fair. We have also evaluated the results without filtering the dataset, and in that case even random feature space gives a F-Measure value of 0.42. This is because of the reason that very common tags are recommended for the test images and there is always a major overlap between common tags of training and test data. The precision, recall, and F-Measure values presented in this section might appear to be low for the reader, but one shall keep in mind the filtering applied on the dataset to make the evaluation transparent. Table 2 consists of nine charts (chi∈{1,2,3},j∈{1,2,3} ) presenting the experimental results. Charts in the first row (ch1,j∈{1,2,3} ) depict the so called micro average evaluation and were generated in accordance to the evaluation criteria (5), (6), and (7)

74

R. Abbasi, M. Grzegorzek, and S. Staab Table 2. Result charts (chi∈{1,2,3},j∈{1,2,3})

ch1,1 – Micro Precision

ch1,2 – Micro Recall

ch1,3 – Micro F-Measure

ch2,1 – Macro Precision

ch2,2 – Macro Recall

ch2,3 – Macro F-Measure

ch3,1 – Micro F-Measure comparing results of two different low-level features Edge Histogram Descriptor (EHD) and Color Layout (CL)

ch3,2 – Micro F-Measure comparison of Cosine (Cos) and Euclidean (Eucl) distances for tag/text based features

ch3,3 – Micro F-Measure comparison of Manhattan (Manh) and Euclidean (Eucl) distances for non-text based features. Dark lines show the results obtained using Manhattan distance and gray lines show results obtained using euclidean distance

respectively. As one can see, in all three cases the results are significantly better when using geo-tags for image description. The performance of the tag recommendation using low-level features and textual tags differs only slightly from the results based on random clustering. For exactly one tag being recommended, the precision amounts to: 0.1385 for geo-tags, 0.0502 for low-level features, 0.0451 for textual tags, and 0.0338 for random clustering. Charts in the second row (ch2,j∈{1,2,3} ) present the so called macro average over tags evaluation and were generated in accordance to the evaluation criteria

Large Scale Tag Recommendation Using Different Image Representations

75

(8), (9), and (10) respectively. Similar to the micro average evaluation, the results here are significantly better for geo-tags, while the performance in case of textual tags, low-level features, and random clustering is almost the same. For exactly one tag being recommended, the macro precision for geo-tags amounts to 0.1584, for low-level features - 0.0521, for textual tags - 0.0414, and the baseline is 0.0312. In the third row of charts (ch3,j∈{1,2,3} ) in Table 2 some further evaluations can be found. In the first and second row charts (chi∈{1,2},j∈{1,2,3} ) the Edge Histogram Descriptor (EHD) was applied whenever low-level features were used and cosine similarity was used for tag based feature space. This has got an experimental reason. As you can see in Chart (ch3,1 ), the EHD performs slightly better than the Color Layout (CL) in terms of micro F-Measure and chart (ch3,2 ) shows a clear advantage of the cosine distance over the euclidean distance for tag based features. And finally, Chart (ch3,3 ) explains why using the simple Euclidean distance has appeared to be sufficient in our approach. The results remain almost the same when using Manhattan distance.

6

Conclusion

In our paper, we exploited three kinds of image description techniques, namely geo-tags, tags, and low-level features, to suggest possible annotations for new images uploaded to a social tagging system. In order to compare the benefits each of these description types brings to a tag recommender system on its own, we investigated them independently of each other. The evaluation was done on a large-scale image database. For experiments we used the CoPhIR dataset [3] including images uploaded to Flickr by hundreds of thousands of different users. The processing chain of our algorithm for generating image annotations contains: (i) clustering the images, (ii) finding representative tags for the clusters, (iii) classification of new images and tag recommendation. The results showed that geo-tags are the most helpful image descriptors for tag recommendation, while textual tags and low-level features provide only a slightly better performance than the random baseline. In the future, we will keep investigating the tag recommendation problem for large-scale heterogeneous image archives. We will further develop our framework to allow comprehensive experimental studies. We will also investigate the problem for some more domain dependent data collections.

Acknowledgments This work has been partially supported by the European project Semiotic Dynamics in Online Social Communities (Tagora, FP6-2005-34721). We would also like to acknowledge Higher Education Commission of Pakistan (HEC) and German Academic Exchange Service (DAAD) for providing scholarship and support to Rabeeh Abbasi for conducting his PhD.

76

R. Abbasi, M. Grzegorzek, and S. Staab

References 1. Adrian, B., Sauermann, L., Roth-Berghofer, T.: Contag: A semantic tag recommendation system. In: Pellegrini, T., Schaffert, S. (eds.) Proceedings of I-Semantics 2007, September 2007, pp. 297–304. JUCS (2007) 2. Basile, P., Gendarmi, D., Lanubile, F., Semeraro, G.: Recommending smart tags in a social bookmarking system. In: Bridging the Gap between Semantic Web and Web 2.0 (SemNet 2007), pp. 22–29 (2007) 3. Bolettieri, P., Esuli, A., Falchi, F., Lucchese, C., Perego, R., Piccioli, T., Rabitti, F.: CoPhIR: a test collection for content-based image retrieval. CoRR, abs/0905.4627v2 (2009) 4. Cristani, M., Perina, A., Castellani, U., Murino, V.: Content visualization and management of geo-located image databases. In: CHI 2008: CHI 2008 extended abstracts on Human factors in computing systems, pp. 2823–2828. ACM, New York (2008) 5. Han, J., Kamber, M.: Data Mining: Concepts and Techniques, 2nd edn. Morgan Kaufmann, San Francisco (2006) 6. Heymann, P., Ramage, D., Garcia-Molina, H.: Social tag prediction. In: SIGIR 2008: Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval, pp. 531–538. ACM, New York (2008) 7. J¨ aschke, R., Marinho, L., Hotho, A., Schmidt-Thieme, L., Stumme, G.: Tag recommendations in folksonomies. In: Kok, J.N., Koronacki, J., Lopez de Mantaras, R., Matwin, S., Mladeniˇc, D., Skowron, A. (eds.) PKDD 2007. LNCS (LNAI), vol. 4702, pp. 506–514. Springer, Heidelberg (2007) 8. Kennedy, L., Naaman, M., Ahern, S., Nair, R., Rattenbury, T.: How flickr helps us make sense of the world: context and content in community-contributed media collections. In: MULTIMEDIA 2007: Proceedings of the 15th international conference on Multimedia, pp. 631–640. ACM, New York (2007) 9. Kennedy, L.S., Naaman, M.: Generating diverse and representative image search results for landmarks. In: WWW 2008: Proceeding of the 17th international conference on World Wide Web, pp. 297–306. ACM, New York (2008) 10. Mardia, K., Kent, J., Bibby, J.: Multivariate Analysis. Academic Press, London (1979) 11. Mo¨ellic, P.-A., Haugeard, J.-E., Pitel, G.: Image clustering based on a shared nearest neighbors approach for tagged collections. In: CIVR 2008: Proceedings of the 2008 international conference on Content-based image and video retrieval, pp. 269– 278. ACM, New York (2008) 12. M¨ uller, H., Michoux, N., Bandon, D., Geissbuhler, A.: A review of content-based image retrieval systems in medical applications - clinical benefits and future directions. International Journal of Medical Informatics 73(1), 1–23 (2003) 13. Pentland, A., Picard, R., Sclaroff, S.: Tools for content-based manipulation of image databases. International Journal of Computer Vision 18(3), 233–254 (1996)

Interoperable Multimedia Metadata through Similarity-Based Semantic Web Service Discovery Stefan Dietze1, Neil Benn1, John Domingue1, Alex Conconi2, and Fabio Cattaneo2 1

Knowledge Media Institute, The Open University, MK7 6AA, Milton Keynes, UK {s.dietze,n.j.l.benn,j.b.domingue}@open.ac.uk 2 TXT eSolutions, Via Frigia 27, 20126 Milano, Italy {alex.conconi,fabio.cattaneo}@txt.it

Abstract. The increasing availability of multimedia (MM) resources, Web services as well as content, on the Web raises the need to automatically discover and process resources out of distributed repositories. However, the heterogeneity of applied metadata schemas and vocabularies – ranging from XML-based schemas such as MPEG-7 to formal knowledge representation approaches – raises interoperability problems. To enable MM metadata interoperability by means of automated similarity-computation, we propose a hybrid representation approach which combines symbolic MM metadata representations with a grounding in so-called Conceptual Spaces (CS). In that, we enable automatic computation of similarities across distinct metadata vocabularies and schemas in terms of spatial distances in shared CS. Moreover, such a vector-based approach is particularly well suited to represent MM metadata, given that a majority of MM parameters is provided in terms of quantified metrics. To prove the feasibility of our approach, we provide a prototypical implementation facilitating similarity-based discovery of publicly available MM services, aiming at federated MM content retrieval out of heterogeneous repositories. Keywords: Semantic Web Services, Multimedia, Metadata, Vector Spaces.

1 Introduction A continuously increasing amount of digital multimedia (MM) content is available on the Web, ranging from user-generated video content, commercial Video on Demand (VoD) portfolios to a broad range of streaming and IPTV resources and corresponding metadata records [19]. Besides, it became common practice throughout the last decade, to expose all sorts of MM content and metadata stored in one particular repository through a set of Web services, which provide Web-based access to software functionalities processing MM content and metadata, i.e. to retrieve, transcode or scale MM assets [21]. In line with the increasing usage of the term Web service in a broader sense, in the following we will use it synonymous with any kind of software functionality which is accessible through HTTP or any other IP-based layer, ranging from rather light-weight APIs, REST-based interfaces or standard Web service technology such as SOAP [22], UDDI [23] and WSDL [24]. T.-S. Chua et al. (Eds.): SAMT 2009, LNCS 5887, pp. 77–88, 2009. © Springer-Verlag Berlin Heidelberg 2009

78

S. Dietze et al.

Hence, the increasing accessibility of distributed MM resources – content as well as services – raises the need to automatically discover and compose distributed content. In that, the highly heterogeneous nature of MM resources distributed across distinct repositories leads to the following key challenges: C1. C2.

Discovery of distributed MM services. Discovery of distributed MM content.

However, w.r.t. these goals, several issues apply: Concurrent metadata schemes and vocabularies. Distinct approaches to metadata representation do exist, ranging from light-weight tagging approaches as deployed within user-driven websites such as youtube1 and general-purpose metadata standards such as Dublin Core [5] to fully-fledged domain-specific metadata standards such as MPEG-7 [10]. Besides, concurrent vocabularies – differing in terminology, syntax or language - are widely used to provide metadata records leading to further heterogeneities and ambiguities [11][19]. This issue also applies to Web service metadata provided based on syntactic descriptions such as WSDL [24] or semantic annotations based on OWL-S [12] or WSMO [25]. Lack of metadata comprehensibility and semantic meaningfulness. Metadata records lack expressivity due to merely syntactic annotations – usually based on XML schemas – not exploiting semantics of used structures and terminologies [1] [9][20]. In addition, current MM metadata schemas usually focus on the low-level parameters describing the actual format and audio-visual characteristics of MM assets, although a combined representation of both the actual content as well as its audio-visual format is required [14]. Moreover, even approaches such as [18] which exploit formal semantic representations, e.g. based on Semantic Web (SW) technologies such as OWL2 or RDF-S3, rely on either the common agreement on a shared conceptualisation or the formal representation of mappings, what is costly and error-prone. These issues hinder the automatic composition and processing of MM metadata and resources, and hence, do lead to interoperability issues. Lack of rather fuzzy matchmaking approaches. Current approaches to match between a certain request and available MM resources usually perform strict one-to-one matchmaking and require the subscription to a certain vocabulary from both providers and consumers. In that, only resources from a highly limited number of repositories which represent an exact match with the requested parameters are retrieved, while similar and otherwise related resources which potentially are useful are being left aside. Consequently, in order to enable interoperability between heterogeneous MM resource metadata, representation approaches are required which are meaningful enough to implicitly infer about inherent similarities across concurrent sets of MM annotations. In previous work [4], the authors proposed a representational approach combining symbolic knowledge representation mechanisms – as used by current MM resource metadata approaches – and SW technologies, with a representation in so-called Conceptual Spaces (CS) [8]. The latter consider the representation of 1

http://www.youtube.com http://www.w3.org/OWL/ 3 http://www.w3.org/RDFS/ 2

Interoperable Multimedia Metadata

79

knowledge entities, such as the ones described in MM metadata, through geometrical vector spaces where measurable quality criteria represent individual dimensions. Particular metadata records, i.e. instances, are represented as members, i.e. particular vectors, in a CS what facilitates computation of similarities by means of spatial distances. Here, we propose the application of our hybrid representational approach to model metadata of MM resources – i.e. MM services and MM content – in order to enable computation of similarities across heterogeneous repositories. In particular, low-level audio-visual MM characteristics, which are usually described by means of quantified attributes based on certain metrics, such as the MPEG-7 [10] descriptors Dominant Color or Homogeneous Texture, lend themselves to being represented in terms of vectors. Consequently, our hybrid representational approach appears to be well suited and hence, qualifies well to tackle MM metadata interoperability. The remaining paper is organized as follows. We provide an overview on related work in the area of MM service and content metadata intereoperability in Section 2. Our approach to represent MM metadata is introduced in Section 3 followed by a prototypical application utilising our approach for similarity-based MM resource discovery in Section 4. Section 5 concludes and discusses our work.

2 Related Work To satisfy the content need of a specific consumer, a federated MM content provisioning engine needs to discover (C1), the appropriate MM services (i.e. repositories) and (C2), the appropriate content. The following figure depicts this vision: C1. Discovery of MM services Client

MM Services Web service

MM Content Repository

MM Content Repository

Web service

Web service

MM Content Repository

MM Content Repository

MM Content C2. Discovery of MM content

Fig. 1. Discovery of distributed MM services and content

Given that both MM services and content utilize particular metadata vocabularies and schemas, approaching C1 and C2 requires taking into account related works from the areas of MM service metadata as well as MM content metadata interoperability. 2.1 MM Service Discovery through Semantic Web Services With respect to C1, Semantic Web Services (SWS) technology aims at the automatic discovery, orchestration and invocation of distributed services on the basis of

80

S. Dietze et al.

comprehensive semantic descriptions. SWS are supported through representation standards such as WSMO [25] and OWL-S [12]. We particularly refer to the Web Service Modelling Ontology (WSMO), an established SWS reference ontology and framework. WSMO is currently supported through dedicated reasoners, such as the Internet Reasoning Service IRS-III [2] and WSMX [26], which act as broker environments for SWS. In that, a SWS broker mediates between a service requester and one or more service providers. Based on a client request, the reasoner disovers potentially relevant SWS, invokes selected services and mediates potential mismatches. However, the domain-independent nature of SWS reference models requires their derivation to facilitate the representation of certain domain-specific contexts. While SWS aim at automatic discovery of distributed Web services based on semantic metadata, current approaches usually rely on either the subscription to a common vocabulary and schema – i.e. a common domain ontology – or the manual definition of mappings between distinct service ontologies. In that, the previously introduced issues (Section 1) also apply to SWS technologies, demanding for approaches to deal with heterogeneities between distributed SWS. In that, approaches such as [15] aim at addressing the interoperability issue partially by resolving heterogeneities based on mapping approaches. For instance, [27] provides an attempt to support similarity detection for mediation within SWS composition by exploiting syntactic similarities between SWS representations. However, it can be stated that current approaches rely on the definition of a priori mappings, the agreement of a shared vocabulary or the exploitation of semi-automatic ontology mapping approaches. Hence, providing a more generic solution to automatically resolve heterogeneities between heterogeneous SWS remains a central challenge. 2.2 MM Metadata Interoperability With respect to C2, a broad variety of research aims at interoperability between distributed MM (content) metadata. In general, the need for enriching non-semantic MM metadata through formal semantics is widely accepted [16] to enable more comprehensive query and retrieval facilities. For instance, [19] proposes an approach to semantically enrich MPEG-7 and TV-Anytime metadata through formal semantic expressions. In addition, [18] provides a way of formally expressing semantics of MPEG-7 profiles. While increasing the expressiveness of MPEG-7 based metadata, this work is limited to MPEG-7 exclusively. This also applies to the work proposed in [7], which provides an OWL expression of the MPEG-7 information model. In [9], the author provides a core ontology to annotate MM content to address interoperability. However, this approach relies on the subscription to a common vocabulary/schema – i.e. the suggested core ontology – what is not feasible in Web-scale scenarios. An entirely MPEG-21 based approach for interoperable MM communication is proposed in [17]. The need to automatically discover and compose Web services to enable processing of MM content is expressed in [21], where the authors propose an approach based on OWL-S. However, the interoperability issues between heterogeneous symbolic service annotations (Section 2.1) also apply here. While several approaches try to tackle MM metadata interoperability, it can be stated that the current state of the art usually relies on subscription to common (upperlevel) vocabularies/schemas or the manual definition of mappings. Hence, issues arise

Interoperable Multimedia Metadata

81

when attempting to apply such approaches in Web-scale scenarios [11]. Therefore, analogous to the field of MM service annotations (Section 2.1), we claim that methodologies are required which allow for a more flexible alignment of distinct vocabularies.

3 Approach With respect to the previously introduced issues (Sections 1 and 2), we claim that basing MM metadata representations on merely symbolic representations does not fully enable semantic meaningfulness [4] and hence, limits automatic identification of similarities across distinct schemas and vocabularies. In order to enable interoperability between heterogeneous MM resource metadata – representing MM content or services – representation approaches are required which are semantically meaningful enough to implicitly infer about inherent similarities. In that, we argue that a refinement of symbolic MM metadata through so-called Conceptual Spaces (CS) is better suited to overcome interoperability issues. While previous work [3][4] has shown that this approach can be applied to support interoperability between ontologies, here we apply it to facilitate interoperability between MM services and repositories. 3.1 Grounding MM Metadata in Multiple Conceptual Spaces We propose a two-fold representational approach – combining MM domain ontologies with corresponding representations based on multiple CS – to enable (a) similarity computation across concurrent MM metadata schemas and vocabularies and (b) the conjoint representation of low-level audio-visual features and the content semantics. In that, we consider the representation of a set of n schema entities (concepts) E of a set of MM metadata records (ontology) O through a set of n Conceptual Spaces CS. Note, that we particularly foresee the application of this approach to metadata of both MM services as well as content. Schema entities in the case of MM content are, for instance, the MPEG-7 descriptors such as Scalable Color or Edge Histogram. In the case of MM services, a schema entity could be for instance a WSMO ontology concept. MM metadata values (instances) are represented as members, i.e. vectors, in the respective CS. While still benefiting from implicit similarity information within a CS, our hybrid approach allows maintaining the advantages of symbolic MM metadata representations and comprehensive domain ontologies, i.e. the ability to represent arbitrary relations and axioms. In order to be able to refine and represent ontological concepts within a CS, we formalised the CS model into an ontology [4]. Hence, a CS can simply be instantiated in order to represent a particular MM metadata schema entity. Referring to [8], we formalise a CS as a vector space defined through quality dimensions di. Each dimension is associated with a certain metric scale, e.g. ratio, interval or ordinal scale. To reflect the impact of a specific quality dimension on the entire CS, we consider a prominence value p for each dimension [8]. A particular member M – representing a particular value of a schema entity – in the CS is described through a vector defined by valued dimensions vi. Following this vision, for instance the MPEG-7 schema entity Dominant Color could be represented through a CS defined by means of RGB values, where each of the spectrum colors

82

S. Dietze et al.

represents one particular dimension of the CS. A certain shade of blue would then be represented through a member M1, i.e. a vector with M1={(124, 177, 236)}. Alignment between symbolic MM metadata representations and their corresponding CS (members) is achieved by referring the respective symbolic representation to the corresponding CS ontology containing the respective CS and member instances. In that, ontological MM metadata representations would import the CS ontology, while XML-based metadata could utilise a XML serialization of the CS ontology which is utilized as a particular controlled vocabulary. Hence, content and service semantics which are represented through particular domain ontologies are refined through CS to enable similarity-computation between distinct metadata sets. 3.2 Similarity-Based Discovery of MM Resources We define the semantic similarity between two members of a CS as a function of the Euclidean distance between the points representing each of the members. Hence, with respect to [4], given a CS definition CS and two members V and U, defined by v v0, v1, …,vn and u1, u2,…,un within CS, the distance between V and U is calculated as a normalised function of their Euclidean distance. For further details, please refer to [3][4]. In order to facilitate automated similarity computation between distinct MM metadata vocabularies and schemas, we provided a Web service (WSsim) capable of computing similarities between multiple members in multiple CS. This Web service enables to automatically identify similarities between multiple MM metadata records, and hence, to automatically select the most appropriate (i.e. the most similar) MM metadata record for a given request. In that, given a set of MM metadata records, for instance based on formal semantics or XML, and a set of corresponding CS representations which refine the MM metadata schema and its values by means of vectors, WSsim is able to compute similarities and consequently, to map and mediate between concurrent metadata schemas and vocabularies. This Web service is provided with the actual MM metadata request R and the x MM metadata records MMi that are potentially relevant for R: R ∪ {MM 1 , MM 2 ,..., MM x } . R is provided as a set of measurements, i.e. vectors {v1..vn} representing a set of m Members M(R) in available CS, which describe the desired metadata values, e.g. values measuring a certain MPEG-7 descriptor or the certain criteria describing MM Web service capabilities, such as a specific Quality of Service (QoS). Also, each MMi contains a set of concepts (schema entities) C={c1..cm} and instances (entity values) I={i1..in}. For each Mi within R the corresponding CS representations CS={CS1..CSm} are retrieved by WSsim from the available CS ontology [4]. Similarly, for each MMj members M(MMi) – which refine the instances of MMj and are represented in one of the conceptual spaces CS1..CSm, – are retrieved: CS ∪ M ( R ) ∪ {M ( MM 1 ), M ( MM 2 ),..., M ( MM x )} . Based on the above ontological descriptions, for each member vl within M(R), the Euclidean distances to any member of all M(MMj) which is represented in the same space CSj as vl are computed. Consequently, a set of x sets of distances is computed as Dist(MMi)={Dist(R,MM1), Dist(R,MM2) .. Dist(R,MMx)} where each Dist(R,MMj) contains a set of distances {dist1..distn} where any distk represents the distance between one particular member vi of R and one member refining one instance of the

Interoperable Multimedia Metadata

83

capabilities of MMj. Hence, the overall similarity between the request R and any available MMj could be defined as being reciprocal to the mean value of the individual distances between all instances of their respective capability descriptions:

(

Sim( R, MM j ) = Dist ( R, MM j )

)

−1

⎛ n ⎞ ⎜ ∑ ( dist k ) ⎟ k = 1 ⎟ =⎜ ⎜ ⎟ n ⎜ ⎟ ⎝ ⎠

−1

Finally, a set of x similarity values – computed as described above – which each indicates the similarity between the request R and one of the x available MM records MMj is computed by WSsim: Output (WS sim ) = {Sim ( R, MM 1 ), Sim ( R, MM 2 ),.., Sim ( R, MM x )} . As a result, the most similar MMj, i.e. the closest MM record, can be selected and invoked. In order to ensure a certain degree of overlap between the actual request and the selected MM record, we also defined a threshold similarity value T which determines the minimum similarity which is required.

4 Application – Similarity-Based Selection of Video Services We provided a prototypical implementation which aims at similarity-based retrieval of public MM content. Note, that instead of applying the representational approach to individual MM content metadata, our prototypical application utilizes our approach to annotate MM (Web) services which operate on top of distributed MM content repositories. The available services were annotated following the representational approach proposed in Section 3.1. Hence, our proof-of-concept application facilitates similarity-based selection of MM services (i.e. C1 in Section 1), which in turn process and retrieve MM content (C2). In that, federated retrieval and processing of MM metadata is supported to facilitate interoperability. Our application makes use of standard SWS technology based on WSMO and IRS-III (Section 2.1) to achieve this vision. The application dynamically discovers services which had been created in the context of the EC-funded project NoTube4 and make use of the Youtube-API5 as well as data feeds provided by BBC- Backstage6 and Open Video7. 4.1 Representing MM Services through Multiple CS In fact, five different Web services had been provided, each able to retrieve content from distinct repositories through keyword-based searches. WS1 is able to retrieve content from the Youtube channel of The Open University8, while WS2 provides Youtube content associated with the entertainment category following the Youtube vocabulary. WS3 performs keyword-based searches on top of the Open Video repository, while WS4 operates on top of the news metadata feeds provided by BBC Backstage. In addition, WS5 provides Youtube content suitable for mobiles. 4

http://projects.kmi.open.ac.uk/notube/ http://code.google.com/intl/en/apis/youtube/ 6 http://backstage.bbc.co.uk/ 7 http://www.open-video.org/ 8 http://www.youtube.com/ou 5

84

S. Dietze et al. SWS6: get-video-request M6 ={v1, v2}

M6 ={v1, v2, v3}

2

1

CS1 Purpose Space O1:Purp

O1:Env

O2:Purp

O2:Env

O3:Purp

CS2 Environment Space O3:Env

SWS1: OU-youtube

SWS2: entertain-youtube

SWS3: open-video

WS1: OU-youtube

WS2: entertain-youtube

WS3: open-video

O4:Purp O4:Env . SWS :

O5:Purp O5:Env . SWS :

bbc-backstage

mobile-youtube

WS4: bbc-backstage

WS5: mobile-youtube

4

5

Fig. 2. MM service metadata refined in two distinct CS

Based on the SWS reference model WSMO, we provided service annotations following the approach described in Section 3. In particular, we annotated the Web services in terms of the purpose they serve MM content for and the technical environment supported by the delivered content. In that, a simplified space (CS1: Purpose Space in Figure 3) was defined to refine the notion of purpose by using three dimensions: {((p1*information), (p2*education), (p3*leisure))} = CS1. The dimensions of CS1 are measured on a ratio scale ranging from 0 to 100. For instance, a member P1 in CS1 described by vector {(0, 100, 0)} would indicate a rather educational purpose. In addition, a second space (CS1: Environment Space in Figure 2) was provided to represent technical environments in terms of dimensions measuring the available resolution and bandwidth {((p1*resolution), (p2*bandwidth))} = CS2. For simplification, also the dimensions of CS2 were ranked on a ratio scale. However, it is intended to refine the resolution dimension to apply an interval scale to both dimensions to be able to represent actual resolution and bandwidth measurements. Each dimension was ranked equally with a prominence of 1 in all cases. By applying the representational approach proposed here, each concept of the involved heterogeneous SWS representations of the underlying MM services was refined as shared CS, while instances – used to define MM services and MM requests – were defined as members, i.e. vectors. In that, assumptions (Ass) of available MM services had been described independently in terms of simple conjunctions of instances which were individually refined as vectors in shared CS as shown in Table 1. Table 1. Assumptions of involved SWS (requests) described as vectors in MS1 and MS2 Assumption AssSWSi = ( P1SWSi ∪ P2 SWSi ∪ .. ∪ PnSWSi ) ∪ ( E1SWSi ∪ E2 SWSi ∪ .. ∪ EmSWSi ) Members Pi in CS1 (purpose)

Members Ej in CS2 (environment)

SWS1

P1(SWS1)={(0, 100, 0)}

E1(SWS1)={(100, 100)}

SWS2 SWS3 SWS4

P1(SWS2)={(0, 0, 100)} P1(SWS3)={(50, 50, 0)} P1(SWS4)={(100, 0, 0)} P1(SWS5)={(100, 0, 0)} P2(SWS5)={(0, 100, 0)}

E1(SWS2)={(100, 100)} E1(SWS3)={(100, 100)} E1(SWS4)={(100, 100)}

SWS5

E1(SWS5)={(10, 10)}

Interoperable Multimedia Metadata

85

Each service was associated with a set of members (vectors) in CS1 and CS2 to represent its purpose and the targeted environment. For instance, SWS3 which provides resources from the Open Video repository, which in fact are of rather educational or information nature, was associated with a corresponding purpose vector {(50, 50, 0)}. While SWS5 represents a Web service dedicated to MM content suitable for mobiles, a vector {(10,10)} indicating low resolution and bandwidth values was associated with SWS5. 4.2 Similarity-Based Selection of MM Services and Content An AJAX-based user interface (Fig, 3) was provided which allows users to define MM content requests by providing measurements describing their context, i.e. the purpose and environment, and search input parameters, i.e. a set of keywords. For instance, a user provides a request R with the search keyword “Aerospace” together with measurements which correspond to the following vectors: P1(R)={(60, 55, 5)} in CS1 and P2(R)=(95, 90)} in CS2. These vectors indicate the need for content which serves the need for education or information and which supports a rather high resolution environment. Table 2. Automatically computed similarities between request R and available SWS Similarities SWS1 SWS2 SWS3 SWS4 SWS5

0.023162405 0.014675636 0.08536871 0.02519804 0.01085659

Fig. 3. Screenshot of AJAX interface depicting MM metadata retrieved from the Open Video repository after similarity-based selection & invocation of MM services and metadata

86

S. Dietze et al.

Though no MM service matches these criteria exactly, at runtime similarities are calculated between R and the related SWS (SWS1-SWS5) through the similarity computation service WSsim described in Section 3.2. This led to the calculation of the similarity values shown in Table 2. Given these similarities, our reasoning environment automatically selects the most similar MM service (SWS3) and triggers its invocation. As illustrated above, our application utilises our representational mechanism (Section 3.1) to support similarity-based selection of distributed MM services. Hence, though just deploying our representational approach to MM services rather than MM content, our proof-of-concept prototype illustrates the applicability of our approach for similarity-based MM metadata discovery.

5 Conclusions In order to facilitate interoperability between heterogeneous MM resources distributed across distinct repositories, we identified two major challenges – the discovery of appropriate MM services and the retrieval of the most appropriate MM content. However, addressing these challenges requires interoperability between concurrent metadata annotation schemas and vocabularies. To facilitate such interoperability, we proposed a two-fold representational approach. By representing MM annotation schema entities as dedicated vector spaces, i.e. CS, and corresponding values as vectors, similarities are computable by means of distance metrics. Our approach is realised through a dedicated CS ontology. To prove the feasibility of our approach, we introduce a prototypical application which utilises our representational approach to support discovery of MM services across distributed MM repositories. As a result, we enable similarity-based discovery of the most appropriate MM service for a given request, and hence, enable federated MM content and metadata searches across distributed repositories. However, while the current matchmaking algorithm considers instance similarity as exclusive suitability measure, future work will deal with the combined consideration of logical expressions and instance similarity. We claim that our representational approach is particularly applicable to the domain of MM, where the majority of descriptors is based on quantified metrics, and hence, is well suited for metric-based representations such as vector spaces. The authors would like to highlight that providing the representations proposed here requires an additional effort, which needs to be investigated within future work. In this respect, please note that certain CS, for instance the one describing the notion of color, are reusable given that these are required for a variety of MM parameters. As another restriction, our approach foresees that distinct parties share common CS. However, given the wide-spread usage of upper-level ontologies such as DOLCE [6], SUMO [13] or OpenCyc9 together with availability of common MM metadata standards [10] and ontologies [1][18], the agreement on common CS becomes increasingly applicable. Future work will be concerned with the evaluation of the effort required to utilise our representational model, and also, with carrying out further case studies.

9

http://www.opencyc.org/

Interoperable Multimedia Metadata

87

References [1] Arndt, R., Troncy, R., Staab, S., Hardman, L., Vacura, M.: COMM: Designing a WellFounded Multimedia Ontology for the Web. In: Aberer, K., Choi, K.-S., Noy, N., Allemang, D., Lee, K.-I., Nixon, L.J.B., Golbeck, J., Mika, P., Maynard, D., Mizoguchi, R., Schreiber, G., Cudré-Mauroux, P. (eds.) ASWC 2007 and ISWC 2007. LNCS, vol. 4825, pp. 30–43. Springer, Heidelberg (2007) [2] Cabral, L., Domingue, J., Galizia, S., Gugliotta, A., Norton, B., Tanasescu, V., Pedrinaci, C.: IRS-III: A Broker for Semantic Web Services based Applications. In: Cruz, I., Decker, S., Allemang, D., Preist, C., Schwabe, D., Mika, P., Uschold, M., Aroyo, L.M. (eds.) ISWC 2006. LNCS, vol. 4273, pp. 201–214. Springer, Heidelberg (2006) [3] Dietze, S., Gugliotta, A., Domingue, J.: Conceptual Situation Spaces for SituationDriven Processes. In: Bechhofer, S., Hauswirth, M., Hoffmann, J., Koubarakis, M. (eds.) ESWC 2008. LNCS, vol. 5021, pp. 599–613. Springer, Heidelberg (2008) [4] Dietze, S., Domingue, J.: Exploiting Conceptual Spaces for Ontology Integration, Workshop: Data Integration through Semantic Technology (DIST 2008) Workshop at 3rd Asian Semantic Web Conference (ASWC 2008), Bangkok, Thailand (2008) [5] Dublin Core Metadata Initiative, Dublin Core Metadata Terms (2006), http://dublincore.org/documents/dcmi-terms/ [6] Gangemi, A., Guarino, N., Masolo, C., Oltramari, A., Schneider, L.: Sweetening Ontologies with DOLCE. In: Gómez-Pérez, A., Benjamins, V.R. (eds.) EKAW 2002. LNCS (LNAI), vol. 2473, p. 166. Springer, Heidelberg (2002) [7] Garcia, R., Celma, O.: Semantic Integration and Retrieval of Multimedia Metadata. In: 2nd European Workshop on the Integration of Knowledge, Semantic and Digital Media (2005) [8] Gärdenfors, P.: Conceptual Spaces - The Geometry of Thought. MIT Press, Cambridge (2000) [9] Hunter, J.: Enhancing the semantic interoperability of multimedia through a core ontology. IEEE Trans. Circuits Syst. Video Techn. 13(1), 49–58 (2003) [10] Moving Picture Experts Group, ISO/IEC JTC1/SC29/WG11 – MPEG-7, http://www.chiariglione.org/mpeg/standards/ mpeg-7/mpeg-7.htm [11] Nack, F., van Ossenbruggen, J., Hardman, L.: That Obscure Object of Desire: Multimedia Metadata on the Web (Part II). IEEE MultiMedia 12(1), 54–63 (2005) [12] OWL-S 1.0, http://www.daml.org/services/owl-s/1.0/ [13] Pease, A., Niles, I., Li, J.: The suggested upper merged ontology: A large ontology for the semanticweb and its applications. In: AAAI 2002 Workshop on Ontologies and the Semantic Web. Working Notes (2002) [14] Petridis, K., Bloehdorn, S., Saathoff, C., Simou, N., Dasiopoulou, S., Tzouvaras, V., Handschuh, S., Avrithis, Y., Kompatsiaris, I., Staab, S.: Knowledge Representation and Semantic Annotation of Multimedia Content. IEE Proceedings on Vision Image and Signal Processing, Special issue on Knowledge-Based Digital Media Processing 153(3), 255–262 (2006) [15] Radetzki, U., Cremers, A., Iris, B.: A framework for mediator-based composition of service-oriented software. In: ICWS, pp. 752–755. IEEE Computer Society, Los Alamitos (2004) [16] Stamou, G., van Ossenbruggen, J., Pan, J.Z., Schreiber, G., Smith, J.R.: Multimedia annotations on the semantic Web. IEEE Multimedia 13(1), 86–90 (2006)

88

S. Dietze et al.

[17] Timmerer, C., Hellwagner, H.: Interoperable Adaptive Multimedia Communication. IEEE Multimedia Magazine 12(1), 74–79 (2005) [18] Troncy, R., Bailer, W., Hausenblas, M., Hofmair, P., Schlatte, R.: Enabling Multimedia Metadata Interoperability by Defining Formal Semantics of MPEG-7 Profiles. In: Avrithis, Y., Kompatsiaris, Y., Staab, S., O’Connor, N.E. (eds.) SAMT 2006. LNCS, vol. 4306, pp. 41–55. Springer, Heidelberg (2006) [19] Tsinaraki, C., Polydoros, P., Kazasis, F., Christodoulakis, S.: Ontology-based Semantic Indexing for MPEG-7 and TV-Anytime Audiovisual Content. Special issue of Multimedia Tools and Application Journal on Video Segmentation for Semantic Annotation and Transcoding (2004) [20] Van Ossenbruggen, J., Nack, F., Hardman, L.: That Obscure Object of Desire: Multimedia Metadata on the Web (Part I). IEEE MultiMedia 11(4), 38–48 (2004) [21] Wagner, M., Kellerer, W.: Web services selection for distributed composition of multimedia content. In: Proceedings of the 12th annual ACM international conference on Multimedia, New York, NY, USA, October 10-16 (2004) [22] W3C: Simple Object Access Protocol, SOAP, Version 1.2 Part 0: Primer (2003), http://www.w3.org/TR/soap12-part0/ [23] W3C: Universal Description, Discovery and Integration: UDDI Spec Technical Committee Specification v. 3.0 (2003), http://uddi.org/pubs/uddi-v3.0.120031014.htm [24] W3C: WSDL: Web services Description Language (WSDL) 1.1 (2001), http://www.w3.org/TR/2001/NOTE-wsdl-20010315 [25] WSMO Working Group (2004), D2v1.0: Web service Modeling Ontology (WSMO). Working Draft (2004), http://www.wsmo.org/2004/d2/v1.0/ [26] WSMX Working Group, The Web Service Modelling eXecution environment (2007), http://www.wsmx.org/ [27] Wu, Z., Ranabahu, A., Gomadam, K., Sheth, A.P., Miller, J.A.: Automatic Composition of Semantic Web Services using Process Mediation. In: Proceedings of the 9th International Conference on Enterprise Information Systems (ICEIS 2007), Funchal, Portugal, June 2007, pp. 453–461 (2007)

Semantic Expression and Execution of B2B Contracts on Multimedia Content Víctor Rodríguez-Doncel and Jaime Delgado Distributed Multimedia Applications Group Universitat Politècnica de Catalunya Jordi Girona 1-3, 08034 Barcelona, Spain {victorr,jaime.delgado}@ac.upc.edu

Abstract. Business to business commerce of audiovisual material can be governed by electronic contracts, in the same way as digital licenses govern business to consumer transactions. The digital licenses for end users have been expressed either in proprietary formats or in standard Rights Expression Languages and they can be seen as the electronic replacement of distribution contracts and end user licenses. However, these languages fail to replace the rest of the contracts agreed along the complete Intellectual Property value chain. To represent their corresponding electronic counterpart licenses, a schema based on the standard eContracts and the Media Value Chain Ontology is presented here. It has been conceived to deal with a broader set of parties, to handle typical clauses found in the audiovisual market contracts, and to govern every transaction performed on IP objects. Keywords: MPEG-21.

Contract,

license, DRM,

Intellectual Property,

Ontology,

1 Introduction Perhaps audio and video distribution never received so much public attention as in these days. From the legal changes around the intellectual property (IP) to the newest gadget with new wondrous capabilities to appear in the market, mass consumption of media is nowadays in the spotlight of everybody. Less socially interesting but of no fewer economical importance are the transactions of multimedia material within the business to business (B2B) limits. Technologies to serve media to the consumer have developed very fast, in a controlled manner through web portals and Digital Rights Management (DRM) systems and in non controlled manners through parallel channels like P2P networks, fast download servers, etc. In the business to consumer (B2C) sector, economical transactions have been kept relatively simple –the consumer pays and in exchange can download a file or gain access to a football match streaming, for example. Digital licenses, expressed in one of the existing Rights Expression Languages (REL) allow some degree of complexity, where the transaction can be conditioned to the satisfaction of some conditions (e.g., of temporal or territorial nature), and can define more precisely the action the user can T.-S. Chua et al. (Eds.): SAMT 2009, LNCS 5887, pp. 89–100, 2009. © Springer-Verlag Berlin Heidelberg 2009

90

V. Rodríguez-Doncel and J. Delgado

make (perhaps render but not store nor print). In the B2B sector of audiovisual content, transactions happen in a similar way: there is flow of money and a flow of content in opposite directions, both of which can take place in the digital space. The quintessence remains the same as in the B2C case, but complexities arise in the conditions and the nature of the agreements, and a pre-filled license does not suffice. Written contracts regulate the economical transactions instead of digital licenses, and technology is not relied as it is in the retail segment. The authors of this paper believe that part of this lack of acceptance of digital systems to create, manage, and execute the agreements is due to the lack of maturity in the technology, which has failed to express satisfactorily the terms of real contracts in a digital language and manage and execute them accordingly. This paper recalls the previous attempts of expressing contracts in a digital language and their role as enforcement agents in information systems. It then particularizes to the case of the audiovisual B2B sector, which presents some recurrent patterns in the contract structure, and a very well defined candidate environment for their use: the DRM systems. The paper finally evaluates more complex contracts in the context of its execution and its role as steering documents.

2 Contract Representation 2.1 Contract Representation Overview B2B transactions in the audiovisual market have been regulated with narrative contracts. Contracts are legally binding agreements and they are made of mutual promises between two or more parties to do (or refrain from doing) something. The terms of a contract may be expressed written or orally, implied by conduct, industry custom, and law or by a combination of these. Efforts to represent contracts electronically are not new –they are as old as computers, and even making them part of digital systems is not new. Along with the development of computer sciences and network communications, the electronic representation of contracts played each time a more active role. Thus, in the earliest Electronic Data Interchange (EDI) standards, about fourty years ago, only bills and invoices were exchanged, but slowly the exchanged messages became richer in their expressivity and their role in an integrated information system was each time more important. Besides propietary systems where information adquired an ad-hoc structure there have been some remmarkable attemps to structure electronically the information in contracts. COSMOS [1] was an e-commerce architecture supporting catalogue browsing, contract negotiation and contract execution. It defined a contract model in UML and proposed a CORBA-based software architecture in a coherent manner. DocLog [2] was an electronic contract representation language introduced in the 2000 with a ‘XML like’ structure, which anticipated the next generation of XML-based contract representations. When XML was mature enough it was seen as a good container of contract clauses, and thus the new format specifications came under the form of a XML Schema or a DTD (Document Type Definition). An effort to achieve a common XML contract representation was the Contract Expression Language (CEL) [3],

Semantic Expression and Execution of B2B Contracts on Multimedia Content

91

developed by the Content Reference Forum. It formalized a language that enabled machine-readable representation of typical terms found in content distribution contracts and was compliant with the Business Collaboration Framework [4], but it was not finally standardized. In the following years, the advent of the Semantic Web reached the contract expression formats, and new representations evolved from the syntactic representation level to the semantic one ([5][6][7]) being developed domain ontologies in the KIF (Knowledge Interchange Format) or OWL (Ontology Web Language) languages. Still climbing levels in the Semantic Web layered model, RuleML first and SWRL (Semantic Web Rules Language) after were enacted as the new model container for electronic contracts, given that a contract declares a set of rules [8]. SWRL provides a Web-oriented abstract syntax and declarative knowledge representation semantics for rules; but the concrete syntax can have the form of a RDF Schema (Resource Description Framework), thus providing a seamless integration with OWL ontologies. Some of these contract models have also been aimed at governing Information Technology systems [9][10]. 2.2 Contract Representation with eContracts Currently, the most widely acknowledged standard is the eContracts, promoted by the OASIS consortium. Their electronic contract representation banks on XML again, but it has gained rapid acceptance. This is the culmination of the LegalXML eContracts Technical Committee, which started in 2002 to evaluate a possible eContracts Schema, and which achieved its final form in 2007 [11]. The model proposed in this chapter uses this eContract standard as a framework for the execution of contracts in the audiovisual B2B sector. Being eContracts the container, contents are expressed with the help of some existing ontologies in the field, like the Media Value Chain Ontology [21] and the principles of deontic logic for contracts execution in other formalizations.

Fig. 1. Top elements in the eContracts XML contract

92

V. Rodríguez-Doncel and J. Delgado

eContracts is a standard aimed at representing general contracts, having no particular field in scope. eContracts documents are composed of general paragraphs and clauses, being the main XML elements the ec:item, the ec:title, the ec:block and the ec:text with the item element used recursively. Figure 1 shows the XML schema for the root and the main elements (the root element is, of course, the ec:contract). ecmetadata elements in the contracts allow the specification of contract date, creator or title (using the Dublin Core metadata elements). Contract parties are declared in the ec:contract-front part, being the clauses under the ec:body element. 2.3 Represention of Audiovisual Contracts The most common clauses found in the body of multimedia contents contracts have been studied [12] by analyzing some paper contracts in the sector. Besides the metadata and header clasues, the main kinds of clauses are listed here: • • • • • • • • • •

Rights. Actions that the licensee can execute. Resource. Object of the actions to be executed. Report and Auditing. Obligation to report on sales or action executions. Fee. Fee to be payed in exchange to a transfer of IP or content delivery. Territory. Spatial condition imposed on the execution of an action. Term. Temporal condition imposed on the execution of an action. Confidentiality. Prohibition to release information around the agreement. Disclaimer. Denial of responsibilities on certain issues. Jurisdiction. Agreed legal frame and court on case of dispute. Breach and termination. End of the contract extintion conditions.

These kinds of clauses may constitute refinements of the existing eContracts elements like ec:item or ec:block. But a better refinement of the structure could be given, specifying the nature of the clause. Clauses can be classified according to one of the standard deontic logics notions of ‘prohibition’, ‘obligation’ and ‘permission’. Some other clauses are auxiliar of the former, only describing facts agreed by both parties (assertions). Each of the clauses in the contract can be classified as one of them (prohibition, obligation, permission or assertion) (see Table 1). Contract metadata and contract front constitute also assertions. The namespace aec stands for audiovisual electronic contracts schema. Table 1. Kind of clauses and their meaning Kind of clause 1. aec:permission 2. aec:prohibition 3. aec:obligation 4. aec:assertion ec:metadata ec:contract-front

Meaning What the licensee can do What the licensee cannot do What the licensee must do What both parties agree it is Data on the contract itself Contract heading

Example clause Licensee rights Confidentiality Fee, territory, term Jurisdiction Contract date Contract parties

Semantic Expression and Execution of B2B Contracts on Multimedia Content

93

3 Contract Representation with Rights Expresion Languages and the Media Value Chain Ontology A parallel and practical effort in automating the trade of multimedia assets has been made with the Digital Right Management (DRM) systems and the electronic licenses as the contract-like elements governing the transactions. These licenses are expressed in a Rights Expression Language (REL) and they can be seen as effective electronic contracts that are being enforced. Examples of RELs are the MPEG-21 REL [13] and the Open Digital Rights Language (ODRL) [14]. This joint analysis of RELs as electronic contracts is not new [12] but so far it has not received enough attention. In both MPEG-21 REL and OMA DRM, an XML file contains a license which expresses the rights one of the parties has and the conditions that have to hold. And in both cases, these licenses can represent both in-force contracts or license offers. Comparing with Section 2.3, contract clauses representable in a REL license are only those directly needed for a rights enforcement: the parties clause, the rights clause, and some of their enforceable conditions (fee, territory, grant etc.). 3.1 Parties in REL Licenses Licenses refer always to two parties, likewise most of real contracts. Actually an MPEG-21 license may content several grants each of them with a different party, but then we can consider the grant as the basic license unit. In MPEG-21 language, parties are called issuer and principal, while in ODRL they are directly referred as parties, classified as ‘end users’ and ‘right holders’. No more information is given about who might be these parties, excepting that they are uniquely identified, and that one of them (the rights issuer) electronically signs the document. In the framework of MPEG-21, the concept of ‘user’ include “individuals, consumers, communities, organizations, corporations, consortia, governments and other standards bodies and initiatives around the world” [15]. In ODRL, parties can be humans, organizations, and defined roles. According to the standards, users are only defined by the actions they perform, but if we attend to the expressivity of both RELs, in the licenses there can be only end users and distributors. This is enough for most of DRM platforms, but a contract model should consider all the user roles appearing in the complete media value chain. 3.2 Rights and Conditions in REL Licenses Rights and conditions are declared together in REL licenses, and they are declared in only one direction: rights mean rights of the licensee, and conditions mean conditions that have to be met for the licensee to execute his rights. In practice, contracts in the audiovisual sector include clauses in both directions: each of the parties may have rights, obligations and prohibitions. The rights defined in MPEG-21 REL and ODRL (Table 2) are only focused in one of the directions, and they are not enough to describe all the actions that are permitted in contracts. The list of actions and rights needed to express the contract information is given in Table III, which was elaborated after a systematic analysis of existing contracts in the audiovisual market [12].

94

V. Rodríguez-Doncel and J. Delgado Table 2. MPEG-21 REL rights and ODRL permissions MPEG-21 REL rights

ODRL permissions Usage

End user Enlarge Reduce Move Adapt Extract Embed

End user Play Print execute Install Uninstall delete

Distributor Issue Revoke Obtain modify

Reuse

Asset Management End-user Display Modify Move Print Excerpt Duplicate Play Annotate Delete Execute Aggregate Verify Backup/Restore Install/Uninstall

Transfer Distributor Sell Lend Give Lease

Table 3. Main actions and rights to be considered in a contract representation Most common rights appeared in contracts Reproduce Download Upload MakeAvailable PubliclyPerform

Broadcast Copy Print Record Modify

Adapt Convert Transcode Remix Distribute

Lease License Promote Stream Translate

Advertise Dub Transmit Exhibit Sell

The comparison shows that MPEG-21 rights and ODRL permissions do not completely represent the information expressed in B2B contracts, and although RELs foresee mechanisms for the extension of the rights list, the main unaddressed issue is that they were not B2B conceived. 3.3 Parties and Rights in the Media Value Chain Ontology Much of the mismatch between RELs expressivity and contracts reality can be shortened with the help of the Media Value Chain Ontology (MVCO) [16]. XML representation of contracts under the form of REL licenses is of limited expressivity compared to the ontology-based contracts presented in Section 2. However none of the domain ontologies in Section 2 has been applied in the context of a content distribution system or a DRM system. The Media Value Chain Ontology is a semantic representation of the intellectual property along the Value Chain conceived in the framework of the MPEG-21 standard. The MVCO is based on work by the authors [17] and from an ontology that is part of the Interoperable DRM Platform (IDP), published by the Digital Media Project. The Media Value Chain Ontology is represented using the expressivity of OWLDL, and thus each class is well defined and related to a set of attributes and to other classes in a very precise way. In practice, applications can be deployed where the particular users, IP entities, actions etc. are instances of the ontology. The model defines the minimal set of kinds of Intellectual Property, the roles of the users interacting with them, and the relevant actions regarding the Intellectual Property law.

Semantic Expression and Execution of B2B Contracts on Multimedia Content

95

Although the MVCO was not intended to describe specifically contracts, their vocabulary is useful given that contracts on audiovisual material are contracts essentially on its intellectual property. If every contract represents an agreement between two parties who belong to the value chain, contracts can be classified according to the signing parties. Figure 2 shows the typical name of the contract types and relates them with the parties, including the contract between End User and Distributor (usually an oral contract). Creator Adaptation Contract Synchronization Contract

Execution Contract Performance Contract

Adaptor

Instantiator

Edition Contract Exploitation Contract Producer Distribution Contract

Broadcast Contract

Distributor

Broadcaster Cable TV Contract etc.

End user license End User

Fig. 2. Kinds of contracts in the media value chain

An example of eContracts fragment declaring the parties using the MVCO is given in Figure 3. 00 <ec:contract xmlns="urn:oasis:names:tc:eContracts:1:0"> 01 <ec:contract-front> 02 <ec:parties> 03 <ec:party><mvco:Distributor rdf:about="#Alice"/> 04 <ec:party><mvco:EndUser rdf:about="#Bob"/> 05 06 Fig. 3. eContracts parties declaration using the MVCO expressions

The IP value chain considers the different kinds of IP entities linked from the original work to the final product, as can be seen in Figure 4. There is a parallel between the chain of IP and the chain of contracts. Figure 4 shows the IP entities along the value chain, starting from work as the original abstract conception of an artist and finishing in the product as the most elaborate IP entity ready to be enjoyed by the end user.

96

V. Rodríguez-Doncel and J. Delgado

Fig. 4. The IP Value Chain

The MVCO can add much of its vocabulary to the semantics on contract representation. For example, three of its main classes are “Action” (comparable to the rights analyzed in the previous section) “User” (comparable to the parties in the contracts) and “IP Entity” (a classification of the objects of the contract from a IP point of view). Table 4 lists these classes and their inmediate subclasses. Table 4. Main classes of the ontology Root classes IP Entities Roles Actions

Subclasses Work, Adaptation, Manifestation, Instance, Copy, Product Creator, Adaptor, Instantiator, Producer, Distributor, EndUser TransformingActions (adapt, perform, etc.), EndUserActions (play etc)

Among the different parties and interests in the value chain, we may find creators, adaptors, performers, producers, distributors or broadcasters, all of them adding value to the product, and all of them tied by agreements in which Intellectual Property rights are handed over in exchange of economic compensations. Contract parties, if expressed in the terms of the MVCO, belong to one of the following classes: Creators, Adaptors, Instantiators, Producers, Distributors and EndUsers. Being broader in scope, the MVCO provides a good starting point for deriving new actions and parties to be used in a contract vocabulary. 3.4 RDF Representation of Deontic Clauses with MVCO OWL-DL is a Description Logics knowledge representation language, whose expression can be mapped to a first order predicate logic system. Predicates are verb phrase templates that describe properties of objects, or a relationship among objects represented by the variables (e.g. “Bob is a Creator”). As the given statements representing

Semantic Expression and Execution of B2B Contracts on Multimedia Content

97

the domain knowledge constitute a formal deductive system, the ontology can be queried (e.g. “has Bob created any Work?”). For each syntactically correct expression, the OWL-DL ontology is able to assert its truth value: either true, false or unkown (for the latter case, note that that OWL uses the open world assumption). All the above makes OWL an ideal mean to handle the truthness of propositions. However, not all the propositions in the English language (or human thinking) convey a truth value. Commands, questions or deontic expressions cannot be said to be true or false, and contracts carry its most valuable information in sentences like these (e.g. “Party A must pay party B in a yearly base”). This kind of expressions lies in the field of deontic logic [22]. Modal logics are concerned with other modalities of existence (usually necessity or possibility), and introduce two new monadic operators related like this:

◊P ↔ ⌐□⌐P and

□P ↔ ⌐◊⌐P The deontic logic is a kind of modal logic of the highest interest to represent contracts, and in place of the operators □, ◊ we can interpret “Obligation”, and “Permission” (in the above expressions, it can be read that “P is obligatory” is equivalent to “it is not permitted not P”). Actually only one of both operators is strictly necessary, as the second can be deduced from the first, but for readability, usually both are kept. In these expressions, P is no more than an alethic proposition. The MVCO defines a class “Fact” with a definite truth value (overcoming the open world assumption which enabled an unkown state), and an object property “hasRequired” which linked to a Permission enables the expression of obligations. The most important clauses found in multimedia content contracts, as they were defined in Section 3 are either alethic sentences (we call them Assertions, and we can represent them with the symbol P) or deontic expressions, the latter being either Permisison (⌐◊⌐P), Prohibition (◊⌐P) or Obligation (◊P). Similar approaches in the treatment of contracts can also be found in the literature [20]. Clauses which are liable to be enforced are enclosed in the aec:enforceable element. 3.5 Semantic Contract Representation We have now all the ingredients to describe a semantic representation of the audiovisual contracts. First we have a good structure given by the eContracts standard, which is naturally extendable. A general purpose standard like eContracts, with only 51 XML elements, cannot cover the details of any particular domain, but it can be refined. Then, we have a rich vocabulary of specific terms from the DRM world (and specified formally in the RELs), which is complemented by the elements in the semantic model of the complete value chain given in the Media Value Chain Ontology. Note that even though MVCO is an OWL-DL ontology, the declaration of its individuals is given in RDF and as such, easily integrable in the eContracts structure. Note that the the multimedia objects, if properly annotated with Multimedia Semantics, can bear information about its intellectual property nature making the integration easier. A real eContract clause carrying RDF triples of the MVCO can take the form shown in Figure 5.

98

V. Rodríguez-Doncel and J. Delgado

00 <ec:body> 01 <ec:item> 02 03 04 <mvco:permitsAction rdf:resource="#Action000"/> 05 <mvco:issuedBy rdf:resource="#Alice"/> 06 <mvco:hasRequired rdf:resource="#Germany"/> 07 08 09 <mvco:MakeAdaptation rdf:about="#Action000"> 10 <mvco:actedBy rdf:resource="#Bob"/> 11 <mvco:actedOver rdf:resource="#mywork1"/> 12 13 14 ISO:DE 15 16 <mvco:Work rdf:about="#mywork1"> 17 <mvco:hasRightsOwner rdf:resource="#Alice"/> 18 19 20 21 22

Fig. 5. eContracts clause integrated with a MVCO Permission

The XML snippet in Figure 5 asserts that there is a work called mywork1 (lines 1618), that there is a Fact called Germany (lines 13-15), and that there is an action that is Bob “making an adaptation over mywork1” (lines 09-12). It also declares that there is a permission given by Alice for Bob to make an adaptation provided that it is in Germany (lines 03-07). All this information is enforceable (line 02).

4 Execution of Semantic Audiovisual Contracts What is called the execution of contracts is no more than the authorisation of some media transfers (or the keys to decrypt) and the automatic dispatch of payment orders. Both transfers of bits and money can be ordered as the result of the execution of some rules given in the audiovisual eContract, fired with some events (purchase, consumption etc.). Figure 6 depicts a general environment of execution of audiovisual contracts, where the “semantic expression of contracts” is the eContracts extension we have described together with the MVCO elements and their extensions, the semantic expression of the context describes also as RDF the environment (spatial, temporal etc.) and the firing events are user request for rights execution or purchases (contract proposal accepts). The detailed syntax of these expressions exceeds the purpose of this paper, but they are a set of SWRL rules which ultimately offer an authorisation result and dispatch event reports.

Semantic Expression and Execution of B2B Contracts on Multimedia Content

99

Fig. 6. Execution environment for audiovisual electronic contracts

5 Conclusions This work acknowledges REL licenses as the governing element in DRM systems for B2C distribution of multimedia content, and declared licenses as the digital version of end user or distributor contracts. However, after an analysis of real contracts in the IP contents B2B market, it was observed that more flexibility was required to cope with the complexity of those narrative contracts. On the other hand, other electronic contract representations lack the needed formalism to steer content distribution systems. The MVCO, a recently presented ontology of the IP value chain model, may overcome the limitations of the existing RELs and may merge well into the OASIS eContract structure. This combination can govern a content distribution system with all the value chain players if some additions are made. In particular, an event description system is needed, and an authorisation mechanism too, capable of processing the dynamic events, the current context and these MVCO-extended eContracts. The execution of SWRL rules can determine this authorisation and make the electronic contracts to be truly semantic containers.

References 1. Kobryn, C., Atkinson, C., Milosevic, Z.: Electronic Contracting with COSMOS - How to Establish, Negotiate and Execute Electronic Contracts on the Internet. In: 2nd Int. Enterprise Distributed Object Computing Workshop (EDOC 1998), CA, USA (1998) 2. Tan, Y.-H., Thoen, W.: DocLog: an electronic contract representation language. In: Proceedings of the 11th International Workshop on Database and Expert Systems Applications (September 2000) 3. CRF Content Reference Forum. CEL: Contract Expression Language (2002), http://www.crforum.org/candidate/CELWG002_celspec.doc 4. Hofreiter, B., Huemer, C.: UN/CEFACT’s Business Collaboration Framework - Motivation and Basic Concepts. In: Proceedings of the Multi-Konferenz Wirtschaftsinformatik, Germany (March 2004) 5. Kabilan, V., Johannesson, P.: Semantic Representation of Contract Knowledge using Multi Tier Ontology. In: Proceedings of the First International Workshop on Semantic Web and Databases, Germany (June 2003)

100

V. Rodríguez-Doncel and J. Delgado

6. Llorente, S., Delgado, J., Rodríguez, E., Barrio, R., Longo, I., Bixio, F.: Generation of Standardised Rights Expressions from Contracts: An Ontology Approach? In: Meersman, R., Tari, Z., Herrero, P. (eds.) OTM-WS 2005. LNCS, vol. 3762, pp. 836–845. Springer, Heidelberg (2005) 7. Yan, Y., Zhang, J., Yan, M.: Ontology Modeling for Contract: Using OWL to Express Semantic Relations. In: Proceedings of the 10th IEEE International Enterprise Distributed Object Computing Conference, pp. 409–412. IEEE Computer Society, Los Alamitos (2006) 8. Paschke, A., Bichler, M., Dietrich, J.: ContractLog: An Approach to Rule Based Monitoring and Execution of Service Level Agreements. In: Adi, A., Stoutenburg, S., Tabet, S. (eds.) RuleML 2005. LNCS, vol. 3791, pp. 209–217. Springer, Heidelberg (2005) 9. Morciniec, M., Salle, M., Monahan, B.: Towards Regulating Electronic Communities with Contracts. In: Proceedings of the 2nd Workshop on Norms and Institution in Multi-agent Systems, Canada (June 2001) 10. Krishna, P.R., Karlapaplem, K., Dani, A.R.: From Contracts to E-Contracts: Modeling and Enactment. Information Technology and Management 6(4), 363–387 (2005) 11. Leff, L., Meyer, P. (eds.): OASIS LegalXML eContracts Version 1.0 Committee Specification (2007), http://docs.oasis-open.org/legalxml-econtracts/CS01/ legalxml-econtracts-specification-1.0.pdf 12. Rodríguez, V., Delgado, J., Rodríguez, E.: From Narrative Contracts to Electronic Licenses: A Guided Translation Process for the Case of Audiovisual Content Management. In: Proceedings of the 3rd International Conference on Automated Production of Cross Media Content for Multi-Channel Distribution, Spain (November 2007) 13. ISO/IEC 21000-5:2004, Information technology — Multimedia framework (MPEG-21) — Part 5: Rights Expression Language (2004) 14. Ianella, R.: Open Digital Rights Language (ODRL) Version 1.1, W3C (September 2002), http://www.w3.org/TR/odrl/ 15. Bormans, J., Keith Hill, K.: MPEG-21 Overview v.5, ISO/IEC JTC1/SC29/WG11/N5231 (October 2002) 16. MPEG-21, Media Value Chain Ontology, Committee Draft, ISO/IEC JTC1/SC29/WG11 N10264, Busan, South Korea (October 2008) 17. Gauvin, M., Delgado, J., Rodriguez-Doncel, V.: Proposed RRD Text for Approved Document No 2 – Technical Reference: Architecture, v. 2.1, Digital Media Project DMP0952/AHG40 (July 2007) 18. Guth, S., Simon, B., Zdun, E.: A Contract and Rights Management Framework Design for Interacting Brokers. In: Proc. of the 36th Hawaii International Conference on System Sciences (HICSS), Big Island, Hawaii/USA (January 2003) 19. AXMEDIS, Specification of Axmedis, AX4HOME Architecture, Automatic Production of Cross Media Content for Multi-Channel Distribution, DE12.1.3.1 (July 2007) 20. Prisacariu, C., Schneider, G.: A Formal Language for Electronic Contracts. In: Bonsangue, M.M., Johnsen, E.B. (eds.) FMOODS 2007. LNCS, vol. 4468, pp. 174–189. Springer, Heidelberg (2007) 21. Rodríguez, V., Gauvin, M., Delgado, J.: An Ontology for the Expression of Intellectual Property Entities and Relations. In: Proceedings of the 5th International Workshop on Security in Information Systems (WOSIS 2007), Portugal (April 2007) 22. Rodríguez, J.: Lógica deóntica: Concepto y Sistemas. Universidad de Valencia, Secretariado de Publicaciones, Valencia (1978) 23. Rodríguez, V., Delgado, J.: Multimedia Content Distribution Governed by OntologyRepresented Contracts. In: Workshop on Multimedia Ontologies and Artificial Intelligence Techniques in Law, Netherlands (December 2008)

A Conceptual Model for Publishing Multimedia Content on the Semantic Web Tobias B¨ urger and Elena Simperl Semantic Technology Institute (STI), University of Innsbruck, Innsbruck, Austria {tobias.buerger,elena.simperl}@sti2.at

Abstract. Retrieving multimedia remains a challenge on the Web of the 21st century. This is due, among other things, to the inherent limitations of machine-driven multimedia understanding, limitations which equally hold on the Web 2.0 or on its semantic counterpart. Describing multimedia resources through metadata is thus often seen as the only viable way to enable efficient multimedia retrieval. Assuming the availability of metadata descriptions, effectively indexing multimedia requires mediating among the wide range of metadata formats presently used to annotate or describe these resources. Semantic technologies have been identified as a potential solution for this large-scale interoperability problem. The work presented in this paper builds upon this last statement. We introduce RICO, a conceptual model and a set of ontologies to mark up multimedia content embedded in Web pages and to deploy such multimedia descriptions on the Semantic Web. By using semantic metadata referencing formal ontologies, we not only provide a basis for the uniform description of multimedia resources on the Web, but also enable automatic mediation between metadata standards, and intelligent multimedia retrieval features.

1

Introduction

In 2001, Tim Berners-Lee et al. introduced the idea of an augmented World Wide Web in which information is meaningful for machines as well as for humans [1]. Since then, and encouraged by phenomena such as the Web 2.0, the Web has turned into a global multimedia environment in which billions of professionally produced, but also more and more end-user generated content is published, shared, and used everyday. In the last decades much research has been done on how to analyze, organize, and annotate the rapidly increasing amounts of multimedia to enable better search, retrieval, and reuse of content (cf [2]. Still, a generic solution applicable at the scale of the current, and future Web has not been delivered yet. This is due, among other things, to the so-called “Semantic Gap” [3], which refers to the inherent gap between low-level multimedia features, which can be feasibly extracted automatically, and high-level, semantic ones, which can not be accurately derived without human intervention [4]. The advent of Web 2.0-style T.-S. Chua et al. (Eds.): SAMT 2009, LNCS 5887, pp. 101–113, 2009. c Springer-Verlag Berlin Heidelberg 2009 

102

T. B¨ urger and E. Simperl

technologies, notably tagging systems, can be seen as a preliminary answer to this problem. The availability of tags, however, does not solve all issues related to multimedia retrieval. A first still open issue is related to the relevance of the available metadata within the retrieval process. There seems to be a gap between the multimedia descriptions available, and the queries end-users issue in Web search engines when looking for multimedia content, as shown, for example, in a recent study on image search behavior [5]. A second equally complex issue is the indexing of multimedia content which can be found on the Web. Efficient indexing techniques require rich descriptions of multimedia resources covering a multitude of aspects [6]: low-level features such as color histograms or motion vectors, high-level features such as aboutness and contextual information referring to the ways content can be used. The diversity of multimedia types and description standards thereof hamper efficient indexing as well. Unlike in classical information retrieval, multimedia retrieval needs techniques to search, index and rank images, videos, music, as well as more interactive resources such as slideshows, learning material, or even 3D presentations. A last issue worthwhile being mentioned is the level of granularity at which metadata are typically provided. A significant share of multimedia content remains largely hidden to multimedia search engines due to the fact that fine-granular descriptions of such content are not available, or are poorly exploited. Parts of multimedia resources are not retrievable as metadata is available at most for resources as a whole. Further on, information about the usage or license conditions is not explicitly linked to the respective multimedia resources, though this information is mostly available in textual form. Providing a common interface to search and retrieve the various types of multimedia available online is challenging. This interface needs to abstract from media types as such, to automatically mediate among the various schemes and formats presently used to annotate and describe multimedia, and to cover all aspects previously mentioned in order to enable fully-fledged multimedia retrieval. In this paper we propose RICO, a conceptual model to describe multimedia resources on the Web. The model implements a multimedia resource-centric view of conventional Web pages, thus being equally applicably to multimedia content published within Web pages (e.g., images embedded in news stories), to social media, or to professional media licensing sites. It is implemented as a set of Semantic Web ontologies (using W3C recommendations such as RDF and OWL) and uses RDFa [7] to mark up multimedia resources inline HTML pages. By using semantic metadata referencing formal ontologies, RICO not only provides a basis for the uniform description of multimedia resources on the Web, but also enables automatic mediation between metadata standards, and intelligent multimedia retrieval features such as automatic reasoning about the contents and structure of multimedia resources and their usage conditions. Compared to related work in the field of semantic multimedia (cf. Section 4) our main contributions are the comprehensiveness of the conceptual model, and the ways it can be deployed on the Semantic Web. RICO captures in-depth information about the multimedia resources themselves, but also contextual

A Conceptual Model for Publishing Multimedia Content

103

information about the current or prospected usage of these resources. RICO allows to relate descriptions, annotations and multimedia resources to each other by means of structural or logical relations. In doing so, it goes beyond the functionality of standards such as MPEG-7, whose Multimedia Description Schemes is restricted to the structure of multimedia resources and associated components, objects and events. The usage context of a particular resource, rights, reviews, opinions etc, is also taken into account. This information is bundled together with the resource itself as a logical package, similarly to standards such as METS [8], IMS Content Packaging [9] or OAI-ORE [10]. The remainder of this paper is organized as follows: We present the RICO conceptual model in Section 2 and the way it can be used on the Semantic Web in Section 3. We compare our approach with related work in Section 4 before summarizing our achievements and outlining future directions of research and development in Section 5.

2

The RICO Conceptual Model

The primary aim of the RICO model is to provide a metadata framework for the description of multimedia content and accompanying information on the Web. RICO is intended to group available information about a multimedia resource into a compound information package to make it available as a single accessible unit of value. This idea is illustrated in Figure 4: There, all information which is available about an image on a Web site is grouped into a single package. The overall design of the RICO model is based on the requirements outlined in [11,12]: The RICO model uses semantics to reduce the issue of ambiguity and to enable interoperability between different standards. Furthermore more richer descriptions are provided along with multimedia resources: on the one hand, descriptions published together with multimedia content on Web sites are explicitly linked to the content itself in order to enable more efficient indexing. On the other hand, multimedia content is accompanied by rich metadata sets following a metadata model which captures a multitude of features proven to be relevant for efficient retrieval purposes (cf. Section 2.2).

Fig. 1. The RICO Conceptual Model: Coverage of Different Metadata Types

104

T. B¨ urger and E. Simperl

The RICO model is separated into two parts (cf. Figure 1): The first part is called the RICO data model and covers structural metadata to describe information about what makes up the information package. The second part, the RICO metadata model, covers descriptive, administrative, and use metadata. Both the data and the metadata model within RICO are implemented as Semantic Web ontologies, which are introduced in Section 2.3. 2.1

The RICO Data Model

The general aim of the RICO data model is to offer a set of well-defined concepts to describe different aspects of multimedia resources and accompanying information on Web pages. As such, its aim is to lay a graph over content published on the Web, associating descriptions in a Web page to it, and to bind metadata to content and descriptive information. As a baseline for our model we use the MPEG-21 Digital Item Declaration (DID) Abstract Model [13]. The assessment of different models was made in accordance to a general comparative framework for multimedia content models introduced in [14]. MPEG-21 fulfills the basic characteristics of an adaptable data model proposed in this framework, that are, granular description of fragments, media elements (resources), grouped resources (or components), and the possibility to identify and select (parts of) resources.

Fig. 2. Main Elements Within the MPEG-21 Digital Item Declaration Model Used in RICO

The basic parts of an MPEG- 21 Digital Item which are interesting from this aspect are depicted in Figure 2: These include most notably containers in which identifiable digital assets are included (items) and which contain (multimedia) resources. Fragments of these resources can be selected and both resources and their fragments can be described via descriptors or annotated via annotations. Descriptors represent author-given metadata, while annotations provide additional information contributed by other users in the life cycle of a resource. The RICO data model is most notably realized by a Semantic Web ontology which covers the MPEG-21 DID Abstract Model and amongst others makes the

A Conceptual Model for Publishing Multimedia Content

105

semantic types of relations between media resources, components, or fragments and their descriptors explicit. By that, the RICO DID ontology overcomes some problems of the MPEG-21 DID XML Schema that can be traced back to the lack of formal semantics. The nesting of elements in the schema, especially for descriptive elements, is ambiguous – sometimes the descriptions belong to the parent element in the XML Schema, sometimes to the siblings of the respective element. The MPEG-21 DID Abstract Model, and implicitly our ontological model thereof, is compatible with the OAI Abstract Information Model [15] as shown in [16] and with learning content models such as the IMS Content packages [9] or METS [8] as proven by the RAMLET working group1 . 2.2

The RICO Metadata Model

The RICO metadata model captures different types of metadata which have been identified as important for multimedia retrieval by media professionals [17]: Bibliographic metadata is concerned with the authorship of content and includes fields such as identification, naming, publication or categorization. Technical metadata typically describes physical properties such as format, bit-rate and what is mostly called low-level features of content [2]. Classification metadata includes keywords or tags, but also domain-specific classification information. Evaluative metadata includes ratings and qualitative assessments of the content. Relational metadata covers relations between the content and other related external resources. Relations may be explicitly specified through the design of the content object or external objects or could be derived from observations or the usage history. Rights metadata describes the terms of use of a multimedia resource. Functional metadata refers to operations that may be supported to alter the presentation of the content, to customize or personalize the content, or to provide access to different versions of the content. The RICO metadata model can be further extended into domain-specific aspects such as educational characteristics for a learning object or preservation data for archival information. In this respect our approach is inline with the vision of Resource Profiles introduced in [18]. At the level of the metadata model we differentiate between authorative and non-authorative metadata. The former is contributed by the author (creator) of the multimedia content and reflects persistent information about it. The latter is provided by consumers or other third parties and refers to contextual and mostly changing aspects related the content [19]. Authorative metadata is typically created manually, whilst non-authorative metadata may either be explicitly provided (through annotations, reviews, ratings, etc.) or automatically generated 1

cf. http://www.ieeeltsc.org/working-groups/wg11CMI/ramlet

106

T. B¨ urger and E. Simperl

(through harvesting or usage analysis or alike). In order to support efficient filtering and optimal usage these different metadata types are conceptually separated into so-called “metadata sets”. 2.3

The RICO Ontologies

The data and metadata model introduced in the previous sections are implemented using a set of Semantic Web ontologies in order to publish semantic descriptions for multimedia resources on the Web. The ontologies were built in order to increase the interoperability of media resource descriptions on the Web. To be compatible with popular ontologies, and to ease the uptake of the RICO model, the RICO ontologies reuse or are based on existing ontologies. The ontologies define their own terms when needed terms do not exist in any suitable ontology or when their meaning should be precisely defined. In the latter case, terms are linked to other vocabularies via OWL/RDF(S) modeling constructs. If possible, terms are aligned to other vocabularies via mappings.

Tag MARCO Intra Relations

FRBR Relationship

MARCO Inter Relations

Commerce Voc. CC Copyright Voc.

RICO

MARCO

DublinCore Dublin Core Terms

DigitalMedia

MPEG-21 DID

FOAF

Annotea Annotations

MARCO Ratings

Fig. 3. The Reusable Intelligent Content Objects (RICO) Ontologies

The import-graph of the RICO ontologies is shown in Figure 3:2 The RICO Core ontology imports several other ontologies as follows: – The MPEG-21 DID ontology implementing the RICO data model. – The Mindswap Digital Media ontology which is used to type multimedia resources.3 2 3

The dark-gray and white ontologies were built in the course of this work. Others were reused (and adapted). http://www.mindswap.org/2005/owl/digital-media (last accessed on 07.03.2009).

A Conceptual Model for Publishing Multimedia Content

107

– The FOAF ontology for representing users and their properties.4 – The Annotea annotations ontology to represent annotations which can be attached to the content and to support referencing of classification information (cf. [20]).5 – The OWL-Lite version of the Dublin Core ontology to attach basic descriptive properties to the elements of the RICO data model.6 – The MARCO Core ontology which covers a core set of properties which are commonly used on the Web to describe bibliographic aspects, classifications of resources, and rights. It particularly makes use of • The MARCO Inter- and Intra- Relations ontology which reflect relations between media resources and between descriptions and the resources.7 • The MARCO Ratings ontology which reflects ratings about qualities of media resources. The MARCO Core ontology further reuses (parts of) the Dublin Core Terms ontology 8 , the Tag ontology 9 , and the Commerce vocabulary 10 . 2.4

RICO Profiles

The RICO model and its ontologies are generic with respect to media types. They can not only be applied to a restricted set of media types, but to the full range of media resource available on the Web. This means on the one hand, that RICO provides a core set of elements which can be used for description and retrieval across media types. On the other hand, the model and the ontologies have to be extended in order to optimize their application for specific media types. This can be done with RICO profiles. The profiling approach was chosen because it increases the reusability of the RICO model, whilst extensions and customizations are easily possible. Media resources of different types vary significantly with respect to many properties. Therefore different technical metadata formats, mechanisms to identify fragments, or to specify licenses, are needed. For technical metadata, a wide range of ontologies were proposed (cf. [2]). For the identification of media fragments, different media-specific solutions exist, such as in MPEG-7, MPEG-21, SMIL, SVG, or CMML (“The Continuous Media Markup Language”).11 Furthermore specialized licensing schemes are available such as the PLUS (“Picture Licensing Universal Standard”). 4 5 6 7 8 9 10 11

An OWL-DL version was used; cf. http://www.mindswap.org/2003/owl/foaf (last accessed on 07.03.2009). The Annotea ontology available at http://www.w3.org/2000/10/annotation-ns# (last accessed on 07.03.2009) was rebuilt in OWL. http://www.dublincore.org (last accessed on 27.03.2009). The MARCO Inter-Relations ontology is based on the Functional Requirements for Bibliographic Records (FRBR) – model [21]. http://dublincore.org/documents/dcmi-terms/ (last accessed on 07.03.2009). http://www.holygoat.co.uk/projects/tags/ http://digitalbazaar.com/commerce (last accessed on 07.03.2009). http://www.w3.org/2008/WebVideo/Fragments/wiki/State_of_the_Art (last accessed on 07.03.2009).

108

T. B¨ urger and E. Simperl

For the description of images on the Web we implemented a basic RICO profile, that is, the RICO4Images profile. This profile uses the EXIF ontology as provided by Kanzaki12 to describe technical aspects and makes use of the svgOutline mechanism to localize image regions as illustrated in [22]. This mechanism suggests to use an SVG [23] description to specify regions in images. Furthermore the profile instantiates the MARCO Ratings ontology to enable rating on image qualities. Steps to define further profiles are outlined in [17].

3

The Usage of the RICO Conceptual Model

The RICO model can be used to describe images published for different purposes such as illustration of Web pages, sharing, or selling as shown in [17].

Fig. 4. An Image Hosted by Flickr Described Using the RICO Model

3.1

Describing Multimedia Content Using the RICO Model

Figure 4 shows an arbitrary image available at Flickr. RICO information is explicitly marked up and related to the image (see numbers 1 − 12 in Figure 4). The example contains three metadata sets: one authorative set which was created by the owner of the image, and two non-authorative sets which are provided 12

http://www.kanzaki.com/ns/exif

A Conceptual Model for Publishing Multimedia Content

109

by the hosting platform and an end-user through a commentary (see number 3 in the Figure). The figure shows only how visible information is related to the image. However, additional (non-visible) metadata could also be provided, e.g., by adding a detailed description of different scenes in a video or further semantic descriptions of the content of the image. [12,17] provides further details on how the RICO ontologies can be used to describe images using the RICO ontologies. 3.2

Embedding RICO Descriptions into HTML Using RDFa

The semantic descriptions according to the RICO model can be published as a compound package together with the multimedia resource within an HTML page. Typically compound packages are published using XML to link resources and their metadata (e.g., in MPEG-21, IMS Content Packaging [9] or METS [8]). In our case, the compound package information is published inline within an HTML page using RDFa. RDFa is a serialization syntax for the RDF data model. It defines how an RDF graph is embedded in an (X)HTML page using a set of defined attributes such as @about, @rel, @instanceof, and others. By using Ramm.x (“RDFa enabled multimedia metadata”) [24], external metadata descriptions can additionally be included in a RICO by reference. An example showing the description of the Flickr image using the RICO model is provided on the Web along with a demonstration of the automatic extraction of RDF triples from the HTML page using available online services.13 3.3

Tools

In order to demonstrate the model we implemented a Firefox browser plug-in which can be used to manually annotate images using the RICO ontologies. The plug-in stores annotations directly in the DOM of the HTML page using RDFa. The plug-in enables to load ontologies to be used for annotation, and to export and validate the generated RDF data.14

4

Related Work

Especially in recent years much research has been done on the specification of ontologies that aim to combine traditional multimedia description models to allow reasoning over the structure and semantics of multimedia content (cf. [11,2]). The RICO conceptual model and its associated ontologies can be used to mark up illustrative images in conventional Web pages, images or videos hosted 13 14

http://tobiasbuerger.com/icontent/rico/example/ (last accessed on 07.03.2009). The plug-in is available on request.

110

T. B¨ urger and E. Simperl

by social media sharing sites, slides and images embedded in commercial offerings, or multimedia content deployed in blogs and wikis. The aim of the model is not to provide a sophisticated description facility for the structural and content related semantics as it is the case with MPEG-7 [25] or one of the recently proposed MPEG-7-based ontologies such as COMM [26]. We rather propose a common core, grounded into available standards in the multimedia area and other related area, to describe existing Web pages from the perspective of multimedia resources through a basic model which can be extended with existing ontologies. One model with a similar scope is the hMedia microformat:15 It provides a basic vocabulary to mark up multimedia resources on Web sites using propertyvalue pairs. Also the ramm.x model [24] has a similar scope. It provides a small, but extensible vocabulary for marking up multimedia resources and to embed semantic metadata inside of HTML pages. It mainly covers the inclusion of traditional metadata via service transformations, but does not provide a detailed vocabulary to describe multimedia resources. The aim of the SMIL MetaInformation-Module16 is to publish RDF-based metadata in SMIL presentations. It is very general and does not provide additional information about how it can be used in concrete applications. Another relevant approach is Adobe’s Extensible Metadata Platform (XMP).17 XMP specifies means to publish RDF-based metadata into PDFs or other document formats. Furthermore we want to acknowledge the work done by Creative Commons to describe and embed licensing data using RDF which is exploited in searches by Yahoo or in Flickr.18 The scope of the work in the W3C on an ontology for media resources on the Web is comparable to the scope of the MARCO Core ontology introduced in Section 2.3.19 More heavyweight approaches include Intelligent Content models as previously assessed for example in [27] which cover a broad range of aspects. Most of these approaches are too heavyweight for our proposed model. Traditional, non-semantic models include the standardized framework of MPEG-7 [25] or packaging formats from the archival or eLearning domains such as the IMS Content Packaging format [9], the Metadata Encoding and Transmission Standard (METS) [8], the MPEG-21 Digital Item Declaration (DID) [13] and most recently the OAI-Object Reuse and Exchange - model (OAI-ORE) [10]. The OAI-ORE model was designed as an exchange format for scholarly works. Its compound object model is similar to our model as it also provides facilities to publish semantic descriptions as an overlay graph over Web pages [28]. The approach however does not focus on multimedia aspects, but on grouping of resources into one single manageable entity. 15 16 17 18 19

More information available at http://wiki.digitalbazaar.com/en/Media_Info_Microformat http://www.w3.org/TR/2007/WD-SMIL3-20070713/smil-metadata.html http://www.adobe.com/products/xmp/ See http://search.creativecommons.org/ http://www.w3.org/2008/WebVideo/Annotations/

A Conceptual Model for Publishing Multimedia Content

5

111

Conclusions

In this paper we presented the RICO conceptual model. The model can be used to describe multimedia resources which are embedded in typical Web pages, or published on arbitrary media sharing platforms or commercial sites on the Web. It consists of a data model that supports adaptability of content, a metadata model that enables efficient retrieval of content and a deployment mechanism to publish RICO information inline of HTML pages. The model is implemented using a set of Semantic Web ontologies which resort to existing standards. In order to ensure interoperability with existing standards in the field we based our work on the MPEG-21 DID model which itself is compatible to METS, OAI and others. For the same purpose we reused several established ontologies or ontology-like formats such as the Dublin Core schema, the FOAF ontology, the Annotea annotations scheme and the Mindswap Digital Media ontology. We evaluated the compatibility of the metadata model with existing standards by defining mappings as outlined in [17]. Our next steps include further tools to foster the uptake of the model and to automate metadata generation. Future versions of the RICO model will incorporate results of the W3C Media Annotation and Media Fragments working groups on fragment identification and media resource metadata.20 Acknowledgments. The research leading to this paper was partially supported by the European Commission under contract IST-FP6-027122 “SALERO”.

References 1. Berners-Lee, T., Hendler, J., Lassila, O.: The semantic web. The Scientific American 284(5), 28–37 (2001) 2. B¨ urger, T., Hausenblas, M.: Metadata standards and ontologies for multimedia content. In: Handbook of Metadata, Semantics and Ontologies. World Scientific Publishing Co., Singapore (2010) (to be published) 3. Smeulders, A.W.M., Worring, M., Santini, S., Gupta, A., Jain, R.: Content-based image retrieval at the end of the early years. IEEE Trans. Pattern Anal. Mach. Intell. 22(12), 1349–1380 (2000) 4. Hollink, L., Schreiber, G., Wielinga, B., Worring, M.: Classification of user image descriptions. Int. J. Hum.-Comput. Stud. 61(5), 601–626 (2004) 5. Pu, H.T.: An analysis of failed queries for web image retrieval. J. Inf. Sci. 34(3), 275–289 (2008) 6. Gong, Z., Leong Hou, U., Cheang, C.W.: Web image indexing by using associated texts. Knowl. Inf. Syst. 10(2), 243–264 (2006) 7. Adida, B., Birbeck, M. (eds.): Rdfa primer, w3c working group note, October 14 (2008), http://www.w3.org/TR/xhtml-rdfa-primer/ 8. The Library of Congress: Metadata encoding and transmission standard: Primer and reference material version 1.6 (September 2007), http://www.loc.gov/standards/mets/mets-schemadocs.html 20

http://www.w3.org/2008/WebVideo/

112

T. B¨ urger and E. Simperl

9. IMS Global Learning Consortium: Ms content packaging information model, version 1.1.4 final specification (October 2004), http://www.imsglobal.org/content/packaging/ cpv1p1p4/imscp infov1p1p4.html 10. Lagoze, C., de Sompel, H.V., Johnston, P., Nelson, M., Sanderson, R., Warner, S.: Ore user guide – primer (October 2008), http://www.openarchives.org/ore/1.0/primer.html 11. B¨ urger, T., Hausenblas, M.: Why real-world multimedia assets fail to enter the semantic web. In: Proc. of the Int. Workshop on Semantic Authoring, Annotation and Knowledge Markup, SAAKM 2007 (2007) 12. Buerger, T.: Towards increased reuse: Exploiting content related and social features of multimedia content on the semantic web. In: Proceedings of the Workshop on Interacting with Multimedia Content on the Web (IMC-SSW), co-located with SAMT 2008, December 3-5, Koblenz, Germany (2008) 13. Bormans, J., Hill, K.: Mpeg-21 overview v.5, ISO/IEC JTC1/SC29/WG11 (2002) 14. Boll, S., Klas, W.: Zyx – a multimedia document model for reuse and adaptation of multimedia content. IEEE Transactions on Knowledge and Data Engineering 13(3), 361–382 (2001) 15. CCSDS: Reference model for an open archival information system. Blue Book 1, Consultative Committee for Space Data Systems, CCSDS Secretariat Program Integration Division (Code M-3) National Aeronautics and Space Administration Washington, DC 20546, USA (January 2002) 16. Bekaert, J., Kooning, E.D., van de Sompel, H.: Representing digital assets using mpeg-21 digital item declaration. International Journal on Digital Libraries 6(2), 159–173 (2006) 17. B¨ urger, T.: An Intelligent Content Model for the Semantic Web. PhD thesis, University of Innsbruck (June 2009) 18. Downes, S.: Resource profiles. Journal of Interactive Media in Education Special Issue on the Educational Semantic Web 5 (2004) 19. Recker, M., Wiley, D.: A non-authoritative educational metadata ontology for filtering and recommending learning objects. Journal of Interactive Learning Environments (2001) 20. Kahan, J., Koivunen, M.R.: Annotea: an open RDF infrastructure for shared web annotations. In: Proceedings of the 10th International World Wide Web Conference, pp. 623–632 (2001) 21. IFLA Study Group on the Functional Requirements for Bibliographic Records: Functional requirements for bibliographic records, final report (1998), http://www.ifla.org/VII/s13/frbr/frbr.htm 22. Troncy, R., van Ossenbruggen, J., Pan, J.Z., Stamou, G.: Image annotation on the semantic web, w3c incubator group report (August 14, 2007), http://www.w3.org/2005/Incubator/mmsem/XGR-image-annotation/ 23. Jackson, D. (ed.): C.N.: Scalable vector graphics (svg) full 1.2 specification, w3c working draft (April 13, 2005), http://www.w3.org/TR/SVG12/ 24. Hausenblas, M., Bailer, W., B¨ urger, T., Troncy, R.: Ramm.x: Deploying multimedia metadata on the semantic web. In: Proceedings of SAMT 2007, December 4-7, Genova, Italy (2007) 25. Mart´ınez-Sanchez, J.M., Koenen, R., Pereira, F.: Mpeg-7: The generic multimedia content description standard, part 1. IEEE MultiMedia 9(2), 78–87 (2002)

A Conceptual Model for Publishing Multimedia Content

113

26. Arndt, R., Troncy, R., Staab, S., Hardman, L., Vacura, M.: Comm: Designing a well-founded multimedia ontology for the web. In: Aberer, K., Choi, K.-S., Noy, N., Allemang, D., Lee, K.-I., Nixon, L.J.B., Golbeck, J., Mika, P., Maynard, D., Mizoguchi, R., Schreiber, G., Cudr´e-Mauroux, P. (eds.) ASWC 2007 and ISWC 2007. LNCS, vol. 4825, pp. 30–43. Springer, Heidelberg (2007) 27. B¨ urger, T., Toma, I., Shafiq, O., D¨ ogl, D.: State of the art in sws, grid computing and intelligent content objects - can they meet? GRISINO Deliverable D1.1 (October 2006), http://www.grisino.at 28. van de Sompel, H., Lagoze, C.: Interoperability for the discovery, use, and re-use of units of scholarly communication. CTWatch Quarterly 3(3) (August 2007)

CAIN-21: An Extensible and Metadata-Driven Multimedia Adaptation Engine in the MPEG-21 Framework* Fernando López1, José M. Martínez1, and Narciso García2 1

VPULab, EPS – Universidad Autónoma de Madrid, Spain {f.lopez,josem.martinez}@uam.es 2 GTI, ETSIT – Universidad Politécnica de Madrid, Spain [email protected]

Abstract. This paper presents the CAIN-21 multimedia adaptation engine, which facilitates the integration of pluggable multimedia adaptation modules, chooses the chain of adaptations to perform and manages its execution. Evolving from CAIN, CAIN-21 complies better with the MPEG-21 framework. Its new features and improvements are discussed in this paper. In addition, the pros and cons are explained with respect to others multimedia adaptation engines, including early CAIN. Keywords: multimedia, adaptation, metadata, mpeg-7 mpeg-21.

1 Introduction As time goes by, the variety of multimedia formats and devices has significantly increased, and still does. Multimedia content providers need to distribute their photos, videos and audio to a wide-range of devices and independently of the underlying delivery technology. This technological breakthrough has been explained through the Universal Multimedia Access (UMA) paradigm [1], where content adaptation plays an important role. The purpose of multimedia adaptation is to carry out changes in the multimedia content in order to enable its consumption in different terminals, operating with different access networks and satisfying the preferences of the multimedia system and users as far as possible. Frequently, the multimedia adaptation tools are intended to adapt a specific kind of media (e.g. audio, video, images), or even a specific media format (e.g. H.264/AVC), to a set of constraints imposed by the usage environment: namely the terminal, the network and the preferences of the multimedia system and users. The MPEG-21 [2] standard has taken into account the importance of interoperability among multimedia services. It proposes a framework that facilities the integration *

Work financed by the European Commission (IST-IP-001765 – aceMedia, IST-FP6-027685 – MESH), the Spanish Government (TEC2007-65400 - SemanticVideo), the Ministerio de Educación y Ciencia of Spanish (through the FPU fellowship grant issued to the first author) and the Comunidad de Madrid (S-0505/TIC-0223 - ProMultiDis-CM).

T.-S. Chua et al. (Eds.): SAMT 2009, LNCS 5887, pp. 114–125, 2009. © Springer-Verlag Berlin Heidelberg 2009

CAIN-21: An Extensible and Metadata-Driven Multimedia Adaptation Engine

115

of multimedia services and standards in order to harmonise multimedia technologies. In addition, the MPEG-21 framework has introduced general multimedia elements that can be instantiated in different scenarios. The MPEG-21 framework and its elements have been frequently used (for example in [3][4][5]) to describe multimedia systems. This paper presents an adaptation engine named CAIN-21 (Content Adaptation INtegrator in the MPEG-21 framework), which has evolved from CAIN [6]. The source code along with an online demo of its functionalities can be publicly accessed at cain21.sourceforge.net. The main objective of CAIN-21 is to provide a framework in which different multimedia adaptation tools can be integrated and tested. With the extensibility mechanism that CAIN-21 incorporates, adaptation tools can be added in a pluggable manner. Moreover, this adaptation engine includes an automatic decision process that enables the quick generation of new types of multimedia adaptations. Specifically, by combining existing multimedia adaptations tools, CAIN-21 is capable of addressing a wider range of multimedia adaptations. In the rest of this paper, Section 0 will review the state of the art concerning metadata-driven multimedia adaptation. Section 0 explains the main features and elements of CAIN-21. This section also provides a review of early CAIN and explains the new features and advantages that the new multimedia adaptation engine has incorporated. Section 0 provides a comparative analysis between CAIN-21 and other multimedia adaptation engines. Finally, Section 0 concludes the papers and gathers the advantages of the adaptation techniques explained in this paper.

2 State of the Art Metadata-driven multimedia adaptation [7] makes use of metadata to automatically decide the adaptation to perform and subsequently execute it. This metadata may come from both manual annotations and automatic content analysis techniques. The MPEG-7 [8] standard specifies description tools for both sources of metadata. In addition, the MPEG-21 [2] standard specifies the metadata to describe the complete multimedia system. Metadata-driven content adaptation has been studied at resource level [3], at system level [4] and at scene level [5][9]. Initial proposals for metadata-driven adaptation were implemented using XML to describe both, the multimedia content and the usage environment [10][11]. After their standardization by ISO, the MPEG-7 and MPEG-21 standards have gained popularity among the content adaptation community. Descriptions have been frequently classified in [12]: (1) content descriptions, with information of the media resource along with its metadata: title, media format, variations, natural language description, audiovisual features, etc. MPEG-7 has been frequently used to specify the metadata representation. In the MPEG-21 framework, a Component element comprises the media resource and its corresponding metadata. A Digital Item (DI) is a container of Component elements, additional nested DIs and metadata. (2) Service provider environment descriptions, which usually contain restrictions imposed by the service provider. Streaming bandwidth and privacy policies are examples of these descriptions. The MPEG-21 Part-7 Universal Constraints Description Tools (UCD Tools) has been frequently used in this case [12][13][14]. (3)

116

F. López, J.M. Martínez, and N. García

Usage environment descriptions, which describe the content consumer environment. Examples of these descriptions are the features of the terminal, the network characteristics or the preferences of the multimedia system and users. The MPEG-21 Part 7 Usage Environment Description Tools (UED Tools) have been used in this case [3][4][12]. Roughly speaking, the constraints of the content provider as well as the consumer ones are described with the UCD Tools. The usage environment of the consumer is described with the UED Tools. The adaptation engine is situated between both extremes. This adaptation engine can be implemented in the content provider's multimedia server or in the content consumer's terminal. The adaptation engine can even be located in a proxy. This idea has been proposed to perform distributed multimedia adaptation [15]. Within the multimedia adaptation engine, metadata-driven approaches such as variation selection, transcoding or transmoding have been examined [16]. Metadatadriven adaptation has also been used to perform scalable media adaptation [13][14]. In this area, the AdaptationQoS Tools [17] have been frequently used to decide the optimal adaptation over a media resource. First, these methods store the set of feasible adaptation in AQoS. Second, they eliminate the solutions that do not satisfy all the UCD limit or UED constraints. Third, they use the UCD optimization constraints to decide the optimal solution. The Bitstream Syntax Description Tools (BSD Tools) and BSDLink Tools [18] may assist the adaptation process by describing the high-level structure of the stream. Then by executing Extensible Stylesheet Language Transformations (XSLT) over the BSD, the resource adaptation is carried out efficiently. In the area of scene level adaptation, metadata-driven adaptation has been used to transcode web pages [19]. In this case, external annotations are associated to existing web documents in order to transcode them to the constraints and preferences of the user. Another complementary approach for dealing with multimedia content has been proposed by the Semantic Web, which advocates ontology based approaches [20][21][22] so that the machine will be capable of understanding both the content and the relationships among its metadata descriptions. In [23], Semantic Web technologies are used to perform scene of interest selection and scalable video frame rate adaptation.

3 CAIN-21: Architecture, Functionalities and Improvements 3.1 Interfaces CAIN-21 serves adaptation requests through two external programming interfaces (see Fig. 1 below): (1) The media level transcoding interface is devoted to perform blind adaptation (i.e. semantic-less adaptation) of a media resource. In addition to media level, this interface is also capable of performing system level adaptation of videos composed of one or more audio and visual streams. The transcoding operations defined in the media level transcoding interface are implemented in the Tlib module that includes conventional software libraries such as ffmpeg, imagemagick as well as JNI-based custom libraries. (2) The DI level adaptation interface is in charge of performing system level (semantic or blind) adaptations where metadata is used for helping during the adaptation.

CAIN-21: An Extensible and Metadata-Driven Multimedia Adaptation Engine

117

The DI level adaptation interface deals with three types of MPEG-21 DIs: (1) the Content DI is a DI that conveys the media resource along with its metadata; (2) the Context DIs are repositories that contain the information of the usage environment along with the information that the adaptation engine employs in deciding and executing the most suitable adaptation (i.e. the adaptation capabilities and the properties to be taken into account); (3) the Configuration DI states which terminal, network and user − from the ones available in Context DI − must be used to serve the adaptation. The purpose of the Configuration DI [24] is to decouple the Content DI and the Context DIs. Instead of storing information of the adaptation state in the Content DI (as the MPEG-21 Part-2 Choice mechanism or MPEG-21 Part-7 DIA Configuration do), the Configuration DI stores all the information related to the adaptation that is to be carried out. In this way, the Content DI will not be modified when the target usage environment (located in a Context DI) changes. Usage examples and further explanations of this mechanism can be found in [24]. In CAIN-21, metadata-driven adaptation is performed through the DI level interface and at Component level. An MPEG-21 Component includes a media resource (in the Resource element) and its metadata (in the Descriptor element). The DI level adaptation interface provides two different operations. The first one modifies the existing Component and the second operation adds a new Component element to the DI. More specifically: (1) transform() takes a Component from the Content DI and modifies its media resource and metadata in order to adapt it to the usage environment; (2) addVariation() takes a Component from the DI and creates a new Component ready to be consumed in the usage environment. This adapted Component is added to the Content DI at the end of the adaptation. 3.2 Architecture This section provides a detailed description of the modules that CAIN-21 incorporates. Fig. 1 depicts CAIN-21's functional modules and the control flow along the adaptation process. Adaptation Management Module (AMM) The AMM is in charge of coordinating the whole DI level adaptation process. Modules below the AMM perform different tasks initiated by the AMM. Adaptation Decision Module (ADM) and Adaptation Execution Module (AEM) In the last years a great deal of research has been made in order to automatically construct multimedia adaptation plans (see e.g. [21][22][25]). The distinction between the notion of Adaptation Decision Module (ADM) and the Adaptation Execution Module (AEM) has been frequently proposed [5][12][13][25]. CAIN-21 also includes this distinction and we have done a great deal of work in order to automate the decision and execution process. In CAIN-21, the pluggable adaptation modules are implemented by means of Component Adaptation Tools (CATs). The ADM uses metadata to decide [26] the sequence of conversions and parameters that should be executed over a specific Component element of the Content DI. Subsequently, the AEM executes the corresponding sequence of CATs over this Component. Caching

118

F. López, J.M. Martínez, and N. García

Fig. 1. Modules and control flow within CAIN-21

mechanisms can speed up any or both of these steps by avoiding deciding or executing several times adaptations that comprise the same parameters. Context Repository The Context Repository encompasses three types of Context DIs (see Fig. 1,). The Usage Environment DI is a Context DI that describes the available usage environments using the MPEG-21 UED Tools. Each CAT Capabilities DI describes the different conversions that one CAT is capable of undertaking. Each conversion has a set of valid input and output properties along with its corresponding values. CAIN-21 has introduced an addressing mechanism [24] in which changes in the underlying metadata descriptors will not imply changes in its source code. With this mechanism, metadata is dealt by means of properties. The Properties DI is intended to store a set of keys and corresponding XPath/XPointer expressions. Parsing Module (PM) The PM is the module in charge of resolving the values of the aforementioned properties. Firstly, the PM accesses the Properties DI in order to obtain the set of property keys and corresponding XPath/XPointer expressions. Secondly, on resolving these expressions, the values of these properties are generated. During this step the rest of the metadata is loaded from the Content DI, Configuration DI, Usage Environment DI and CAT Capabilities DI. At the end of this step all the metadata is represented as a set of properties. The value of these properties can be multi-valued (e.g. bitrate = [1000..200000], audio_format = {aac, mp3}).

CAIN-21: An Extensible and Metadata-Driven Multimedia Adaptation Engine

119

Coupling Module (CM) There exist a wide variety of multimedia representation standards and systems. CAIN-21 is an adaptation engine that is designed with the ability to be integrated in heterogeneous multimedia systems. The CM is the gateway for other components that may be using external technology (i.e., non-MPEG-21 technology) to represent multimedia (e.g. HTML, SMIL, NewsML or MPEG-4 BIFS). The CM enables the integration of CAIN-21 adaptation services into those heterogeneous multimedia systems. With this purpose, this module transforms the external representation of multimedia onto an MPEG-21 compliant input DI that is capable of being processed by CAIN-21. In addition, the CM is in charge of transforming the adapted output DI onto its external representation. Different instances of the CM are interchangeable modules created to interact with specific external systems. 3.3 Control Flow Numbers in Fig. 1 indicate the control flow of the tasks involved in the adaptation process. (1) On interacting with external systems, the CM transforms the external multimedia representation into MPEG-21 compliant DIs that CAIN-21 is capable of processing. (2) The AMM is in charge of coordinating the whole DI level adaptation process. (3) When a Content DI along with a Configuration DI arrives (via the above transform() or addVariation() operations of the DI level interface) the AMM invokes the ADM and the AEM to decide (4) and execute (5) the corresponding adaptation over an specific Component. (6) The CATs frequently employs the TLib in order to adapt the media resource. The CATs might also change or append information to the Descriptor element of the Component so that the subsequent CATs may use it. (7) When all the conversions of the sequence have been executed, (8) the AMM returns the adapted Content DI to the caller. (9) Frequently, the adapted Content DI may need to be transformed to an external representation. In this case, the CM will perform this transformation. 3.4 Early CAIN Early versions of CAIN received a media resource, an MPEG-7 description of this resource and an MPEG-21 description of the usage context. After its execution, CAIN produced an adapted media along with its MPEG-7 media description. Subsequent versions of CAIN developed its modular architecture. Fig. 2 shows the architecture as published in [6]. In this version, CAIN comprises a Decision Module (DM) and a set of adaptation operations called Content Adaptation Tools (CATs), Encoders and Decoders. In response to an external invocation, the multimedia resource, MPEG-7 description of this resource and an MPEG-21 description of the context were parsed and the DM performed three main steps in sequence: (1) Selecting the target media parameters, (2) selecting an adaptation tool capable of performing the adaptation and lastly, (3) the selected CAT/Encoder/Decoder was launched. The next step in the development of CAIN was twofold [27]: (1) the development of an extensibility mechanism and (2) the development of a formal decision process capable of dealing with this extensibility. The extensibility mechanism proposes the

120

F. López, J.M. Martínez, and N. García

Fig. 2. Early CAIN

use of a CAT Capabilities document to describe the adaptation capabilities of each pluggable CAT. The automatic decision mechanism was intended to select a CAT capable of performing the adaptation and the parameters to use. 3.5 From CAIN to CAIN-21 The first set of changes relates to the PM. In early CAIN the PM was in charge of parsing all the metadata: the MPEG-7 Part 5 MediaDescriptionType describing the media and the MPEG-21 Part 7 description of the UED. The changes undergone in the PM have been in two directions: Firstly, these elements are currently represented by means of the Content DI, Context DIs and Configuration DI as explained above. Secondly, and more important, the PM implemented in CAIN required changes in the source code of CAIN whenever a new description was added or changed. CAIN-21 has introduced a mechanism [24] in which metadata is dealt through properties stored in the Properties DI (see section 0). In this way, changes in the set of properties managed by the adaptation engine imply only changes in the Properties DI. Additionally, with this on-demand mechanism only the values of the properties used by the decision-making process are evaluated. During the execution of early CAIN, it received a document with the UED. The existence of only one Terminal, Network and User element was assumed. Different UED documents represented different adaptation environments. CAIN-21 has gathered the UED in a Context DI referred as the Usage Environment DI. With CAIN-21 more than one Terminal, Network and User elements can be stored in the Usage Environment DI. Once CAIN-21 is deployed in a multimedia system, the target Terminal,

CAIN-21: An Extensible and Metadata-Driven Multimedia Adaptation Engine

121

Network and User elements can be addressed by means of the Adaptation Request Configuration Tool (ARC Tool) [24]. As explained in section 0, CAIN incorporated three kinds of adaptation modules (see Fig. 2 (b)): Content Adaptation Tools (CATs), Encoders and Decoders. The third major change in CAIN-21 gathered all of these modules under the concept of Component Adaptation Tool (CAT). In CAIN-21 adaptations are always performed at Component level. An MPEG-21 Component includes a media resource and its metadata. Early CAIN included only one operation named adapt() with four parameters [27]: the input content, the output format, the output folder and a set of properties reserved for future functionalities. CAIN-21 divided this operation into two different operations: (1) transform() and addVariation() explained in section 0. Additionally, in CAIN-21, the number of parameters is variable and determined as a subset of the properties gathered from metadata. Further advances in the implementation of an automatic decision mechanism have motivated changes in the CAT Capabilities description mechanism. As further explained in [24][26], in order to express disjunction in the adaptation capabilities, each CAT capabilities document has to be divided into several ConversionCapabilitiesType elements. In order to allow the existence of multi-valued properties, the CAT Capabilities description mechanism was changed. In this way the properties of the CAT capabilities may be single-valuated (e.g. format = {mpeg2}), multi-valuated (e.g. color_space = {rgb, grayscale}), ranges (e.g. bitrate=[100..400000]) or compound properties (e.g. frame_size= {144x176, 288x352}). Ranges were also allowed in compound properties (e.g. frame_size = [10..5000]x[10..5000]). In the current CAT Capabilities description model [24], each ConversionCapabilitiesType element contains preconditions and postconditions [26] which are used by the automatic decisionmaking process. The adaptation of large media resources such as videos may imply long delays if the resource needs to be adapted before being delivered. Early CAIN only supported the Forward Adaptation Mode. [28] With this mode adaptation takes place as soon as the user requests the resource. The client characteristics, preferences and natural environment can be taken into account, but if the resource adaptation process is time consuming, the user has to wait until the whole resource is adapted. This kind of adaptation is useful for small resources (e.g. images), but it is undesirable for long resources (e.g. video or audio). CAIN-21 introduces the Online Adaptation Mode in which the media resource can start to be delivered before the whole media resource has been adapted. Lastly, in CAIN-21 a great deal of research has been made in order to automatically construct multimedia adaptation plans. As explained in [27], early CAIN was "not truly extensible in the sense it is currently, that is, it was possible to add additional CATs but it was needed to code or recode some parts in the core of CAIN". In order to automate the decision-making process an Artificial Intelligence planner [21][26] was implemented in the ADM of CAIN-21. This approach uses a description of the input and output parameters of the CATs as preconditions and postconditions, respectively. The planner (under development in [24]) computes a sequence of zero or more CATs along with its parameters in order to adapt the media to the usage environment. CAIN was only capable of selecting and executing one CAT in order to perform the adaptation. CAIN-21 can perform multistep adaptation, i.e., CAIN-21 can

122

F. López, J.M. Martínez, and N. García

find and execute sequences of conversions of any length. Another difference between CAIN and CAIN-21 is that the former was only capable of finding one of the feasible solutions to address the adaptation problem, whereas the latter computes all the feasible adaptation solutions (sequences of conversions). Subsequently, only one of these sequences needs to be executed in order to adapt the content. Deciding which of these sequences best suit the usage environment is a work in progress.

4 Comparison with Others Multimedia Adaptation Approaches This section provides a comparative review of four multimedia adaptation engines that work in the MPEG-21 framework: koMMa [21], BSD [14], DCAF [12], NinSuna [23] and CAIN-21. The comparison is based on five aspects: decision-making method, completeness of the computed solutions (i. e. the underlying algorithm finds all the solutions), type of multimedia content addressed, semantic adaptation and MPEG-21 Part-7 DIA tools supporting the adaptation. In reference to the decision-making method, BSD uses optimization methods to adapt the underlying resource. Even though BSD theoretically can be used to adapt any resource, it is only practical with scalable resources. This is because scalable resources are made up of blocks that can be easily modified or removed. DCAF proposes genetic algorithms to perform these optimizations with general video. NinSuna incorporates BSD-like frame-rate reduction. On the other hand, koMMa and CAIN-21 propose knowledge-based methods. The deterministic description of the operations defined in koMMa implies that their outputs are completely defined. Conversely, CAIN-21 allows partially defining the behaviour of the conversions [26] allowing thereby the use of optimization methods inside CATs. Optimization methods usually obtain a complete solution, i.e. all the feasible solutions are obtained and ranked: this is the case of BSD and DCAF. In reference to forward and backward search methods, forward search methods such as the one implemented in koMMa only finds one feasible solution (the first one it finds) whereas backward search planning methods, such as the current version of CAIN-21, obtain a complete set of solutions. NinSuna does not specify the completeness of its decisions. In reference to the supported media, BSD is particularly effective dealing with scalable media, whilst DCAF and NinSuna deal with general video resources. koMMa and CAIN-21 are intended to deal with a wider range of media resources. Currently, CAIN-21 can manage images, audio and video. BSD and DCAF used the gBSD tool [17] and the AdaptationQoS with IOPins linked to semantics to annotate the video stream on a semantic level. NinSuna provides semantic adaptation in the Scenes Of Interest (SOIs) selection and in the framerate reduction. CAIN-21 makes use of Regions of Interest (ROIs) to drive the Image2Video adaptation [29]. Roughly speaking, the UCD Tools have been used in the optimization methods (BSD and DCAF) and the UED Tools have been used in both: optimization and knowledge-based methods. The UCD Tools are frequently related to scalable multimedia content and CAIN-21 enables the use of the UCD Tools inside the CATs that deals with such scalable content.

CAIN-21: An Extensible and Metadata-Driven Multimedia Adaptation Engine

123

Table 1. Comparative of multimedia adaptation approaches

Decisionmaking method Complete solutions Multimedia content Semantic adaptation UCD Tools

koMMa

BSD

DCAF

NinSuna

CAIN-21

Knowledgebased

Optimization

Optimization

BSD frame reduction

Knowledgebased + Optimization

No

Ranking

Ranking

Unspecified

Images + Video No

Scalable media gBSD + AQoS Yes

Video

Video

gBSD + AQoS Yes

SOIs + BSD

Knowledge based + Ranking Images + Video + Audio ROIs

Unspecified

Inside CATs

No

As Table 1 shows, CAIN-21 combines the two major decision-making methods and implements a complete algorithm, i.e., and algorithm that identifies all the feasible adaptations that produce content satisfying the usage environment constraints. CAIN-21 is designed for extensibility and support a wider range of multimedia content that the rest of adaptation engines. Indeed, CAIN-21 is theoretically able of managing all content that can be represented as a DI. The BSD Tools and UCD Tools that implement the optimization method have been put apart from the knowledge-based decision mechanism and transferred to the CATs.

5 Conclusions This paper has explained an extensible and metadata-driven multimedia adaptation engine named CAIN-21. The extensibility mechanism enables the use of existing multimedia adaptation operations in different adaptation scenarios. By combining these adaptation operations a wider range of adaptation scenarios can be addressed. Metadata-driven multimedia adaptation facilitates the automatic identification of sequences of conversions that adapt the media resources to the usage environment. The general architecture of CAIN-21 includes the distinction between the Adaptation Decision Module (ADM) and the Adaptation Execution Module (AEM) which is an approach frequently found in the literature. The Properties DI is an innovative design approach that accesses to multimedia properties on-demand and allows changes in the adaptation capabilities of CAIN-21 without modifying the underlying source code. In order to provide extensibility, CAIN-21 proposes the notion of Component Adaptation Tools (CATs). Even though CAIN-21 uses the MPEG-21 framework to deal with multimedia, it can also provide adaptations to external multimedia representation systems (non-MPEG-21 compliant multimedia systems) by means of the Coupling Module (CM). The comparative analysis shows that CAIN-21 combines knowledge-based and optimization decision methods and is capable of managing a wider range of multimedia content than the others. Its extensibility mechanism paves the ground for an efficient and effective way for in creasing the range of content and formats that can be managed.

124

F. López, J.M. Martínez, and N. García

Acknowledgements We would like to thank to all the people that during these years have contributed to the final status of CAIN-21: Victor Valdés, Javier Molina, Álvaro García, Víctor Fernández-Carjabales, Jesús Bescós, Fernando Barreiro, Luis Herranz, Daniel de Pedro, Guillermo López and Javier Sanz.

References [1] Vetro, A., Christopoulos, C., Ebrahimi, T.: Universal Multimedia Access (special issue). IEEE Signal Processing Magazine 20(2) (2003) [2] Burnett, I.S., Pereira, F., de Walle, R.V., Koenen, R. (eds.): The MPEG-21 Book. John Wiley and Sons, Chichester (2006) [3] Sun, H., Vetro, A., Asai, K.: Resource Adaptation Based on MPEG-21 Usage Environment Descriptions. In: Proceedings of ISCAS 2003, pp. 536–539 (2003) [4] Rong, L., Burnett, I.: Dynamic multimedia adaptation and updating of media streams with MPEG-21. In: Proceedings of CCNC, pp. 436–441 (2004) [5] Pellan, B., Concolato, C.: Metadata-driven Dynamic Scene Adaptation. In: Proceedings of the International Workshop on Image Analysis for Multimedia Interactive Services (WIAMIS 2007), pp. 67–70 (2007) [6] Martínez, J.M., Valdés, V., Bescos, J., Herranz, L.: Introducing CAIN: A metadatadriven content adaptation manager integrating heterogeneous content adaptation tools. In: Proceedings of WIAMIS 2005, pp. 5–10 (2005) [7] van Beek, P., Smith, J.R., Ebrahimi, T., Suzuki, T., Askelof, J.: Metadata-driven multimedia access. IEEE Signal Processing Magazine 20(2), 40–52 (2003) [8] Manjunath, B.S., Salembier, P., Sikora, T. (eds.): Introduction to MPEG-7 Multimedia Content Description Interface. Wiley & Sons, Chichester (2002) [9] Asadi, M.K., Dufourd, J.C.: Context-Aware Semantic Adaptation of Multimedia Presentations. In: Proceedings of ICME 2005, pp. 362–365 (2005) [10] Villard, L., Roisin, C., Layaïda, N.: A XML-based multimedia document processing model for content adaptation. In: King, P., Munson, E.V. (eds.) PODDP 2000 and DDEP 2000. LNCS, vol. 2023, pp. 104–119. Springer, Heidelberg (2004) [11] Phan, T., Zorpas, G., Bagrodia, R.: An Extensible and Scalable Content Adaptation Pipeline Architecture to Support Heterogeneous Clients. In: Proceedings of DCS 2002, pp. 507–516 (2002) [12] Sofokleous, A.A., Angelides, M.C.: DCAF: An MPEG-21 Dynamic Content Adaptation Framework. Multimedia Tools and Applications 40(2), 151–182 (2008) [13] Mukherjee, D., Delfosse, E., Kim, J.G., Wang, Y.: Optimal adaptation decision-taking for terminal and network quality-of-service. IEEE Transactions on Multimedia 7(3), 454–462 (2005) [14] Kofler, I., Seidl, J., Timmerer, C., Hellwagner, H., Djama, I., Ahmed, T.: Using MPEG21 for cross-layer multimedia content adaptation. Journal on Signal, Image and Video Processing 2(4), 355–370 (2008) [15] Hutter, A., Amon, P., Panis, G., Delfosse, E., Ransburg, M., Hellwagner, H.: Automatic adaptation of streaming multimedia content in a dynamic and distributed environment. In: Proceedings of ICIP 2005, pp. 716–719 (2005) [16] Chung-Sheng, L., Mohan, R., Smith, J.R.: Multimedia content description in the InfoPyramid. In: Proceedings of ICASSP 1998, vol. 6, pp. 3789–3792 (1998)

CAIN-21: An Extensible and Metadata-Driven Multimedia Adaptation Engine

125

[17] ISO/IEC 21000-7:2004. Information technology – Multimedia framework (MPEG-21) – Part 7: Digital Item Adaptation [18] Deursen, D.V., de Neve, W., de Schrijver, D., de Walle, R.V.: gBFlavor: a new tool for fast and automatic generation of generic bitstream syntax descriptions. Multimedia Tools and Applications 40(3), 453–494 (2008) [19] Hori, M., Kondoh, G., Ono, K., Hirose, S.i., Singhal, S.: Annotation-based Web content transcoding. In: Proceedings of WWWCCN 2000, pp. 197–211 (2000) [20] Stamou, G., van Ossenbruggen, J., Pan, J.Z., Schreiber, G., Smith, J.Z.: Multimedia annotations on the semantic Web. IEEE Multimedia 13(1), 86–90 (2006) [21] Jannach, D., Leopold, K., Timmerer, C., Hellwagner, H.: A Knowledge-based Framework for Multimedia Adaptation. Applied Intelligence 24(2), 109–125 (2006) [22] Barbosa, V., Andrade, M.T.: MULTICAO: A Semantic Approach to Context-aware Adaptation Decision Taking. In: Proceedings of WIAMIS 2009, pp. 133–136 (2009) [23] Deursen, D.V., Lancker, W.V.: NinSuna: a Format-independent, Semantic-aware Multimedia Content Adaptation Platform. In: Proceedings of the 10th IEEE International Symposium on Multimedia, pp. 491–492 (2008) [24] López, F., Martínez, J.M., García, N.: Towards a fully MPEG-21 compliant adaptation engine: complementary description tools and architectural models. In: Proceedings of AMR 2008 (2008) [25] Lum, W.Y., Lau, F.C.M.: A context-aware decision engine for content adaptation. IEEE Pervasive Computing 1(3), 41–49 (2002) [26] López, F., Jannach, D., Martínez, J.M., Timmerer, C., Hellwagner, H., García, N.: Multimedia Adaptation Decisions Modelled as Non-Deterministic Operations. In: Proceedings of WIAMIS 2008, pp. 46–49 (2008) [27] Molina, J., Martínez, J.M., Valdés, V., López, F.: Extensibility of adaptation capabilities in the CAIN content adaptation engine. In: Poster and demo Proceedings of SAMT 2006, pp. 29–30 (2006) [28] Lee, J.Y.B.: Scalable continuous media streaming systems. Wiley & Sons, Chichester (2007) [29] López, F., Martínez, J.M., García, N.: Automatic adaptation decision-making using an image to video adaptation tool in the MPEG-21 framework. In: Proceedings of the 10th International Workshop on Image Analysis for Multimedia Interactive Services (WIAMIS 2009), pp. 222–225 (2009)

Shot Boundary Detection Based on Eigen Coefficients and Small Eigen Value Punitha P. and Joemon M. Jose Department of Computing Science, University of Glasgow, Glasgow, United Kingdom {punitha,jj}@dcs.gla.ac.uk

Abstract. Detection of shot boundaries in a video has been an active for quite a long time, till the TRECVID community almost declared it as a solved problem. A problem is assumed to be solved when there is no significant improvement being achieved from that of the state-of-the art methodologies. However, certain aspects can still be researched and improved. For instance, finding appropriate parameters instead of empirical thresholds to detect the shot boundaries is very challenging and is still being researched. In this paper, we present a fast, adaptive and non-parametric approach for detecting shot boundaries. Appearance based model is used to compute the difference between two subsequent frames. These frame distances, are then used to locate the shot boundaries. The proposed shot boundary detection algorithm uses an asymmetric region of support that automatically adapts to the shot boundaries. Experiments have been conducted to verify the effectiveness and applicability of the proposed method for adaptive shot segmentation. Keywords: Video retrieval, eigen value, non-parametric, shot boundary detection.

1 Introduction Proliferated production of videos has made the area of content based video retrieval and the underlying areas such as, visual content interpretation, analysis, and management, the most acclaimed and a focused area of research. In spite of the length of the video, most videos are handled in smaller chunks, either as a set of keyframes or as shorter video clips. This universal approach has made the very first step in video analysis, the shot boundary detection, an indispensable component of any video analysis and interpretation system [5, 15 ]. Due to the complexity and of shot boundaries, estimating the correct shot boundaries is more challenging. A shot in general, is a video segment, where the visual content of the segment remains consistent. The shot boundaries detection is difficult because of various video editing tools that result with a range of shot boundary types, e.g., abrupt, dissolve, fade in, fade out etc. Shot transitions can be of two types, abrupt and gradual. An abrupt shot transition is usually easier to detect than a gradual shot transition. Many different algorithms have been proposed in the literature [2, 3] to detect different types of transitions, and there have even been some attempts to handle all transitions at once [6]. It is very tough to T.-S. Chua et al. (Eds.): SAMT 2009, LNCS 5887, pp. 126–136, 2009. © Springer-Verlag Berlin Heidelberg 2009

Shot Boundary Detection Based on Eigen Coefficients and Small Eigen Value

127

detect all types of shot transitions using a single approach as the transitions are highly complex. One can still find such an attempt in [6]. Shot boundary detection algorithms are either based upon the pixel based difference between frames [14, 12, 10, 13, 8, 9, 6] or the motion difference on the temporal axis [3]. Despite the approach that is used, shot boundary detection algorithms require a prefixed threshold of difference value. The most challenging part of shot segmentation is fixing up such parameters or thresholds irrespective of the video genre and the type of shot transition, to determine a shot boundary. It is difficult to detect all types of shot transitions, at the same time with a fixed parameter. Fuzzy logic based algorithms [2] are not very sensitive to noise unlike direct threshold based approaches, but still requires threshold selection. A very few attempts can be found in the literature to detect shot transitions adaptively. An attempt to adaptively select a global threshold to achieve a high shot cut detection rate was also made in [17]. An approach based on adaptive selection of the threshold using the average of the weighted variance of the change in previous detected shot to the current frame and the next frame was proposed in [11] and was reported to produce 94% accurate results. In this paper, we propose an adaptive non-parametric method for detecting shot boundaries, without being specific to the type of transition. The method proposed to adaptively find the shot boundaries is inspired by a non-parametric corner point detection method [7, 1]. The distance between two subsequent frames is computed in terms of eigen distance between two frames. This vector of distances with its index can be perceived as the profile signature of an open contour. Therefore, each distance with its index forms a co-ordinate point on a two dimensional plane. The adapted corner detection method then determines the shot boundaries by computing the statistical and geometrical properties associated with the small eigen value for each co-ordinate point. For each point representing a frame, three features viz., region of support, confidence value and curvature value are computed and based on these features the true shot boundaries are located. The results indicate that the proposed algorithm is effective for various type video sequences. The various combination of the features help in detecting all types of shot boundaries. This remaining part of this paper is organised as follows. In Section 2 we define the various video transitions. Section 3, presents an overview of the corner detection method presented in [1, 7]. Section 4 presents the proposed shot segmentation methodology adapted on corner detection mechanism. In Section 5, the performance metrics used to evaluate the proposed method is presented. Section 5, briefs about the experimental results and the paper concludes in Section 6.

2 Video Transitions A video V, in domain 2D+t, can be seen as a sequence of frames ft, and can be described by , where T is the number of frames in the video. The way in which any two video shots are joined together is called the transition [9]. The most common transition is the cut, in which there is a sudden and abrupt change between two frames. Fade is characterised by a progressive darkening of a shot until the last frame becomes completely black. The next most common transition

128

Punitha P. and J.M. Jose

is the cross fade (mix or dissolve), where one shot gradually fades into the next. Fades have a slower, more relaxed feel than a cut. Flash is an increase of the luminosity in a few frames, which is common in television journal videos. Other advanced transitions include wipes and digital effects, which are complex changes, whilst leading into the next shot, such as, colour replacement, animated effects, pixelization, focus drops, lighting effects, a pan from one person to another, or a zoom from a mid-shot to a close-up etc. In the next section, we propose and present an algorithm that can adaptively detect all shot boundaries, irrespective of the types of shot transitions.

3 The Proposed Shot Segmentation Algorithm This section describes the proposed feature extraction, frame difference computation and the algorithm to detect shot boundaries caused due to cut, flash, wipe and cross fade (mix, dissolve). The problem of shot boundary detection can be related to that of corner point detection in objects. This is due to the following underlying similarities, (i) Corner points on a shape curve are effective primitives for shape representation and analysis. Similarly, shot boundaries in videos are effective primitives for video representation. (ii) Corner points on a digital boundary are found at locations where the nature of the boundary changes abruptly and significantly. On similar lines, shot boundaries in a video are also lying at locations, where the frames change abruptly and significantly. However, this definition only makes sense when the video or object is viewed from a global perspective and not at the local frame or point level. 3.1 Feature Extraction and Frame Difference Computation As pixel based difference analysis remains as one of the most preferred approaches to find the dissimilarity between the frames, we also use pixel based frame difference to locate the shot boundary. As evident from, PCA is a linear method for data feature extraction. It is a mathematical technique used to analyse correlated random variables to reduce the dimensionality of a data set. This reduction is achieved by selecting the first few principal components. These components capture the most relevant features to use in classifying a group of objects to be recognised. Given the ith frame as the current frame, its previous frame is considered as the reference frame. Both frames are normalised for their intensity values in the RGB plane and an average intensity frame is obtained. Two reflective frames lying on opposite sides of the average frame are found. The eigen vectors and the eigen values of the two reflective frames are then computed and are used to calculate the representative eigen co-efficients of the two frames. Let

be the matrix representing the two frames under process and

be

the average of normalised F. The average matrix is obtained by replicating the columns with average vector to simplify the matrix subtraction. Then,

(1)

Shot Boundary Detection Based on Eigen Coefficients and Small Eigen Value

129

A diagonal matrix D of eigen values and a full matrix V whose columns are the corresponding eigen vectors such that, XV = VD are computed. Using, X, V and D, the eigen coefficient matrix,

is computed. Ecoeff is a 2x2 matrix where the first row corresponds to the reference frame and the 2nd row corresponds to the current frame. The difference between these eigen coefficients of the two frames is recorded along with the current frame number as a boundary point. The process is repeated for all frames in sequence and the boundary points B, where

n is the number of

frames in the video, are the interger and real space. These boundary points are then used to locate the shot boundaries using the corner detection inspired nonparametric approach. 3.2 Shot Boundary Detection In this section, we first present an overview of the corner detection method proposed in [7, 1] and then present how it is adopted for shot boundary detection. An Overview of Non-parametric Corner Detection Approach [7, 1] In the domain of object detection, localising the true corner points using the local features. The paper presented asymmetric region of support to find true corner points. This subsection gives an excerpt from the original paper to keep the flow a smooth read. Let where, , is a neighbour of (mod n), and n be a close curve described in a clockwise direction. Let be the point of interest for which the region of support has to be determined to decide if is a corner point or not. The region of support of , consists of left arm , right arm , and the point itself. That is, and , where and denote, respectively, the sizes of left and right arms which are decided adaptively based on local properties of the contour and are not necessarily equal implying that the region of support are not necessarily symmetric. It has been shown in [16], that for a straight line segment, the small eigen value in the continuous domain is zero, regardless of its length and orientation. Hence, the maximum sequence of points, on either side of the point , for which the small eigen value

130

Punitha P. and J.M. Jose

associated with the covariance matrix of the sequence of points approximates zero, is selected as the points of the respective arms. To compute the right arm, initially, . Let be the geometrical centroid of and let be the entries of the covariance matrix of . The small eigenvalue is computed as given in equation (3),

Since, any two points forms a straight line, the small eigenvalue associated with two points is zero. The set is updated by adding the next point , in sequence and the small eigenvalue of is computed. In order to avoid recomputation of the small eigenvalue as and when the set is updated, the small eigenvalue is computed based on the previous set information as in [12]. This process of updating the set by adding the next point in sequence and computing the small eigen value associated with the updated set is repeated until no more approximates zero. Once this condition is true, the value of r is taken as the size of the right arm and the points in is the right arm of point . The left arm is computed in a similar manner, but with the points preceding the point . Finally, the region of support of point is the set with points which are in sequence from . Since, the points mark the end points of a segment, their confidence values are incremented by one. It can be noticed that the size of the determined region of support varies from point to point depending on the local property of the curve within the vicinity of the point of interest and, thus, the proposed method determines adaptively the region of support which is not necessarily symmetric. Once the region of support of a point is determined, the curvature at that point is then estimated as the reciprocal of the angle made at that point due to its left and right arms. Determination of region of support of all points helps in computing the size of the region of support of each point, curvature at each point and also to compute how many times (limit value) a point has been the endpoint of the region of support of other points. It is observed experimentally that the limit value, the size of the region of support and the curvature of an actual corner point are relatively larger than the respective values of its neighbors. Non-parametric Shot Boundary Detection Since after the computation of the frame difference, the frame number and the distance, can be visualised as a point on a open contour, we refer each point p(x, y) in the 2D plane, with x representing the frame number and y represents the eigen distance of frame x, to its next frame in the video. Hence, similar to the approach referred and explained in previous section, for each point p, three features, (i) the region of support (ii) the curvature at p, formed by the two arms and (iii) the confidence value, as described earlier are extracted.

Shot Boundary Detection Based on Eigen Coefficients and Small Eigen Value

131

Now, if is the point of interest for which the region of support has to be determined to decide if marks a shot boundary, the region of support of , left arm , right arm , are computed as explained above. Once the region of support of a point is determined, the curvature at that point is estimated in three different ways and are labelled, , , and . is computed as the reciprocal of the angle made at that point due to the left and right arms, , and are computed as given by (4) and (5) respectively,

Where, is the slope of the line joining the points and is the slope of the line joining the points . , helps in detecting the cuts or abrupt change in the frames forming a clear cut shot boundary. But, is not very helpful to find shot boundaries caused due to gradual change in frames. This problem is generally caused when the subsequent frame difference is considered, as the subsequent frame difference remains small when there is only a slight change occurring between frames. At this point it seems plausible to assume that if instead of computing the difference between subsequent frames, if the difference between these frames is computed with respect to third frame, which could be a template frame/ image, it would provide more informative, helpful and discriminating differences. However, this is in fact not the case, as the selection of a template image/frame becomes a real challenge, as the resultant difference would depend completely on the template selected. Instead to overcome this problem, we use Von Mises probability density function , as used by [5] to find the distribution of angles at a point, on the contour and is given by (5),

To make the shot boundary detection more effective, instead of deciding a frame to be shot boundary frame, by just looking into the curvature corresponding to the frame, we also use the region of support and the confidence value at the point. For a frame marking the shot boundary, the three features, the confidence value, region of support and the curvature are expected to be larger than that of the neighbour frames as shown in Fig. 1-5 for first few frames of a video. Following some experimentation it was observed that the shot cuts and flash are evident and easy to detect by curvature , although it can be ascertained with region

132

Punitha P. and J.M. Jose

Fig. 1. Frame Difference

Fig. 2. Confidence values for first 1000 frames of a video

Fig. 3. Region of support for first 1000 frames of a video

Shot Boundary Detection Based on Eigen Coefficients and Small Eigen Value

Fig. 4.

133

- Curvature values helpful to detect shot cut

Fig. 5.

- Curvature values for gradual change

of support and confidence value. However, when it is detection of shot boundaries due to gradual changes in the frames, the curvature alone is not sufficient. Therefore, selecting points on the boundary curve, corresponding to frames with local maximum curvature and local maximum confidence values, with a longer region of support becomes essential and effective.

4 Performance Metric To test the robustness and effectiveness of the proposed algorithm, we manually identified video shot boundaries in a number of videos, this will be used as reference for classifying the type of detection as correct, false or missed. Accordingly, #Total represents the number of shot boundaries, #Correct represents the number of events

134

Punitha P. and J.M. Jose

correctly detected, #False represents the number of detected shot boundaries, which do not actually represent the shot boundaries and #Missed represents the number of undetected shots. Two performance metrics, the hit rate (HR) and error rate (ER) are used to evaluate our proposed methodology. Hit rate and error rate are given in (7) and (8) below # Correct HR = (7 ) # Total ER =

# Missed + # False # Total

(8 )

5 Experimental Results The approach presented in this paper was applied on two genres of videos, news and commercial TV program sequences. The video sequences included cut, fade, dissolve (blur and mix), and wipes. The sequences together lasted for 3739 secs, with 112086 frames. The dataset comprised of 79 cuts, 11 flashes, 45 fades, 113 crossfades (mix, blur and dissolve), and 5 wipes. Table 1. Shot cut detection rate

A B C D

E

Feature used + confidence value

HR 0.42

FR 0.57

+ confidence value

0.5 0.57 0.42 1

0.85 0.64 0.57 0.42

Region of support + confidence value Union of above results

The frame difference between the subsequent frames of a video was computed using eigen distance computation as explained in Section 3.1. The first frame index and its eigen distance to the next frame was regarded as a point on a open contour. Now the task reduces to finding the points on this contour which could be the possible shot cuts or corner points in object detection domain. For every point which represents a frame from a video, we computed the three features, (i) the region of support (ii) the curvature at p, formed by the two arms and (iii) the confidence value, as described earlier are extracted as explained in Section 3. Using these three features, for the video considered for experiments, we detected all 79 cuts with , with 100% accuracy, but only a very few flash, fade and dissolve shot boundaries were detected. When the common peaks were used in features, and the confidence value, we were able to find more boundaries caused due to fades. But, this also resulted with more fault detection. Using in combination to peak points of confidence, not only gave the shots undetected in previous setups, but had minimal fault rate. However, a small number of shot boundaries remained undetected. The combination of a longer region

Shot Boundary Detection Based on Eigen Coefficients and Small Eigen Value

135

of support and the confidence values was able to detect most of the undetected shots in the above setup. But, when all three features were used in combination, many shots went undetected. Instead of using all three features together, we aggregated the results of earlier mention combination and were able to detect all shots, but at additional fault rate. The results are tabulated in Table 1.

Fig. 6. Hit Rate(dotted line) and Error Rate(solid line) in boundary detection for various combinations of features

6 Discussion and Conclusion In this paper, we have made a successful attempt in exploring a model which overcomes the necessity of varying thresholds for shot boundary detection. The paper presents an adaptive and non-parametric approach for shot boundary detection, without limiting itself to a specific type of shot transition. The difference between two subsequent frames in the video is computed in terms of eigen distance between two subsequent frames of a video. The frame difference and the first frame number is regarded as the co-ordinates point on an open contour. The corner detection method is then used to determine the shot boundaries based on automatic computation of frame support, for each frame in the video using the statistical and geometrical properties associated with the small eigen value of the covariance matrix of a sequence of connected points on the open contour. The results obtained for a synthetic video of length 4000 secs, shown in Figure 6 depicts that we achieve a high detection rate when the results of all feature listed in the column ‘Features’ of Table 1 is used, however we also have a trade off with the false detection rate. As a future work, we aim to reduce the false shot boundary detection rate by considering other heuristics. In addition, though the method was not proposed to

136

Punitha P. and J.M. Jose

distinguish various types of transition, the experimental results have opened up the avenues to explore possible detection of type of transition with the help of above mentioned features.

Acknowledgement This research was supported by the EU commission, FP-027122-SALERO.

References 1. Dinesh, R., Guru, D.S.: Efficient Non-parametric corner detection: An approach based on small eigenvalue. In: IEEE Fourth Canadian conference on computer and robot vision (2007) 2. Fang, H., Jiang, J., Feng, Y.: A fuzzy logic approach for detection of video shot boundaries. Pattern Recognition 39, 2092–2100 (2006) 3. Fang, H., Jiang, J.: Predictive based cross line for fast motion estimation in MPEG-4 videos. In: Proceedings of IS&T/SPIE’s symposium on Electronic Imaging: Science & Technology, San Jose, USA, January 2004, pp. 175–183 (2004) 4. Feldman, J., Singh, M.: Information along contours and object boundaries. Psychological Review 112, 243–252 (2005) 5. Goyal, A., Punitha, P., Hopfgartner, F., Jose, J.M.: Split and Merge Based Story Segmentation in News Videos. In: Boughanem, M., et al. (eds.) ECIR 2009. LNCS, vol. 5478, pp. 766–770. Springer, Heidelberg (2009) 6. Guimaraes, S.J.F., Couprie, M., Araujo, A.D.A., Leite, N.J.: Video segmentation based on 2D image analysis. Pattern Recognition Letters 24, 947–957 (2003) 7. Guru, D.S., Dinesh, R.: Non-parametric adaptive region of support useful for corner detection: a novel approach. Pattern Recognition 37, 165–168 (2004) 8. Hanjalic, A.: Shot boundary detection: unravelled and resolved? IEEE trans. Circuits Syst., Video Technol. 12(2) (2002) 9. Huang, C., Liao, B.: A robust scene-change detection method for video segmentation. IEEE trans. Circuits Syst., Video Technol. 11(12) (2001) 10. Jadon, R.S., Chaudhury, S., Biswas, K.K.: A fuzzy theoretic approach for video segmentation using syntactic features. Pattern Recognition Letters 22, 1359–1369 (2001) 11. Kim, W.H., Jeong, Y.J., Moon, K.S., Kim, J.N.: Adaptive shot change detection technique for real time operation on PMP. In: Proc. of third Intnl. Conf. on Convergence and Hybrid Technology, pp. 295–298 (2008) 12. Liu, T.Y., Lo, K.T., Zhang, X.D., Feng, J.: A new cut detection, algorithm with constant false alarm ratio for video segmentation. J. Vis. Commun. Image R. 15, 132–144 (2004) 13. Lo, C., Wang, S.: Video segmentation using a histogram based fuzzy c-means clustering algorithm. Computing, Standards and Interface 23, 429–438 (2001) 14. Porter, S., Mirmehdi, M., Thosmas, B.: Temporal video segmentation and classification of edit effects. Image and Vision Computing 21, 1098–1106 (2003) 15. Punitha, P., Urruty, T., Feng, Y., Halvey, M., Goyal, A., Hannah, D., Klampanos, I., Stathopoulos, V., Villa, R., Jose, J.: Glasgow University at TRECVID 2008, TRECVID workshop at NIST Gaithersburg, MD, USA (2008) 16. Teh, C.H., Chin, R.T.: On the detection of dominant points on digital curves. IEEE trans. Pattern Analysis and Machine Intelligence 11, 856–859 (1989) 17. Whitehead, B.P., Laganiere, R.: Feature based cut detection with automatic threshold selection. In: Enser, P.G.B., Kompatsiaris, Y., O’Connor, N.E., Smeaton, A., Smeulders, A.W.M. (eds.) CIVR 2004. LNCS, vol. 3115, pp. 410–418. Springer, Heidelberg (2004)

Shape-Based Autotagging of 3D Models for Retrieval Ryutarou Ohbuchi and Shun Kawamura Graduate School of Medicine and Engineering, University of Yamanashi, 4-30 Takeda, Kofu-shi, Yamanashi-ken, 400-8511, Japan [email protected], [email protected]

Abstract. This paper describes an automatic annotation, or autotagging, algorithm that attaches textual tags to 3D models based on their shape and semantic classes. The proposed method employs Manifold Ranking by Zhou et al, an algorithm that takes into account both local and global distributions of feature points, for tag relevance computation. Using Manifold Ranking, our method propagates multiple tags attached to a training subset of models in a database to the other tag-less models. After the relevance values for multiple tags are computed for tag-less points, the method selects, based on the distribution of feature points for each tag, the threshold at which the tag is selected or discarded for the points. Experimental evaluation of the method using a text-based 3D model retrieval setting showed that the proposed method is effective in autotagging 3D shape models. Keywords: 3D geometric modeling, content based retrieval, automatic annotation, manifold ranking, text tags, semantic retrieval.

1 Introduction Specification of a query is one of the most fundamental issues in retrieving multimedia objects such as images and 3D geometric models. These data objects may be searched either by example(s) or by text(s), each approach having its own advantages and disadvantages. To retrieve 3D models, a query for the Query-By-Example (QBE) approach could be sketches in either 2D [12] or 3D of the desired shape, an example 3D model, or a photograph of a real-world object having the desired shape. Most of the published work on 3D model retrieval used the QBE approach [2, 5, 6, 7, 9, 10, 12, 14]. While the QBE approach is quite powerful and useful in many application scenarios, it does have drawbacks. For example, it is often impractical to find a 3D model that have a shape similar enough to the desired one. Or, a user may find sketching a complex shape difficult. Capturing semantic aspect of the desired shape is also difficult with the QBE approach. Previous work used either single-class learning via relevance feedback [6, 7] or off-line multi-class learning [10] for semantics. An alternative, Query-By-Text (QBT) approach, employs a text as a query, as in text-based search engines. While the QBT is disadvantageous in directly specifying a desired shape, it has an advantage in describing semantic content of the desired shape. T.-S. Chua et al. (Eds.): SAMT 2009, LNCS 5887, pp. 137–148, 2009. © Springer-Verlag Berlin Heidelberg 2009

138

R. Ohbuchi and S. Kawamura

For example, a word “mammal” could be used to retrieve both dolphin and cat that have very different geometric shapes. The issue in QBT of 3D models is that most of the 3D models are not associated with text tags, and that adding consistent tags manually to a large number of 3D models can be quite difficult. An automatic or semiautomatic approach, e.g., propagation of tags from a set of tagged models to a large number of tag-less models, is necessary. The authors know of only two examples, one by Zhang et al [16] and the other by Goldfeder et al [3] that studied autotagging of 3D models. The method by Zhang et al [16] associates a list of attributes, not text tags, with a 3D model. The method employs biased kernel regression to propagate probabilities of the attributes from the manually tagged source models to the tag-less models. For retrieval, the method treats the list of probabilities as a feature vector, and computes distances among feature vectors. Thus, it is not strictly a QBT method. The method by Goldfeder et al [3] is a QBT approach, in which text tags are propagated from tagged 3D models to tag-less 3D models. To propagate tags, it uses local structure of the shape feature space via nearest neighbor search. For experiments, they used a set of 192,343 mesh-based 3D models having text tags in a snapshot of Google 3D Warehouse (G3W). The G3W is an evolving set of a large number of 3D models contributed by users, and it contains errors and inconsistencies in its tags. Unfortunately, the G3W snapshot they have used is not available for use at this time. Goldfeder et al evaluated tagging performance of their method by using a retrieval task, by comparing retrieval performance of their auto-tagging based QBT algorithm with that of a QBE algorithm that uses 3D models as its queries. The paper [3] reported that the QBT performed equal or better than the QBE algorithm they have compared against. In this paper, we propose a novel off-line semi-supervised algorithm for autotagging 3D model collections that exploits both local and global distribution of tagged and tag-less shape features by means of the Manifold Ranking (MR) algorithm by Zhou, et al. [17, 18]. From a set of shape features extracted from a set of tagged 3D models that shares a same text tag, the MR algorithm diffuses a relevance rank value indicating the likelihood of tag-less models to have the tag. This MR process is performed once for each of keyword to determine a set of tags a tag-less model should receive. To discard an unreliable tag attached to the tag-less model, compactness of the distribution of the tagged models sharing the tag is taken into consideration. An evaluation of the proposed algorithm by using a 3D model retrieval scenario showed that a QBT retrieval method by using the text tags added by the method significantly outperforms a QBE retrieval method that used 3D shape feature comparison.

2 Autotagging 3D Models via Multiple Manifold Ranking Our algorithm, whose outline is shown in Figure 1, consists of the following three steps. Details of the steps will be described in the following subsections. (1) (2)

Shape feature extraction: Extract a shape feature from the input 3D model. Tag propagation: For each tag, propagate, in the shape feature space, the likelihood of a 3D model having the tag from the tagged “training example” 3D models to the tag-less models. The tag likelihood propagation is performed by

Shape-Based Autotagging of 3D Models for Retrieval

139

using the Manifold Ranking (MR) algorithm by Zhou, et al [17, 18]. Repeat the MR-based tag likelihood propagation for all the tags. (3) Tag selection: A tag-less model now has multiple likelihood values for multiple tags. Select, based on the mutual distance of the tagged and tag-less models, the most likely tags to be attached to the model. Extract feature

Estimate TRR

Select tags Chair:0.5

Untagged model 0.3 0.5

0.5

Feature space Tagged models

Fig. 1. The method estimates a Tag Relevance Rank (TRR), a likelihood of a model having a text tag, by Manifold Ranking [17, 18]. Estimation of TRR is performed for every tag. Then most relevant of the tags are selected for a 3D model by using tag-specific threshold values.

2.1 Feature Extraction So far as a shape descriptor, or a feature, of a 3D model is a vector, it can be used in the proposed autotagging algorithm. A feature for comparing 3D shapes is required to have a set of invariances. Typically, certain invariance to shape representation, invariance to geometrical transformation up to similarity transformation, and invariance to geometrical and topological noise are expected. One might need to compare a CAD model defined as a 3D solid by using curved surface patches with a polygon soup model or a point set model. Invariance against three rotational degrees of freedom is not trivial to achieve [14]. Topological and geometrical noises, such as holes, variation in vertex connectivity in a mesh, displacements of vertices, need to be tolerated. Depending on an application, additional invariance, such as invariance against joint articulation, may be required. For the experiment described in this paper, we used our extension of Wahl’s Surflet Pair Relation Histograms (SPRH) [15], a shape feature that has nice invariance properties, readily available, and has reasonable retrieval performance. The Windows 32bit executable of our extension of the SPRH is available at our web site [8]. The SPRH was originally developed for oriented point set models acquired by Laser range scanners. We added a step to convert a surface based model to an oriented point-set model by using quasi-Monte Carlo sampling of surfaces [10]. As a result, the extended SPRH is able to accept popular surface-based shape representations such as polygon soup and closed polygonal mesh. The SPRH feature is a joint histogram of quadruples (δ, α, β, γ), in which δ is the distance and α, β, and γ are the angles between a pair of oriented points. By default, each of the four quantities has 5 bins, making a SPRH feature a ͷସ ൌ ͸ʹͷ‫ ܦ‬vector. The SPRH is invariant to similarity

140

R. Ohbuchi and S. Kawamura

transformation without requiring pose normalization. It is also insensitive to various noises, such as variation in mesh tessellation or connectivity, holes, or small vertex displacements. 2.2 Tag Propagation and Selection Our method uses the Manifold Ranking (MR) [17, 18] algorithm to estimate the relevance of a text tag to tag-less 3D models. The MR algorithm is a graph-based learning algorithm that can be used either in unsupervised, supervised, or semi-supervised modes. Our method uses the MR in semi-supervised mode. Intuitively, the MR resembles to solving a diffusion equation on an irregular connectivity mesh in a highdimensional feature space. The mesh is created by connecting, by their proximity, the feature points in the feature space. For example, given the 625D feature of the SPRH, the mesh is embedded in a 625D space with its feature points having 625D coordinate. In our autotagging framework, the mesh for the MR is created from union of feature vectors of the tagged as well as tag-less 3D models. Typically, a text tag is associated with multiple tagged (or “source”) 3D models so that a MR process would have multiple sources for diffusion. Given N text tags, or keywords, the proposed algorithms runs N such MR processes to diffuse Tag Relevance Rank (TRR) value for the N tags onto all the tag-less 3D models. At the equilibrium of solving the diffusion equation iteratively, the higher the diffused TRR value, the higher the likelihood of the tag-less model having the tag. Let us assume propagation of a tag from a set of tagged 3D models to the other 3D models without the tag. Let χ = { x1 ,L, xs , xs +1 ,L, xt , xt +1,L, xn } be a set of n features in a m-dimensional feature space \ m , in which first t points from x1 to xt are the feature points for the tagged models. Among the tagged points, points from x1 to xs are the source set, and the point from xs +1 to xt are the probe set. This splitting of the tagged points is done to estimate a TRR threshold value th, which decides if a tag should be attached to a tag-less point or not. The source set and the probe set is of equal size, and are drawn randomly from the set of t points sharing the same tag. To put it briefly, by performing TRR diffusion within a set of models sharing a tag, the tightness of the distribution of the tag is estimated to compute the th. Details on tag selection will be explained below. The rest of the points from xt +1 to xn are the tag-less ones for which the TRR values need be computed. Let d : F u F o \ denote a distance metric on χ , e.g., L1-norm or Cosine distance, that assigns a pair of points xi and xj a distance d ( xi , x j ) . Let f : F o \ be a ranking function that assigns each xi a ranking score T fi , thereby forming a rank vector f = ⎣⎡ f1 ,L , f n ⎦⎤ . Let the n-dimensional binary-valued T vector y = ⎣⎡ y1 ,L , yn ⎦⎤ be a label vector, in which yi = 1 for the tagged (“source”) point and yi = 0 for the rest, i.e., points to which tags are assigned. We first create the affinity matrix W where Wij indicates the similarity between samples xi and x j ;

⎧ ⎛ d ( xi , x j ) ⎞ ⎪ exp ⎜⎜ − ⎟⎟ if i ≠ j Wij = ⎨ σ ⎝ ⎠ ⎪ otherwise ⎩ 0

(1)

Shape-Based Autotagging of 3D Models for Retrieval

141

The distance metric d(xi, xj) used in forming W affects ranking performance. We will compare several distance measures for their performance in Section 3.1. The positive parameter σ defines the radius of influence. Note that Wii = 0 since there is no ark connecting a point with itself. The matrix W is positive symmetric. A normalized graph Laplacian S is then defined as; −

1

S = D 2 WD

−

1 2

(2)

where D is a diagonal matrix in which Dij equals to sum of i-th row of W , that is,

Dij = ∑ j Wij Then the ranking vector f = ⎣⎡ f1 ,L , f n ⎦⎤ can be estimated by iterating T

the following until convergence;

f (t +1) = α Sf (t ) + (1 − α ) y

(3)

As the equilibrium is reached, each tag-less point would have a TRR value associated with them, indicating the likelihood of the point having the tag. We call the ranking vector at the equilibrium f * . The algorithm performs the tag propagation process described above n times, once for each of the n tags. After all the tags have been diffused, a point in the feature space would have n positive TRR values corresponding to the n tags. A TRR value ݂௜ indicates how relevant the tag is to the point i. The system must now decide what subset of the n tags should be attached to the point. To decide, our method employs a simple proving to estimate the cut-off threshold th of the TRR for each tag. As described above, the method randomly splits the set of t tagged points sharing the same tag into two (roughly) equal-sized subsets, the source set and the probe set. After the MR, the TRR values of the points in the probe set, that is, the points from xs +1 to xt in the set of features χ , are used to estimate a threshold th by using the following equation; ‫ ݄ݐ‬ൌ ‹ ሺ݂௜ ሻ ௦ାଵஸ௜ஸ௧

(4)

The threshold th is computed for each tag. If the TRR ݂௜ ൒ th at a tag-less point, the tag i is attached to the point. The tag is also given the TRR value ݂௜ as its confidence value at the point. In general, a point may be associated with multiple tags if more than one tag has TRR values greater than or equal to their respective threshold values. As a reviewer pointed out, the use of minimum to compute the threshold th is debatable, as it is not robust against outliers. However, given small size of some of the classes in the benchmark dataset, e.g., just 4 models per class and 2 models in its prove set, we could not come up with a better method that is statistically robust. In the example of Figure 2, tagged points belonging to the tag “car” has the tightest distribution, with the similarity threshold th=0.8. The TRR of the “car” tag at the tagless point in question is 0.5, which is below the threshold th=0.8. Thus, the “car” tag is not attached to the point. Similarly, the “animal” tag is not attached to the point. The “chair” tag, on the other hand, has the TRR=0.5, while the threshold for the “chair” tag is th=0.4. Thus, the point is associated with the tag “chair”.

142

R. Ohbuchi and S. Kawamura

Tag Propagation animal

Tag-less model

Tag Selection animal: th=0.6 TRR=0.3 TRR=0.5

Tag(s): animal car chair

TRR=0.5 car

chair

car: th=0.8

chair: th=0.4

Fig. 2. A tag is attached to a feature if its Tag Relevance Rank (TRR) propagated from features with the tag exceeds the threshold th specific to the tag. The threshold is the estimated minimum TRR of the prove set of features having the tag.

3 Experiments and Results As the research on autotagging, or automatic annotation of 3D models just started, there is no established benchmark for an objective evaluation. Ideally, one should use a large enough corpus, something similar to the snapshot of G3W containing 190k models used by Goldfeder et al, for tagging. Unfortunately, the G3W snapshot used by Goldfeder et al is not available for us. In addition to the database, one also needs a method to evaluate the tagging performance numerically and objectively. We resorted to using a well accepted benchmark for QBE 3D model retrieval, the Princeton Shape Benchmark (PSB) [13] for our evaluation. The 1,814 model PSB contains two equal-sized subsets, the train set and test set, each consisting of 907 models. We regard class labels of the PSB train set as text tags, and transfer the tags from the train set to the test set. We then regarded the retrieval performance derived under the QBT framework as the performance index of the autotagging method. The PSB has a 4-level class hierarchy, from the most detailed “Base” level classes to the most abstract “Coarse 3” level classes containing only two classes; “natural” and “manmade”. Of these four levels, we use the Base level classes only in the following experiments. The Base level class itself has four internal levels of hierarchy. If leaf nodes of the Base level are counted, there are about 90 classes in the Base level of both train and test set. Note that some of the classes in the Base class of the train set do not have their counterparts in the Base level of the test set, and vice versa. If both leaf and non-leaf nodes of the Base level classes are counted there are about 130 classes. (To be exact, there are 129 classes in the train set and 131 classes in the test set.) Note also that the size of a class at the leaf-node of the Base class can be quite small, ranging from just four per class to a couple of dozen per class. As the feature vector, we used the SPRH by Wahl [15] modified to accept polygon soup and polygonal mesh models found in the PSB. Note that the performance of the SPRH is only modest by current state of the art. Here we do not aim for the highest retrieval performance, but we want to evaluate our autotagging algorithm by using a QBT framework.

Shape-Based Autotagging of 3D Models for Retrieval

143

3. 1 Distance Measures for Similarity Matrix This set of experiments compares, using the retrieval performance, the autotagging performance of various distance measures used in forming the affinity matrix W in equation (1). For this experiment, we used 63 class labels that are common among the train and test sets of the PSB. These 63 classes are selected from the Base level, and include both leaf and non-leaf nodes in the Base level. These 63 classes correspond to 811 models in the train set and 823 models in the test set. We used the 3D models in the train set as the tagged models, and propagated their class labels to the models in the test set. We compared five distances measures, that are, Lk -norm having k=0.5, 1.0, and 2.0, the cosine (cos) measure, and the Kullback-Leibler divergence (KLD). In the following equations, x = ( xi ) and y = ( yi ) are the feature vectors and n is the dimension of the vectors. The Lk -norm is defined by the following equation; ⎡ d k ( x, y ) = ⎢ ⎣

∑( n

i

xi − yi

k

)

⎤ ⎥⎦

1

k

(5)

The original paper on MR used the Euclidian distance, or L2 -norm with k=2.0, to form the affinity matrix [17, 18]. The Manhattan distance with k=1.0 is often said to perform better in higher dimension. Aggarwal, et al in [1] argued that k 3 and it appears to decrease the precision of the first two results. From an analysis of the documents retrieved, it seems that, with the second iteration, the context is somehow expanded. That is, at least with the users considered, the level 2 context appears to be less specific than the level 1, leading to the loss of precision. In order to analyze the phenomenon further, we selected a few queries and repeated the procedure up the level 4 context. The results are shown in figure 2 We can notice 1.00

qa

0.00 0.00

q a

a q

q

a

qa

q

a q relational a rewriting active bug address

q a

6.00 1.00

qa

a q qa q neighbouring a concentration nerve weak disorders

6.00

Fig. 2. Precision of the results for the computing context (left) and the neurophysiology context (right) as a function of the level of context

that the presence of “bumps” in the context is quite frequent, although the general trend, as expected, it towards an increase in precision with the context level. Here, too, we observe a phenomenon that we already pointed out in [4]: certain queries, mainly in the neurophysology context, are very specific and carry with them enough context information that adding context results only in marginal improvement. The query “nerve” is an example of this phenomenon. The precision seems to plateau after three or four iteration, indicating that at that time the context is quite formed and sufficient to specify the semantics of the desired query.

References 1. Kaski, S.: Computationally efficient approximation of a probabilistic model for document representation in the WEBSOM full-text analysis method. Neural Processing letters 5(2) (1997) 2. Kohonen, T.: Self-organizing maps. Springer, Heidelberg (2001) 3. Santini, S.: Ontology: use and abuse. In: Boujemaa, N., Detyniecki, M., N¨urnberger, A. (eds.) AMR 2007. LNCS, vol. 4918, pp. 17–31. Springer, Heidelberg (2008) 4. Santini, S., Dumitrescu, A.: Context as a non-ontological determinant of semantics. In: Duke, D., Hardman, L., Hauptmann, A., Paulus, D., Staab, S. (eds.) SAMT 2008. LNCS, vol. 5392, pp. 121–136. Springer, Heidelberg (2008)

OntoFilm: A Core Ontology for Film Production Ajay Chakravarthy, Richard Beales, Nikos Matskanis, and Xiaoyu Yang IT Innovation Centre, 2 Venture Road, Southampton, SO16 7NP United Kingdom {ajc,rmb,nm,kxy}@it-innovation.soton.ac.uk

Abstract. In this paper we present OntoFilm, a core ontology for film production. OntoFilm provides a standardized model which conceptualizes the domain and workflows used at various stages of the film production process starting from pre-production and planning, shooting on set, right through to editing and post-production. The main contributions in this paper are: we discuss how OntoFilm models the semantics necessary to interpret these workflows consistently for all users (Directors, DoP’s, grips, post-production, lighting). We also discuss how our ontology forms a common bridge between the low level descriptive metadata generated for the video footage and the high level semantics used in software tools during the production process. Keywords: Ontologies, Semantic Web, Film Production, Post Production, Digital Media, Semantic Gap.

1 Introduction The process of film production is lengthy, time consuming and often an expensive affair. Film directors currently rely on paper based storyboards or manually-created ‘pre-vis’ animations to record and express their creative intent on how a scene could be shot. There is no formal way to describe this intent. Similarly in post production a tremendous amount of metadata is available describing the footage that has been captured, but this is in the form of paper notes and must be sorted and input into the postproduction software manually. This is a time-consuming, laborious and error-prone process. There is a need for a standardized knowledge model which will act as a common knowledge sharing platform between the software tools used throughout film production. This research is being carried out as part of the ANSWER1 project. ANSWER is a novel approach to the creative process of film making. ANSWER introduces a symbolic language for directors to create detailed scene descriptions, much like a musician would use music scores to compose a song. ANSWER also aims to provide tools and technologies which will enable scene authoring through filming notation. Users (directors) will be able to define and visualize film scenes before shooting begins. The aim of our research within ANSWER is to create a conceptual knowledge model which facilitates knowledge flow among the various software components used and is a first step towards bridging the semantic gap between high level pre-production 1

www.answer-project.org/

T.-S. Chua et al. (Eds.): SAMT 2009, LNCS 5887, pp. 177–181, 2009. © Springer-Verlag Berlin Heidelberg 2009

178

A. Chakravarthy et al.

concepts and low level post-production metadata. In its initial stages OntoFilm is mainly concerned with modelling film production. However, the film and games industries are converging in the way that their creative content is authored, so we will extend the results to include the needs of the games industry, thus offering a bridge between digital media production and animation for game design. In this paper we will present how we have attempted to model the workflow in three main stages of film production, i.e. pre-production, on-location, and post-production. Prior to starting development on OntoFilm we have studied state of the art multimedia ontologies such as MPEG-7 [2] and COMM [1]. We have reused concepts from the MPEG-7 ontology (e.g. for Camera Motion, Motion Trajectories and Motion Activity).

2 Pre-production Model The pre-production model covers the concepts used during the planning stage of the film. Script writers and film directors use paper based scripts and story boards to plan shots within the film. Often directors add additional notes on script elements to record their creative ideas. This additional metadata is very useful in post-production when locating footage and during video and dialogue synchronization. However paper based scripts cannot be used directly to produce machine understandable metadata. OntoFilm introduces the concept of Shooting Script element. After consultation with directors and analyzing existing film and television scripts we have identified the main elements of a movie script and their relationships such as Scene, Shot, Character Name, Dialogue, Parenthetical and Transition Element. The Movie Script Markup Language (MSML) is a movie script schema specification developed as part of the ANSWER ontology specification after studying existing schemas used within the film domain such as Celtx2. MSML is encoded within OntoFilm and provides the semantics necessary for software tools to annotate and parse movie scripts according to this schema. Figure 1 shows the conceptual schema of MSML.

Fig. 1. Movie Script Markup Language conceptual schema

2

http://wiki.celtx.com/index.php?title=The_Script_Editor

OntoFilm: A Core Ontology for Film Production

179

The graphical user interface developed within ANSWER allows directors to visualize and annotate scripts just as they would on a paper based script, the difference being all the concepts being annotated are persisted as OWL instances. Further the director can select any element on the script and add free text annotations to record his/her creative intent. Other pre-production concepts include various physical setups required on set such as Optical Attachments, Optical Components, Costume, Props, Shots (e.g. long shot, POV shot, medium shot etc). Detailed descriptions of the aforementioned classes are published in [4]. We have done initial experimentation with SWRL3 based rule modelling e.g. for automatic classification of the different Cameras used during filming. Figure 2 shows a rule for classifying Film Camera.

Fig. 2. SWRL rule showing classification conditions for Film Camera

The isFilm_Camera rule classifies the Camera instance according to if it has Emulsion Film as capturing media. The camera’s hasCropFactor data property can then be calculated by taking into account its classification, film or sensor size by the calcFilmCropFactor and calcDigitalCropFactor.

3 On-Set Production Model Directors often follow a set of filming conventions while shooting a scene. Although these are not mandatory, these rules are recommended to avoid causing confusion to the viewing audience. In OntoFilm we have conceptualized these rules with Film Grammar concepts. A few examples of these rules include: • •

180 Degree Rule: This rule deals with the spatial relation between characters on-screen. It is used to maintain consistent screen direction between characters or between a character and an object. 30 Degree Rule: If a shot is going from one character to another without an intervening shot of something else, the camera angle should change by at least 30 degrees.

The on set production model also includes the Director Notation (DN) ontology. DN is a symbolic language which can be used to describe detailed scene descriptions during film shooting [3]. The graphical user interface developed in ANSWER will allow directors to create detailed scene descriptions using DN. A rule engine automatically translates these DN scores and generates animates pre-visualizations. The DN ontology is a system level ontology which acts as a bridge between the user enabled tools for using DN to describe a scene and the rule engine which translates the DN scores into animations. Details of the DirectorNotation ontology will not be expanded since it’s outside the scope of this paper. 3

http://www.w3.org/Submission/SWRL/

180

A. Chakravarthy et al.

4 Post-Production Model Our approach to modelling Post-Production for OntoFilm has been to focus on the workflow involved, as it was considered this would draw attention to the points at which metadata was lost or needed to be transferred manually. Significant amount of machine-readable metadata is automatically generated in post production to describe the footage from the point at which it is ingested (digitised), e.g. descriptions of media on which the files were held and software processes applied to alter the visual appearance of the footage. Furthermore, low-level video features, such as regions of colour or movement, can both be annotated automatically using the MPEG-7 standard. However there is a stark disconnect between these various technical parameters and the high-level features that they contain or represent. The result is that post-production operators can easily identify footage that had been scanned by a particular machine, but not find a sequence of shots that contained a scene description e.g. protagonists in a car chase – this information is buried in the reams of paper notes that accompany the production4. We have modelled the various post-production steps, e.g. Edit_Room (which involves assembly of selected material from video footage produced on a day to day basis), Commercial_Preview (where a candidate edit is screened in a cinema to judge audience reaction), and linked them not only technical descriptions of the media involved, but also to scene descriptions from the On-Set Production model and planning notes from the Pre-Production model. In this way, it is now possible to construct a unified semantic representation of the action portrayed on scene, the actors, props and locations involved, the planning rationale for creating the scene and the technical means with which it was acquired and the processing applied in post. For example the low level footage descriptor Scene Tint (which is the predominant colour of the shot) relates to the high level concept of Content (which describes the mood the scene).

5 Conclusions and Future Work The OntoFilm ontology is being developed after direct consultations with film directors (STEFI5 Productions), post production software developers (DFT6) and game developers (Larian7 Studios). The feedback from these collaborative meetings confirmed the need for development of a common ontological framework for the digital media domain, which will act as a bridge for knowledge transfer between these domains consistently for all users. We are currently investigating integrating OntoFilm with the COMM API. The COMM ontology combines the advantages of extensibility and scalability of a web-based solution with the accumulated experience of MPEG-7 [1]. The main advantage of doing this would be software programs will be easily able to access the ontology through programmatic interfaces. We are also looking at 4

5 6 7

This information has been gathered after having technical meetings with post production teamDFT Bones@Weiterstadt, Germany. http://www.stefi.gr/ http://www.dft-film.com/ http://www.larian.com/

OntoFilm: A Core Ontology for Film Production

181

current efforts in the field of digital media ontologies, namely the iMP8 and Salero9 projects. However our work differs from these multimedia ontologies because OntoFilm is specific to film production and covers this domain in great depth with a rich set of semantics. Software programs developed for digital media production will be able to leverage our ontology directly.

References 1. Arndt, R., Troncy, R., Staab, S., Hardman, L., Vacura, M.: COMM: Designing a WellFounded Multimedia Ontology for the Web. In: Aberer, K., Choi, K.-S., Noy, N., Allemang, D., Lee, K.-I., Nixon, L.J.B., Golbeck, J., Mika, P., Maynard, D., Mizoguchi, R., Schreiber, G., Cudré-Mauroux, P. (eds.) ASWC 2007 and ISWC 2007. LNCS, vol. 4825, pp. 30–43. Springer, Heidelberg (2007) 2. Martinez, M.J.: MPEG-7 Overview: International organization for standardisation (ISO//IEC/JTCI/SC29/WG11). Coding of moving pictures and audio. Palma be Mallorca (October 2004) 3. Yannopoulos, A., Savrami, K., Varvarigou, T.: DirectorNotation as a Tool for AmI & Intelligent Content: an Introduction by Example. Accepted for publication in the International Journal of Cognitive Informatics and Natural Intelligence, IJCiNi (2008) 4. D4.1B: Mid-Term Ontology Report, ANSWER Project Deliverable (May 2009)

8 9

http://imp-project.eu/ http://www.salero.eu/

RelFinder: Revealing Relationships in RDF Knowledge Bases Philipp Heim1 , Sebastian Hellmann2 , Jens Lehmann2 , Steffen Lohmann3 , and Timo Stegemann3 1

University of Stuttgart, Visualization and Interactive Systems, Universit¨ atsstr. 38, 70569 Stuttgart, Germany [email protected] 2 University of Leipzig, Agile Knowledge Engineering and Semantic Web, Johannisgasse 26, 04103 Leipzig, Germany {hellmann,lehmann}@informatik.uni-leipzig.de 3 University of Duisburg-Essen, Interactive Systems and Interaction Design, Lotharstr. 65, 47057 Duisburg, Germany {steffen.lohmann,timo.stegemann}@uni-due.de

Abstract. The Semantic Web has recently seen a rise of large knowledge bases (such as DBpedia) that are freely accessible via SPARQL endpoints. The structured representation of the contained information opens up new possibilities in the way it can be accessed and queried. In this paper, we present an approach that extracts a graph covering relationships between two objects of interest. We show an interactive visualization of this graph that supports the systematic analysis of the found relationships by providing highlighting, previewing, and filtering features. Keywords: semantic user interfaces, semantic web, relationship discovery, linked data, dbpedia, graph visualization.

1

Introduction

The Semantic Web enables answers to new kinds of user questions. Unlike searching for keywords in Web pages (as e.g. in Google), information can be accessed according to its semantics. The information is stored in structured form in knowledge bases using formal languages such as RDF 1 or OWL2 and consisting of statements about real world objects like ’Washington’ or ’Barack Obama’. Each object has a unique identifier (URI ) and is usually assigned to ontological classes, such as ’city’ or ’person’, and an arbitrary number of properties that define links between the objects (e.g., ’lives in’). Given this semantically annotated and linked data, new ways to reveal relationships within the contained information are possible. A common visualization for linked data are graphs, such as the Paged Graph Visualization [2]. In order to find relationships in these visualizations, users normally apply one of the following two strategies: They either choose a starting point 1 2

http://www.w3.org/RDF/ http://www.w3.org/TR/owl-features/

T.-S. Chua et al. (Eds.): SAMT 2009, LNCS 5887, pp. 182–187, 2009. c Springer-Verlag Berlin Heidelberg 2009 

RelFinder: Revealing Relationships in RDF Knowledge Bases

183

and incrementally explore the graph by following certain edges, or they start with a visualization of the entire graph and then filter out irrelevant data. Some more sophisticated solutions are based on the concept of faceted search. The tool gFacet [4], for instance, groups object data into facets that are represented by nodes and can be used to filter a user-defined result set. However, all these approaches require the user to manually explore the visualization in order to find relationships between two objects of interest. This kind of trial-and-error search can be very time consuming, especially in large knowledge bases that contain many data links. As a solution to this problem, we propose an approach that automatically reveals relationships between two known objects and displays them as a graph. The relationships are found by an algorithm that is based on a concept proposed in [5] and that can be applied to large knowledge bases, such as DBpedia [1] or the whole LOD-Cloud 3 . Since the graph that visualizes the relationships can still become large, we added interactive features and filtering options to the user interface that enable a reduction of displayed nodes and facilitate understanding. We present an implementation of this approach – the RelFinder – and demonstrate its applicability by an example from the knowledge base DBpedia [1].

2

RelFinder

The RelFinder is implemented in Adobe Flex4 and runs in all Web browsers with an installed Flash Player5. In the following, we first explain its general functionality before describing the involved mechanisms in more detail. The search terms that are entered by the user in the two input fields in the upper left corner (Fig. 1, A) get mapped to unique objects of the knowledge base. These constitute the left and right starting nodes in the graph visualization (Fig. 1, B) that get then connected by relations and objects found in between them by the algorithm. If a certain node is selected all graph elements that connect this node with the starting nodes are highlighted forming one or more paths through the graph (Fig. 1, C). In addition, further information about the selected object is displayed in the sidebar (Fig. 1, D). Filters can be applied to increase or reduce the number of relationships that are shown in the graph and to focus on certain aspects of interest (Fig. 1, E). 2.1

Disambiguation

Ideally, the search terms that are entered by the user can be uniquely matched to objects of the knowledge base without any disambiguation. However, if multiple matches are possible (e.g., in case of homonyms, polysemes, or incomplete user input) the user is supported by a disambiguation feature. Generally, a list of objects with labels that enclose the search terms is already shown below the 3 4 5

http://esw.w3.org/topic/SweoIG/TaskForces/CommunityProjects/ LinkingOpenData http://www.adobe.com/products/flex The current version of the RelFinder is accessible at http://relfinder.dbpedia.org

184

P. Heim et al.

Fig. 1. Revealing relationships between Kurt G¨ odel und Albert Einstein

input box while the user enters the terms (Fig. 1, A). This disambiguation list results from a query against the SPARQL endpoint of the selected knowledge base. The following code shows the DBpedia optimized SPARQL query for the user input ’Einstein’6 : SELECT ?s ?l count(?s) as ?count WHERE { ?someobj ?p ?s . ?s ?l . ?l bif:contains ’"Einstein"’ . FILTER (!regex(str(?s), ’^http://dbpedia.org/resource/Category:’)). FILTER (!regex(str(?s), ’^http://dbpedia.org/resource/List’)). FILTER (!regex(str(?s), ’^http://sw.opencyc.org/’)). FILTER (lang(?l) = ’en’). FILTER (!isLiteral(?someobj)). } ORDER BY DESC(?count) LIMIT 20

The disambiguation list is sorted by relevance using the ’count’ value (or alternatively a string comparison if ’count’ is not supported by the endpoint). ’count’ is also used to decide if a user’s search term can be automatically matched to an 6

A configuration file allows to freely define the queried endpoint, the element that is queried (typically ’rdfs:label’), and properties that should be ignored. It is also possible to deactivate specific syntax elements, such as ’count’ or ’bif:contains’, in case a SPARQL endpoint does not support these.

RelFinder: Revealing Relationships in RDF Knowledge Bases

185

object of the knowledge base or if a manual disambiguation is necessary. An automatic match is performed in one of the following two cases: 1) if the user input and the label of the most relevant object are completely equal, or 2) if the user input is contained in the label of the most relevant object and this object has a much higher count value than the second relevant object of the disambiguation list (ten times higher by default). Thus, the automatic disambiguation is rather defensive in order to prevent false matches. If the user does not select an entry from the disambiguation list and if no automatic match is possible, the entries from the disambiguation list are shown again in a pop-up dialog that explicitly asks the user to provide the intended meaning of the search term by selecting the corresponding object. 2.2

Searching for Relationships

A query building process composed of several SPARQL queries searches for relationships between the given objects of interest. Since the shortest connection is not known in advance, the process searches iteratively for connections with increasing length, starting from zero. As a constraint, the direction of the property relations within each connection chain is only allowed to change once. We defined this constraint due to performance reasons and because multiple changes in the direction of the edges are difficult to be followed and understood by the user. If our objects of interests are a and b this results in the following search patterns: a → ··· → b a ← ··· ← b a → ··· → c ← ··· ← b a ← ··· ← c → ··· → b Thus, we are looking either for one-way relationships (first two lines) or those with an object c in between such that there is a one-way relationship each from a and from b to c or from c to a and b (last two lines). Note that c is not known in advance but found within the searching process. The algorithm has several parameters: 1) It can be configured to suppress circles in extracted relationships. With the help of SPARQL filters, any object is only allowed to occur once in each connecting relationship. 2) Objects and properties can be ignored using regular expression patterns on their labels or URIs, which is useful if someone is not interested in certain objects or properties. More importantly, also structural relations between objects can be omitted, such as whether two objects belong to the same class or to the same part of a class hierarchy (i.e., ignoring rdf:type and rdfs:subClassOf properties). We decided to remove these by default, since they normally yield a multitude of relationships of minor interest which can be better explored in more traditional ways such a hierarchy browsers. 3) A maximum length of the returned relationships can be defined. 4) The SPARQL endpoint to use can be configured. An exemplary SPARQL query that searches for relationships of the type Kurt G¨ odel ← of 1 ← c → os1 → Albert Einstein is given here (filter omitted):

186

P. Heim et al.

SELECT * WHERE { db:Kurt_G¨ odel ?pf1 ?of1 . ?of1 ?pf2 ?c . db:Albert_Einstein ?ps1 ?os1 . ?os1 ?ps2 ?c . FILTER ... } LIMIT 20 2.3

Graph Visualization

The found relationships are added one by one to the graph, beginning with the shortest (i.e., direct relationships and relationships with only one object in between, if there are any). All objects are visualized as nodes connected by edges that are labeled and directed according to the property relation they represent (Fig. 1, B). Since the labels of the edges are crucial for understanding the relationships they serve as flexible articulations in the force-directed layout [3], what reduces overlaps but cannot completely avoid them. Interactive Features. To further reduce overlaps in the graph, we implemented a pinning feature that enables users to manually drag single nodes away from agglomerations and forces them to stay at the position they got dropped (pinned nodes are indicated by needle symbols as can be seen in Fig. 1, F). Especially in situations where many nodes are connected by many edges and thus are likely to overlap in the automatic layout, manual adjustments in combination with our pinning feature are helpful to produce an understandable graph layout that facilitates visual tracking. As already mentioned, visual tracking is additionally supported by the possibility of highlighting all paths that connect a selected node with the starting nodes (Fig. 1, C). If a certain node is selected in the graph, further information about the corresponding object is displayed in the sidebar (e.g., in case of DBpedia these are a title, short abstract, and image extracted from Wikipedia, Fig. 1, D). Moreover, the ontological class an object belongs to is highlighted in the list that is shown in the sidebar (Fig. 1, E). Vice versa, all corresponding nodes in the graph are highlighted if an ontological class is selected from the list. Filtering Options. The shown relationships can be filtered in two ways: 1) According to their length (i.e., the number of objects in between) and 2) according to the ontological classes the objects belong to. For instance, relationships consisting of several objects could be regarded as too far-fetched or objects belonging to certain classes might not be of interest for a user’s goals (Fig. 1, E) and are therefore removed from the graph. Filtering helps to reduce the number of displayed relationships in the graph and can hence prevent the graph from getting overly cluttered. For each search process, the filters are automatically set to initial values that avoid an overcluttered graph, if possible.

RelFinder: Revealing Relationships in RDF Knowledge Bases

3

187

Conclusion and Outlook

We introduced an approach in this paper that uses properties in semantically annotated data to automatically find relationships between any pair of user-defined objects and visualizes them in a force-directed graph layout. The RelFinder can therefore save a lot of time which would otherwise be lost in searching these relationships manually. Since the amount of found relationships can be large, we additionally provide two types of filters that can be used to reduce the displayed relationships according to their length and the ontological classes they belong to and thus allow focusing on only a relevant part of the relationships. Together with a pinning feature that lets users rearrange the graph layout manually, we therefore provide a semi-automatic approach. The basic mechanisms of the RelFinder work with every SPARQL endpoint and can therefore be applied to arbitrary knowledge bases with only little configuration effort. Since the SPARQL queries are composed on the client they are comparably independent of server routines and resources. We are currently testing the RelFinder on different knowledge bases in various scenarios. Future work includes a comprehensive evaluation that further examines the potentials and benefits of querying structured knowledge bases compared to common Web search with respect to the discovery and analysis of relationships between certain objects of interests.

References 1. Auer, S., Bizer, C., Kobilarov, G., Lehmann, J., Cyganiak, R., Ives, Z.: DBpedia: a nucleus for a web of open data. In: Aberer, K., Choi, K.-S., Noy, N., Allemang, D., Lee, K.-I., Nixon, L.J.B., Golbeck, J., Mika, P., Maynard, D., Mizoguchi, R., Schreiber, G., Cudr´e-Mauroux, P. (eds.) ASWC 2007 and ISWC 2007. LNCS, vol. 4825, pp. 722–735. Springer, Heidelberg (2007) 2. Deligiannidis, L., Kochut, K., Sheth, A.: RDF data exploration and visualization. In: Proceedings of the ACM first workshop on CyberInfrastructure 2007, pp. 39–46. ACM Press, New York (2007) 3. Fruchterman, T., Reingold, E.: Graph drawing by force-directed placement. Softw. Pract. Exper. 21(11), 1129–1164 (1991) 4. Heim, P., Ziegler, J., Lohmann, S.: gFacet: A browser for the web of data. In: Proceedings of the International Workshop on Interacting with Multimedia Content in the Social Semantic Web (IMC-SSW 2008), pp. 49–58. CEUR-WS (2008) 5. Lehmann, J., Sch¨ uppel, J., Auer, S.: Discovering unknown connections – the DBpedia relationship finder. In: Proceedings of the 1st SABRE Conference on Social Semantic Web, CSSW (2007)

Image Annotation Refinement Using Web-Based Keyword Correlation Ainhoa Llorente, Enrico Motta, and Stefan R¨ uger Knowledge Media Institute, The Open University, Walton Hall, Milton Keynes, MK7 6AA, U.K. {a.llorente,e.motta,s.rueger}@open.ac.uk

Abstract. This paper describes a novel approach that automatically refines the image annotations generated by a non-parametric density estimation model. We re-rank these initial annotations following a heuristic algorithm, which uses semantic relatedness measures based on keyword correlation on the Web. Existing approaches that rely on keyword cooccurrence can exhibit limitations, as their performance depend on the quality and coverage provided by the training data. Additionally, WordNet based correlation approaches are not able to cope with words that are not in the thesaurus. We illustrate the effectiveness of our Web-based approach by showing some promising results obtained on two datasets, Corel 5k, and ImageCLEF2009. Keywords: Automated image annotation, Normalized Google Distance, semantic similarity.

1

Introduction and Related Work

Automated image annotation refers to the process of learning statistical models from a training set of pre-annotated images in order to generate annotations for unseen images using visual feature extracting technology. The early attempts were focused on algorithms that explore the correlation between words and image features. More recently, there are some efforts [1,2,3,4,5,6,7,8] which attempt to benefit from exploiting the correlation between words computed using semantic similarity measures. According to the knowledge base used as source of information, this new generation of algorithms can be classified into those that use a corpus as the training set [6], those that employ a thesaurus such as WordNet [7], and those that perform the statistical correlation using the Web [8]. For each one of the mentioned categories, a variety of semantic similarity measures have been proposed. The most important limitation, affecting approaches that rely on a training set, is that they are limited to the scope of the topics represented in the collection. This information may not be enough for detecting annotations that are not correlated with others. For instance, if our collection contains a lot of images of animals in a circus and just a few of animals in the wildlife, the co-occurrence approach T.-S. Chua et al. (Eds.): SAMT 2009, LNCS 5887, pp. 188–191, 2009. c Springer-Verlag Berlin Heidelberg 2009 

Image Annotation Refinement Using Web-Based Keyword Correlation

189

will penalize combinations such as “lion” and “savannah” while promoting associations such as “lion” and “chair”, given that the training set is dominated by images of lions in a circus (i.e. consider a lion tamer controlling a lion with a chair). Additionally, WordNet based correlation approaches need to handle some words that do not exist or have no available relations with other words of the thesaurus. Our research follows the approach that uses Web-based semantic similarity measures in an attempt to overcome the limitations of approaches based on correlations in the training set or in WordNet. The semantic similarity measure used in this research is based on the Normalized Google Distance (NGD) between two terms x and y that was defined by Cilibrasi and Vit´ anyi [9] as: NGD(x, y) =

max{log f (x), log f (y)} − log f (x, y) , log N − min{log f (x), log f (y)}

(1)

where f (x) and f (y) are the counts for search terms x and y using Google and f (x, y) is, the number of web pages found on which both x and y occur. N is the total number of web pages searched by Google which, in 2007, was estimated to be more than 8bn pages. Cilibrasi and Vit´ anyi call the Normalized Web Distance (NWD) when they used any web-based search engine as source of frequencies. Gracia and Mena [10] applied a transformation on Equation 1 proposing their web-based semantic relatedness measure between x and y, as: relWeb(x, y) = e−2 NWD(x,y)

(2)

This transformation was done in order to get a proper relatedness measure that is a bounded value (in the range [0, 1]) and at the same time increases inversely to distance. The rest of this paper is organised as follows: Section 2 describes our algorithm while Section 3 shows our experiments and results. Finally, Section 4 contains our conclusions and plans for future work.

2

Image Annotation Refinement

The model used as baseline annotation is that developed by Yavlinsky et al. [11] who use global features together with a non-parametric density estimation. This algorithm yields a probability value, p(ωj |Ji ), of each keyword ωj being present in an image, Ji , of the test set, which is used as a confidence score for our candidate annotations. The candidate annotations are those five with the highest scores. Thus, the confidence score of keyword ωj is: Confidence score(ωj ) = p(ωj |Ji ) ≈ p(ωj |x1 , ..., xd ),

(3)

where x = (x1 , ..., xd ) is a vector of real-valued image features. As a result of our algorithm’s focus on refining baseline image annotations, we concentrate on

190

A. Llorente, E. Motta, and S. R¨ uger

test images with at least one accurate keyword among the candidates. In order to meet this requirement, we select images from the test set whose confidence score is greater than a threshold that is set after performing cross-validation on the training set. Then, we apply the semantic similarity measure defined in Equation 2 to pairs of candidate annotations until we detect candidates which are not semantically related to the others. Therefore, we reduce the confidence score of these “noisy” candidates. Finally, we get rid of these irrelevant annotations after re-ranking and selecting, again, the five with the highest confidences scores.

3

Experimental Work and Results

The proposed algorithm was evaluated on the Corel 5k and the ImageCLEF2009 collections (Table 1). The Corel 5k dataset is a collection with a training set of 4,500 images and a test set of 500 images. The vocabulary is made up of 374 words. The ImageCLEF2009 dataset is made up of 5,000 training images and 13,000 test images. The vocabulary consists of 53 words. In both cases, we followed a 10-fold cross-validation for tuning the system parameters. Performance is evaluated with the mean average precision (MAP), which is the average precision, over all queries, at the ranks where recall changes, where relevant items occur. Table 1. Results obtained for the two datasets Collection Metric Baseline Google Yahoo Corel 5k MAP 0.2861 0.2882 0.2901 ImageCLEF2009 MAP 0.2613 0.2736 0.2720

We represent our results in Table 1 where we have used two different Websearch engines. Independently of the search engine used, we observe an increment in the performance over the baseline. For the Corel 5k, an increment of 1.4% in the MAP is obtained while for the ImageCLEF2009 we get 4.7% increment over the baseline. In both cases, we used a combination of Tamura texture and CIELAB colour descriptors.

4

Conclusions and Future Work

We have presented a new approach in automated image annotation that prunes “noisy” keywords using semantic similarity measures by means of Web-search engines such as Google and Yahoo. One of the main advantages of using this approach compared to others is the flexibility that comes from the use of a Webbased search engine where you can type almost any word and you can expect to get some results. Another point is that it is not necessary to know the sense

Image Annotation Refinement Using Web-Based Keyword Correlation

191

of the word, contrary to WordNet, where a disambiguation task is necessary before estimating the semantic similarity. However, the most important benefit of this approach is that we are not limited to the scope of topics provided by a training set as it happens in statistical correlation cases. As future work, we plan to extend this approach and use Wikipedia as the source of background knowledge, a hybrid solution that can be considered a Web-based and a thesaurusbased measure. Acknowledgments. Thanks to Jorge Gracia for his helpful comments. This work was partially funded by the EU Pharos project (IST-FP6-45035) and by Santander Corporation.

References 1. Jin, R., Chai, J.Y., Si, L.: Effective automatic image annotation via a coherent language model and active learning. In: Proceedings of the 12th International ACM Conference on Multimedia, pp. 892–899 (2004) 2. Jin, Y., Khan, L., Wang, L., Awad, M.: Image annotations by combining multiple evidence & WordNet. In: Proceedings of the 13th International ACM Conference on Multimedia, pp. 706–715 (2005) 3. Liu, J., Li, M., Ma, W.Y., Liu, Q., Lu, H.: An adaptive graph model for automatic image annotation. In: Proceedings of the 8th ACM International Workshop on Multimedia Information Retrieval, pp. 61–70 (2006) 4. Zhou, X., Wang, M., Zhang, Q., Zhang, J., Shi, B.: Automatic image annotation by an iterative approach: incorporating keyword correlations and region matching. In: Proceedings of the International ACM Conference on Image and Video Retrieval, pp. 25–32 (2007) 5. Llorente, A., R¨ uger, S.: Using second order statistics to enhance automated image annotation. In: Boughanem, M., et al. (eds.) ECIR 2009. LNCS, vol. 5478, pp. 570–577. Springer, Heidelberg (2009) 6. Wang, C., Jing, F., Zhang, L., Zhang, H.J.: Image annotation refinement using random walk with restarts. In: Proceedings of the 14th annual ACM International Conference on Multimedia, pp. 647–650. ACM, New York (2006) 7. Jin, Y., Wang, L., Khan, L.: Improving image annotations using WordNet. In: Candan, K.S., Celentano, A. (eds.) MIS 2005. LNCS, vol. 3665, pp. 115–130. Springer, Heidelberg (2005) 8. Liu, J., Wang, B., Li, M., Li, Z., Ma, W., Lu, H., Ma, S.: Dual cross-media relevance model for image annotation. In: Proceedings of the 15th International Conference on Multimedia, pp. 605–614 (2007) 9. Cilibrasi, R., Vitanyi, P.: The Google similarity distance. IEEE Transactions on Knowledge and Data Engineering 19(3), 370–383 (2007) 10. Gracia, J., Mena, E.: Web-based measure of semantic relatedness. In: Bailey, J., Maier, D., Schewe, K.-D., Thalheim, B., Wang, X.S. (eds.) WISE 2008. LNCS, vol. 5175, pp. 136–150. Springer, Heidelberg (2008) 11. Yavlinsky, A., Schofield, E., R¨ uger, S.: Automated image annotation using global features and robust nonparametric density estimation. In: Proceedings of the International ACM Conference on Image and Video Retrieval, pp. 507–517 (2005)

Automatic Rating and Selection of Digital Photographs Daniel Kormann, Peter Dunker, and Ronny Paduschek Fraunhofer Institute for Digital Media Technology, Ehrenbergstrasse 31, 98693 Ilmenau, Germany [email protected], [email protected], [email protected]

Abstract. This paper presents methods for automatically rating and selecting digital photographs. The importance of each photograph is estimated by analyzing its content as well as its time-metadata. The presence of people is estimated by combining face and skin detection. Finally, the appeal of each photograph is calculated using a trained SVM classifier. The results of a conducted user study show that the automatically obtained rating coincides well with the perception of the test persons.

1

Introduction

The digitalization has led to an ever increasing size of personal photo collections. Today, consumers often take hundreds or even thousands of pictures of an event. For many intended purposes like the creation of slideshows or image galleries, a rating of the images would be desirable, so that the top-scoring images could be automatically selected. Existing approaches solve this task by screening for low quality images or by considering certain image content aspects like colorfulness [1] . Other works deal with the estimation of aesthetics in digital images [2]. Compared to the above mentioned work, our method focuses on the selection of the “best” images by considering various criteria.

2

Overview

We want to single out the aspects upon which a human observer would rate digital photographs. Based on a study of Savakis et al. [7] as well as our own conducted online survey, the following three criteria can be stated, which are all considered within our system, resulting in separate scores: Image appeal - Is the image appealing, is it a successful photography? Image importance - Does the image show an important subject or event? Presence of people - Does the image show people, friends and family? 2.1

Image Importance

In order to calculate an importance score for each image i, we consider three aspects: First, if something interesting is happening during an event, the photographic rate rises. Hence, we calculate the local photographic rate fi for each T.-S. Chua et al. (Eds.): SAMT 2009, LNCS 5887, pp. 192–195, 2009. c Springer-Verlag Berlin Heidelberg 2009 

Automatic Rating and Selection of Digital Photographs

193

image i using a weighted window. Second, if a photographer takes many pictures of one subject, this implies that the subject is of a certain importance to him. To detect whether two images are showing the same subject, the SIFT algorithm is used [6]. If there is a reasonable number of matching SIFT keypoint descriptors between two images, they are regarded as showing the same subject. si denotes the number of images showing the same subject as image i. Third, if a photographer takes two or more near identical images, it implies that this particular shot is of special importance to him. Such duplicate images are detected using the MPEG-7 color layout descriptor [4], where an empirically determined threshold defines whether two images are regarded as duplicates. Similar to above, di denotes the number of duplicate images of image i. A weighted combination results in a final importance score siimportance = u1 ∗ fi /fmax + u2 ∗ di /dmax + u3 ∗ si /smax , where fmax , smax and dmax are the maximum values for the entire photo collection, and ui are weighting factors. 2.2

Image Appeal

Machine learning techniques are used to calculate an appeal score for each image. Therefore, 15 low- and mid-level image features were designed, covering the aspects technical quality, composition, simplicity and colorfulness (see Tab. 1). Fourteen test persons of different age groups were asked to provide personal photographs of particularly high and low image appeal. Thus, a training database consisting of 840 images was built. Using this database, an SVM classifier was trained. The resulting SVM model is used to estimate the image appeal. The probability estimation for the “high appeal” class membership of image i is used as the final appeal score siappeal . Table 1. Overview of selected appeal classification features

2.3

technical quality

simplicity

highlight/shadow clipping sharpness

number of salient regions number of distinct hues

colorfulness

composition

color distribution standard deviation hue-channel

rule of thirds centrality

Presence of People

In order to rate the presence of people in each photograph i, we combine face and skin detection. The number of faces ni in each image as well as their relative size ai are obtained using a face detector based on Haar features and AdaBoost training. The relative amount of skin si in each image is calculated using a skin detector trained on skin colors. All three aspects are combined to a final people score sipeople = v1 ∗ log(ni ) + v2 ∗ ai + v3 ∗ si where vi are weighting factors.

194

D. Kormann, P. Dunker, and R. Paduschek

Fig. 1. Example of an event with calculated appeal score (AS) and importance score (IS) and the resulting 3 selected images (green) after duplicate removal

2.4

Ranking Framework

The three calculated scores are combined to obtain a final combined score for ranking the images: sicombined = w1 ∗ siimportance + w2 ∗ siappeal + w3 ∗ sipeople . The weighting wi of each score can be determined by the user or specified by certain presets, according to the application scenario. Due to the calculated importance score, the top-scoring images usually contain many pictures showing the same subject. We again use SIFT keypoints for detecting similar subjects. With this information, only the highest rated images which are showing different subjects can be selected (as depicted in Fig. 1). The representativity of the selected images can be increased by performing event detection prior to the ranking process, and selecting the highest-rated images from each event separately. To detect events, we combine and modify two existing approaches [3] [5]. The time gaps between images are first clustered using the k-means algorithm (k = 2). These initial events are then further partitioned by looking for outliers.

3

Evaluation

As the user’s satisfaction with the rating of the images is the main measure for the quality of our method, a user study was conducted. Fourteen participants provided personal photo collections. Images were rated and selected using different presets. Each test person rated the selected images of his/her own photo collection by filling in a questionnaire, which was designed according to guidelines of qualitative research, e.g. using 5-point Likert scales and cross-check questions. The boxplot in Fig. 2 exemplarily shows one result of the user study. It represents the overall satisfaction of the test persons with the selected images. As can be clearly seen, the images selected by our method obtained much better overall ratings than randomly selected images. The test results in general are promising. The calculated image appeal was reflected well by the user ratings. Throughout, selections generated by our method obtained considerably better ratings compared to random selections.

Automatic Rating and Selection of Digital Photographs

195

Fig. 2. Overall user-satisfaction with the selected images (1-worst, 5-best)

4

Conclusions

We briefly presented our method for rating and selecting personal digital photographs, which rates images based on various criteria, decreases redundancy and increases representativity amongst the selected images. A conducted user study delivered promising results, and verifies our concept of the automatic selection of images.

References 1. Boll, S., Sandhaus, P., Scherp, A., Thieme, S.: MetaXa - Context-and ContentDriven Metadata Enhancement for Personal Photo Books. In: Cham, T.-J., Cai, J., Dorai, C., Rajan, D., Chua, T.-S., Chia, L.-T. (eds.) MMM 2007. LNCS, vol. 4351, pp. 332–343. Springer, Heidelberg (2007) 2. Datta, R., Joshi, D., Li, J., Wang, J.: Studying Aesthetics in Photographic Images Using a Computational Approach. In: Nguyen, N.T., Kowalczyk, R., Chen, S.-M. (eds.) IWSM 2000. LNCS (LNAI), vol. 5796, pp. 152–162. Springer, Heidelberg (2009) 3. Graham, A., Garcia-Molina, H., Paepcke, A., Winograd, T.: Time as essence for photo browsing through personal digital libraries. In: Proceedings of the 2nd ACM/IEEE-CS joint conference on Digital libraries, pp. 326–335. ACM Press, New York (2002) 4. Kasutani, E., Yamada, A.: The MPEG-7 color layout descriptor: a compact image feature description for high-speed image/video segment retrieval. In: Proceedings of the 2001 International Conference on Image Processing, vol. 1 (2001) 5. Loui, A., Savakis, A., Co, E., Rochester, N.: Automated event clustering and quality screening of consumer pictures for digital albuming. IEEE Transactions on Multimedia 5(3), 390–402 (2003) 6. Lowe, D.: Distinctive Image Features from Scale-Invariant Keypoints. International Journal of Computer Vision 60(2), 91–110 (2004) 7. Savakis, A., Etz, S., Loui, A., et al.: Evaluation of image appeal in consumer photography. In: Proceedings-SPIE the International Society for Optical Engineering, pp. 111–121 (2000)

Author Index

Abbasi, Rabeeh

65

Leelanupab, Teerapong 3 Lehmann, Jens 182 Lindstaedt, Stefanie 40 Llorente, Ainhoa 188 Lohmann, Steffen 16, 182 L´ opez, Fernando 114

Baeza-Yates, Ricardo 1 Beales, Richard 177 Benn, Neil 77 B¨ urger, Tobias 52, 101 Carvalho, Rodrigo F. 149 Cattaneo, Fabio 77 Chakravarthy, Ajay 177 Ciravegna, Fabio 149 Conconi, Alex 77 Cullen, Charlie 169 Cusano, Claudio 28 Delgado, Jaime 89 Dietze, Stefan 77 Domingue, John 77 Dumitrescu, Alexandra Dunker, Peter 192 Ertl, Thomas

Mart´ınez, Jos´e M. 114 Matskanis, Nikos 177 McCarthy, Evin 169 McAuley, John 169 Motta, Enrico 188 Nagamatsu, Takashi Neuschmied, Helmut Ohbuchi, Ryutarou

173

137

P., Punitha 52, 126 Paduschek, Ronny 192 Pammer, Viktoria 40

16 Rodr´ıguez-Doncel, V´ıctor R¨ uger, Stefan 2, 188

Feng, Yue 3 Fukuhara, Yuki

161

Garc´ıa, Narciso 114 Grzegorzek, Marcin 65 Halb, Wolfgang 52, 165 Heim, Philipp 16, 182 Hellmann, Sebastian 182

Santini, Simone 28, 173 Schettini, Raimondo 28 Simperl, Elena 101 Staab, Steffen 65 Stathopoulos, Vassilios 3 Stegemann, Timo 182 Tetzlaff, Lena

Ishii, Yutaka

161 165

16

161

Jose, Joemon M.

3, 126

Kaieda, Yohei 161 Kamahara, Junzo 161 Kawamura, Shun 137 Kormann, Daniel 192 Kump, Barbara 40

Vaughan, Brian 169 Villa, Robert 52 Weiss, Wolfgang Yang, Xiaoyu Ziegler, J¨ urgen

52 177 16

89