Euro-Par 2010 - Parallel Processing: 16th International Euro-Par Conference, Ischia, Italy, August 31 - September 3, 2010, Proceedings, Part I ... Computer Science and General Issues)

Lecture Notes in Computer Science Commenced Publication in 1973 Founding and Former Series Editors: Gerhard Goos, Juris ...

Author: Pasqua D'Ambra | Mario Guarracino | Domenico Talia

16 downloads 976 Views 10MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form

DOWNLOAD PDF

Lecture Notes in Computer Science Commenced Publication in 1973 Founding and Former Series Editors: Gerhard Goos, Juris Hartmanis, and Jan van Leeuwen

Editorial Board David Hutchison Lancaster University, UK Takeo Kanade Carnegie Mellon University, Pittsburgh, PA, USA Josef Kittler University of Surrey, Guildford, UK Jon M. Kleinberg Cornell University, Ithaca, NY, USA Alfred Kobsa University of California, Irvine, CA, USA Friedemann Mattern ETH Zurich, Switzerland John C. Mitchell Stanford University, CA, USA Moni Naor Weizmann Institute of Science, Rehovot, Israel Oscar Nierstrasz University of Bern, Switzerland C. Pandu Rangan Indian Institute of Technology, Madras, India Bernhard Steffen TU Dortmund University, Germany Madhu Sudan Microsoft Research, Cambridge, MA, USA Demetri Terzopoulos University of California, Los Angeles, CA, USA Doug Tygar University of California, Berkeley, CA, USA Gerhard Weikum Max Planck Institute for Informatics, Saarbruecken, Germany

6271

Pasqua D’Ambra Mario Guarracino Domenico Talia (Eds.)

Euro-Par 2010 Parallel Processing 16th International Euro-Par Conference Ischia, Italy, August 31 - September 3, 2010 Proceedings, Part I

13

Volume Editors Pasqua D’Ambra ICAR-CNR Via P. Castellino 111 80131 Napoli, Italy E-mail: [email protected] Mario Guarracino ICAR-CNR Via P. Castellino 111 80131 Napoli, Italy E-mail: [email protected] Domenico Talia ICAR-CNR Via P. Bucci 41c 87036 Rende, Italy E-mail: [email protected]

Library of Congress Control Number: 2010932506 CR Subject Classification (1998): F.1.2, C.3, C.2.4, D.1, D.4, I.6 LNCS Sublibrary: SL 1 – Theoretical Computer Science and General Issues ISSN ISBN-10 ISBN-13

0302-9743 3-642-15276-7 Springer Berlin Heidelberg New York 978-3-642-15276-4 Springer Berlin Heidelberg New York

This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, re-use of illustrations, recitation, broadcasting, reproduction on microfilms or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable to prosecution under the German Copyright Law. springer.com © Springer-Verlag Berlin Heidelberg 2010 Printed in Germany Typesetting: Camera-ready by author, data conversion by Scientific Publishing Services, Chennai, India Printed on acid-free paper 06/3180

Preface

Euro-Par is an annual series of international conferences dedicated to the promotion and advancement of all aspects of parallel computing. The major themes can be divided into four broad categories: theory, high-performance, cluster and grid, distributed and mobile computing. These categories comprise 14 topics that focus on particular issues. The objective of Euro-Par is to provide a forum within which to promote the development of parallel computing both as an industrial technique and an academic discipline, extending the frontier of both the state of the art and the state of practice. The main audience for and participants in Euro-Par are researchers in academic departments, government laboratories, and industrial organizations. Euro-Par 2010 was the 16th conference in the Euro-Par series, and was organized by the Institute for High-Performance Computing and Networking (ICAR) of the Italian National Research Council (CNR), in Ischia, Italy. Previous EuroPar conferences took place in Stockholm, Lyon, Passau, Southampton, Toulouse, Munich, Manchester, Padderborn, Klagenfurt, Pisa, Lisbon, Dresden, Rennes, Las Palmas, and Delft. Next year the conference will take place in Bordeaux, France. More information on the Euro-Par conference series and organization is available on the wesite http://www.europar.org. As mentioned before, the conference was organized in 14 topics. The paper review process for each topic was managed and supervised by a committee of at least four persons: a Global Chair, a Local Chair, and two members. Some specific topics with a high number of submissions were managed by a larger committee with more members. The final decisions on the acceptance or rejection of the submitted papers were made in a meeting of the Conference Co-chairs and Local Chairs of the topics. The call for papers attracted a total of 256 submissions, representing 41 countries (based on the corresponding authors’ countries). A total number of 938 review reports were collected, which makes an average of 3.66 review reports per paper. In total 90 papers were selected as regular papers to be presented at the conference and included in the conference proceedings, representing 23 countries from all continents, therefore the acceptance rate was of 35%. Three papers were selected as distinguished papers. These papers, which were presented in a separate session, are:

1. Friman Sanchez, Felipe C, Alex Ramirez and Mateo Valero, Long DNA Sequence Comparison on Multicore Architectures 2. Michel Raynal and Damien Imbs, The x-Wait-Freedom Progress Condition 3. Mark James, Paul Springer and Hans Zima, Adaptive Fault Tolerance for Many-Core-Based Space-Borne Computing

VI

Preface

Euro-Par 2010 was very happy to present three invited speakers of high international reputation, who discussed important developments in very interesting areas of parallel and distributed computing: 1. Jack Dongarra (University of Tennessee, Oak Ridge National Laboratory, University of Manchester): Impact of Architecture and Technology for Extreme Scale on Software and Algorithm Design 2. Vittoria Colizza (ISI Foundation): Computational Epidemiology: A New Paradigm in the Fight Against Infectious Diseases 3. Ignacio M. Llorente (Universidad Complutense de Madrid): Innovation in Cloud Computing Architectures During the conference Jack Dongarra received the Third Euro-Par Achievement Award for his leadership in the field of parallel and distributed computing. In this edition, 11 workshops were held in conjunction with the main track of the conference. These workshops were: 1. CoreGrid/ERCIM Workshop (CoreGrid2010) 2. The 5th Workshop on Virtualization in High-Performance Cluster and Grid Computing(VHPC 2010) 3. Workshop on Algorithms, Models and Tools for Parallel Computing on Heterogeneous Platforms (HeteroPar 2010) 4. The 4th Workshop on Highly Parallel Processing on a Chip (HPPC 2010) 5. The 7th International Workshop on the Economics and Business Grids, Clouds, Systems, and Services (GECON 2010) 6. The First Workshop on High-Performance Bioinformatics and Biomedicine (HiBB) 7. The Third Workshop on UnConventional High-Performance Computing (UCHPC 2010) 8. Workshop on HPC Applied to Finance (HPCF 2010) 9. The Third Workshop on Productivity and Performance Tools for HPC Application Development (PROPER2010) 10. Workshop on Cloud Computing: Projects and Initiatives (CCPI 2010) 11. XtreemOS Summit 2010. The 16th Euro-Par conference in Ischia was made possible due to the support of many individuals and organizations. Special thanks are due to the authors of all the submitted papers, the members of the topic committees, and all the reviewers in all topics, for their contributions to the success of the conference. We also thank the members of the Organizing Committee and people of the YES Meet organizing secretariat. We are grateful to the members of the Euro-Par Steering Committee for their support. We acknowledge the help we had from Henk Sips and Dick Epema of the organization of Euro-Par 2009. A number of institutional and industrial sponsors contributed towards the organization of the conference. Their names and logos appear on the Euro-Par 2010 website http://www.europar2010.org.

Preface

VII

It was our pleasure and honor to organize and host Euro-Par 2010 in Ischia. We hope all the participants enjoyed the technical program and the social events organized during the conference. August 2010

Domenico Talia Pasqua D’Ambra Mario Rosario Guarracino

Organization

Euro-Par Steering Committee Chair Chris Lengauer

University of Passau, Germany

Vice-Chair Luc Bougé

ENS Cachan, France

European Respresentatives José Cunha Marco Danelutto Christos Kaklamanis Paul Kelly Harald Kosch Thomas Ludwig Emilio Luque Toms Margalef Wolfgang Nagel Rizos Sakellariou Henk Sips

New University of Lisbon, Portugal University of Pisa, Italy Computer Technology Institute, Greece Imperial College, UK University of Passau, Germany University of Heidelberg, Germany University Autonoma of Barcelona, Spain University Autonoma of Barcelona, Spain Dresden University of Technology, Germany University of Manchester, UK Delft University of Technology, The Netherlands

Honorary Members Ron Perrott Karl Dieter Reinartz

Queen’s University Belfast, UK University of Erlangen-Nuremberg, Germany

Observers Domenico Talia Emmanuel Jeannot

University of Calabria and ICAR-CNR, Italy Laboratoire Bordelais de Recherche en Informatique (LaBRI) / INRIA, France

X

Organization

Euro-Par 2010 Organization Conference Co-chairs Pasqua D’Ambra Mario Guarracino Domenico Talia

ICAR-CNR, Italy ICAR-CNR, Italy ICAR-CNR and University of Calabria, Italy

Local Organizing Committee Laura Antonelli Eugenio Cesario Agostino Forestiero Francesco Gregoretti Ivana Marra Carlo Mastroianni

ICAR-CNR, ICAR-CNR, ICAR-CNR, ICAR-CNR, ICAR-CNR, ICAR-CNR,

Italy Italy Italy Italy Italy Italy

Euro-Par 2010 Program Committee Topic 1: Support Tools and Environments Global Chair Omer Rana

Cardiff University, UK

Local Chair Giandomenico Spezzano

ICAR-CNR, Italy

Members Michael Gerndt Daniel S. Katz

Technical University of Munich, Germany University of Chicago, USA

Topic 2: Performance Prediction and Evaluation Global Chair Stephen Jarvis

Warwick University, UK

Local Chair Massimo Coppola

ISTI-CNR, Italy

Members Junwei Cao Darren Kerbyson

Tsinghua University, China Los Alamos National Laboratory, USA

Organization

XI

Topic 3: Scheduling and Load-Balancing Global Chair Ramin Yahyapour

Technical University of Dortmund, Germany

Local Chair Raffaele Perego

ISTI-CNR, Italy

Members Frederic Desprez Leah Epstein Francesc Guim Bernat

INRIA Rhône-Alpes, France University of Haifa, Israel Intel, Spain

Topic 4: High-Performance Architectures and Compilers Global Chair Pedro Diniz

IST/UTL/INESC-ID, Portugal

Local Chair Marco Danelutto

University of Pisa, Italy

Members Denis Barthou Marc Gonzalez Tallada Michael Huebner Karlsruhe

University of Versailles, France Polytechnic University of Catalonia, Spain Institute of Technology, Germany

Topic 5: Parallel and Distributed Data Management Global Chair Rizos Sakellariou

University of Manchester, UK

Local Chair Salvatore Orlando

University of Venice, Italy

Members Josep-L. Larriba-Pey Srinivasan Parthasarathy Demetrios Zeinalipour

Polytechnic University of Catalonia, Spain Ohio State University, USA University of Cyprus, Cyprus

Topic 6: Grid, Cluster and Cloud Computing Global Chair Kate Keahey

Argonne National Laboratory, USA

XII

Organization

Local Chair Domenico Laforenza

IIT-CNR, Italy

Members Alexander Reinefeld Pierluigi Ritrovato Doug Thain Nancy Wilkins-Diehr

Zuse Institute Berlin, Germany University of Salerno and CRMPA, Italy University of Notre Dame, USA San Diego Supercomputer Center, USA

Topic 7: Peer-to-Peer Computing Global Chair Adriana Iamnitchi

University of South Florida, USA

Local Chair Paolo Trunfio

University of Calabria, Italy

Members Jonathan Ledlie Florian Schintke

Nokia Research, USA Zuse Institute of Berlin, Germany

Topic 8: Distributed Systems and Algorithms Global Chair Pierre Sens

University of Paris 6, France

Local Chair Giovanni Schmid

ICAR-CNR, Italy

Members Pascal Felber Ricardo Jimenez-Peris

University of Neuchatel, Switzerland Polytechnic University of Madrid, Spain

Topic 9: Parallel and Distributed Programming Global Chair Thilo Kielmann

Vrije Universiteit, The Netherlands

Local Chair Andrea Clematis

IMATI-CNR, Italy

Members Sergey Gorlatch Alexey Lastovetsky

University of Munster, Germany University College Dublin, Ireland

Organization

XIII

Topic 10: Parallel Numerical Algorithms Global Chair Patrick Amestoy Local Chair Daniela di Serafino

Members Rob Bisseling Enrique S. Quintana Ort´ı Marian Vajtersic

University of Toulouse, France

Second University of Naples and ICAR-CNR, Italy

Utrecht University, The Netherlands University Jaime I, Spain University of Salzburg, Austria

Topic 11: Multicore and Manycore Programming Global Chair Fabrizio Petrini

IBM, USA

Local Chair Beniamino Di Martino Members Siegfried Benkner Kirk Cameron Dieter Kranzlm¨ uller Jakub Kurzak Jesper Larsson Träff Davide Pasetto

Second University of Naples, Italy

University of Vienna, Austria Virginia Tech, USA Ludwig Maximilians University of Munich, Germany University of Tennessee, USA University of Vienna, Austria IBM, Ireland

Topic 12: Theory and Algorithms for Parallel Computation Global Chair Thomas Rauber

University of Bayreuth , Germany

Local Chair Vittorio Scarano

University of Salerno, Italy

Members Christoph Kessler Yves Robert

Linkoping University, Sweden Normal Superior School of Lyon, France

XIV

Organization

Topic 13: High-Performance Networks Global Chair José Flich

Technical University of Valencia, Spain

Local Chair Alfonso Urso

ICAR-CNR, Italy

Members Ulrich Bruening Giuseppe Di Fatta

Heidelberg University, Germany University of Reading, UK

Topic 14: Mobile and Ubiquitous Computing Global Chair Gregor Schiele

University of Mannheim, Germany

Local Chair Giuseppe De Pietro

ICAR-CNR, Italy

Members Jalal Al-Muhtadi Zhiwen Yu Northwestern

King Saud University, Saudi Arabia Polytechnical University, China

Euro-Par 2010 Referees Virat Agarwal Josep Aguilar Saborit Jalal Al Muhtadi Samer Al-Kiswany Jose Aliaga Pedro Alonso Patrick Amestoy Panayiotis Andreou Artur Andrzejak Ashiq Anjum Benjamin Arai Luciana Arantes Peter Arbenz Mikael Asplund Rocco Aversa Rosa M. Badia José Manuel Bad´ıa-Contelles Mark Baker

Ranieri Baraglia Kevin Barker Denis Barthou Tom Beach Shajulin Benedict Siegfried Benkner Anne Benoit John Bent Anca Berariu Massimo Bernaschi Carlo Bertolli Paolo Bientinesi Angelos Bilas Rob Bisseling Jeremy Blackburn Brian Blake Erik Boman Mathieu Bouillaguet

Organization

Thomas Brady John Bresnahan Ron Brightwell Andrey Brito Maciej Brodowicz Shawn Brown Ulrich Bruening Ali Butt Alfredo Buttari Surendra Byna Joao Cachopo Kirk Cameron Agustin Caminero Ramon Canal Pasquale Cantiello Junwei Cao Gabriele Capannini Emanuele Carlini David Carrera Simon Caton Eugenio Cesario Gregory Chockler Martin Chorley Peter Chronz Giuseppe Ciaccio Mario Ciampi Andrea Clematis Carmela Comito Denis Conan Guojing Cong Massimo Coppola Angelo Corana Julita Corbalan Antonio Coronato Stefania Corsaro Rubén Cuevas Rum´ın Yong Cui Alfredo Cuzzocrea Daniele D’Agostino Pasqua D’Ambra Maurizio D’Arienzo John Daly Marco Danelutto Patrizio Dazzi Giuseppe De Pietro

Valentina De Simone Frederic Desprez Giuseppe Di Fatta Beniamino Di Martino Claudia Di Napoli Daniela di Serafino Pedro Diniz Cristian Dittamo David Dominguez-Sal Rion Dooley Jan Dunnweber Alejandro Duran Pierre-Franois Dutot Jorge Ejarque Artigas Thomas Epperly Leah Epstein Juan Carlos Fabero Pascal Felber Florian Feldhaus Xizhou Feng John Feo Salvatore Filippone Joshua Finnis Jose Flich Gianluigi Folino Agostino Forestiero Giancarlo Fortino Michel Fournié Antonella Galizia Luigi Gallo Efstratios Gallopoulos Alfredo Garro Rong Ge Joanna Geibig Krassimir Georgiev Joseph Gergaud Abdou Germouche Michael Gerndt Vittoria Gianuzzi Domingo Gimenez Maurizio Giordano Harald Gjermundrod Frank Glinka Sergio G´ omez-Villamor Jose Gonzalez

XV

XVI

Organization

Marc Gonzalez Maria Gradinariu Vincent Gramoli Fabola Greve Mario Guarracino Michele Guidolin Carla Guillen Carias Francesc Guim Bernat Thom Haddow Georg Hager Houssam Haitof Paul Hargrove Tim Harris Enric Herrero Josep Ramon Herrero Ignacio Hidalgo Perez Daniel Higuero Sing Wang Ho Yannick Hoarau Torsten Hoefler Mikael Hogqvist Gerard Holzmann Haowei Huang Michael Hubner Mauro Iacono Adriana Iamnitchi Francisco Igual-Pe˜ na Stephen Jarvis Prasad Jayanti Emmanuel Jeannot Shantenu Jha Daniel Jimnez-Gonz´ alez Ricardo Jimenez-Peris Maik Jorra Gopi Kandaswamy Karen Karavanic Daniel Katz Kate Keahey Philipp Kegel Darren Kerbyson Christoph Kessler Thilo Kielmann Hyunjoo Kim Zach King Bjrn Kolbeck

Derrick Kondo Andreas Konstantinidis Nicolas Kourtellis Dieter Kranzlmuller Peter Kropf Nico Kruber Herbert Kuchen Jakub Kurzak David LaBissoniere Giuliano Laccetti Domenico Laforenza Juan Lanchares Marco Lapegna Josep Larriba-Pey Alexey Lastovetsky Rob Latham Jonathan Ledlie Arnaud Legrand Sergey Legtchenko Francesco Lelli Jeff Linderoth Yan Liu David Lowenthal Claudio Lucchese Xiaosong Ma Lucia Maddalena Mahin Mahmoodi Mesaac Makpangou Barnaby Malet Loris Marchal Olivier Marin Stefano Marrone Suresh Marru Paul Marshall Alberto Martn-Huertas Norbert Martnez-Baz´ an Carlo Mastroianni Rafael Mayo Michele Mazzucco Dominik Meilaender Alessio Merlo Marcel Meyer Gregory Michaelson Mauro Migliardi Matteo Migliavacca

Organization

Einat Minkov Sbastien Monnet Raffaele Montella Matteo Mordacchini Christopher Moretti Paolo Mori Francesco Moscato Alexander Moskovsky Sandrine Mouysset Kiran-Kumar Muniswamy-Reddy Victor Muntés-Mulero Alin Murarasu Farrukh Nadeem Jamin Naghmouchi Franco Maria Nardini John-Paul Navarro Libero Nigro Dimitrios Nikolopoulos Praveen Nuthulapati Yury Oleynik Salvatore Orlando Renzo Orsini George Pallis Alexander Papaspyrou Scott Parker Srinivasan Parthasarathy Davide Pasetto Jean-Louis Pazat Francois Pellegrini Olivier Peres Francesca Perla Kathrin Peter Ventsislav Petkov Fabrizio Petrini Marlon Pierce Jean-Marc Pierson Giuseppe Pirr´ o Stefan Plantikow Antonio Plaza Sabri Pllana Alexander Ploss Joseph Polifroni Andrea Pugliese Judy Qiu Alfonso Quarati

Martin Quinson Ioan Raicu Massimiliano Rak Thomas Rauber Alfredo Remon Laura Ricci Pierluigi Ritrovato Etienne Riviere Yves Robert Ivan Rodero Arun Rodrigues Arnold Rosenberg Francois-Henry Rouet Alain Roy Paul Ruth Vladimir Rychkov Rizos Sakellariou Jose Sánchez Martin Sandrieser Vittorio Scarano Daniele Scarpazza Patrick Schafer Valerio Schiavoni Gregor Schiele Michael Schiffers Florian Schintke Giovanni Schmid Erik Schnetter Assaf Schuster Frank Seinstra Pierre Sens Lei Shu Claudio Silvestri Ral Sirvent Tor Skeie David Skinner Warren Smith Julien Sopena Paul Soule Giandomenico Spezzano Kyriakos Stavrou Thomas Steinke Jan Stender Pietro Storniolo Hari Subramoni

XVII

XVIII

Organization

Frederic Suter Pierre Sutra Martin Swany Spencer Swift Alex Tabbal Domenico Talia Ian Taylor Andrei Tchernykh Douglas Thain Gal Thomas Juan Tirado Matthew Tolentino Rafael Tolosana-Calasanz Nicola Tonellotto Jesper Traff Corentin Travers Paolo Trunfio Mauricio Tsugawa Alfonso Urso Marian Vajtersic Jose Valerio Rob van Nieuwpoort Robbert van Renesse Ana Lucia Varbanescu

Salvatore Venticinque Rossano Venturini Antonio Vidal Vicente Vidal Frederic Vivien Edward Walker Hanbo Wang Shaowen Wang Charles Weems Josef Weidendorfer Philipp Wieder Matthew Woitaszek Wenjun Wu Roman Wyrzykowski Dongyan Xu Ramin Yahyapour Kenneth Yoshimoto Choonhan Youn Zhiwen Yu Luca Zanni Demetrios Zeinalipour-Yazti Wensheng Zhang Wolfgang Ziegler Hans Zima

Table of Contents – Part I

Topic 1: Support Tools and Environments Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Omer Rana, Giandomenico Spezzano, Michael Gerndt, and Daniel S. Katz

1

Starsscheck: A Tool to Find Errors in Task-Based Parallel Programs . . . . Paul M. Carpenter, Alex Ramirez, and Eduard Ayguade

2

Automated Tuning in Parallel Sorting on Multi-Core Architectures . . . . . Haibo Lin, Chao Li, Qian Wang, Yi Zhao, Ninghe Pan, Xiaotong Zhuang, and Ling Shao

14

Estimating and Exploiting Potential Parallelism by Source-Level Dependence Profiling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jonathan Mak, Karl-Filip Faxén, Sverker Janson, and Alan Mycroft

26

Source-to-Source Optimization of CUDA C for GPU Accelerated Cardiac Cell Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fred V. Lionetti, Andrew D. McCulloch, and Scott B. Baden

38

Efficient Graph Partitioning Algorithms for Collaborative Grid Workflow Developer Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Gergely Sipos and Péter Kacsuk

50

Profile-Driven Selective Program Loading . . . . . . . . . . . . . . . . . . . . . . . . . . . Tugrul Ince and Jeffrey K. Hollingsworth Characterizing the Impact of Using Spare-Cores on Application Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . José Carlos Sancho, Darren J. Kerbyson, and Michael Lang

62

74

Topic 2: Performance Prediction and Evaluation Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Stephen Jarvis, Massimo Coppola, Junwei Cao, and Darren Kerbyson A Model for Space-Correlated Failures in Large-Scale Distributed Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Matthieu Gallet, Nezih Yigitbasi, Bahman Javadi, Derrick Kondo, Alexandru Iosup, and Dick Epema

86

88

XX


Architecture Exploration for Efficient Data Transfer and Storage in Data-Parallel Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Rosilde Corvino, Abdoulaye Gamatié, and Pierre Boulet

101

jitSim: A Simulator for Predicting Scalability of Parallel Applications in Presence of OS Jitter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Pradipta De and Vijay Mann

117

pCFS vs. PVFS: Comparing a Highly-Available Symmetrical Parallel Cluster File System with an Asymmetrical Parallel File System . . . . . . . . Paulo A. Lopes and Pedro D. Medeiros

131

Comparing Scalability Prediction Strategies on an SMP of CMPs . . . . . . Karan Singh, Matthew Curtis-Maury, Sally A. McKee, Filip Blagojević, Dimitrios S. Nikolopoulos, Bronis R. de Supinski, and Martin Schulz

143

Topic 3: Scheduling and Load-Balancing Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ramin Yahyapour, Raffaele Perego, Frédéric Desprez, Leah Epstein, and Francesc Guim Bernat

156

A Fast 5/2-Approximation Algorithm for Hierarchical Scheduling . . . . . . Marin Bougeret, Pierre-Fran¸cois Dutot, Klaus Jansen, Christina Otte, and Denis Trystram

157

Non-clairvoyant Scheduling of Multiple Bag-of-Tasks Applications . . . . . . Henri Casanova, Matthieu Gallet, and Frédéric Vivien

168

Extremal Optimization Approach Applied to Initial Mapping of Distributed Java Programs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ivanoe De Falco, Eryk Laskowski, Richard Olejnik, Umberto Scafuri, Ernesto Tarantino, and Marek Tudruj A Delay-Based Dynamic Load Balancing Method and Its Stability Analysis and Simulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Qingyang Meng, Jianzhong Qiao, Shukuan Lin, Enze Wang, and Peng Han

180

192

Code Scheduling for Optimizing Parallelism and Data Locality . . . . . . . . . Taylan Yemliha, Mahmut Kandemir, Ozcan Ozturk, Emre Kultursay, and Sai Prashanth Muralidhara

204

Hierarchical Work-Stealing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jean-Noël Quintin and Frédéric Wagner

217

Optimum Diffusion for Load Balancing in Mesh Networks . . . . . . . . . . . . . George S. Markomanolis and Nikolaos M. Missirlis

230


A Dynamic, Distributed, Hierarchical Load Balancing for HLA-Based Simulations on Large-Scale Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . Robson Eduardo De Grande and Azzedine Boukerche

XXI

242

Topic 4: High Performance Architectures and Compilers Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Pedro C. Diniz, Marco Danelutto, Denis Barthou, Marc Gonzales, and Michael H¨ ubner

254

Power-Efficient Spilling Techniques for Chip Multiprocessors . . . . . . . . . . Enric Herrero, José Gonz´ alez, and Ramon Canal

256

Scalable Object-Aware Hardware Transactional Memory . . . . . . . . . . . . . . Behram Khan, Matthew Horsnell, Mikel Lujan, and Ian Watson

268

Efficient Address Mapping of Shared Cache for On-Chip Many-Core Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fenglong Song, Dongrui Fan, Zhiyong Liu, Junchao Zhang, Lei Yu, and Weizhi Xu Thread Owned Block Cache: Managing Latency in Many-Core Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fenglong Song, Zhiyong Liu, Dongrui Fan, Hao Zhang, Lei Yu, and Shibin Tang Extending the Cell SPE with Energy Efficient Branch Prediction . . . . . . . Martijn Briejer, Cor Meenderinck, and Ben Juurlink

280

292

304

Topic 5: Parallel and Distributed Data Management Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Rizos Sakellariou, Salvatore Orlando, Josep Lluis Larriba-Pey, Srinivasan Parthasarathy, and Demetrios Zeinalipour-Yazti

316

Federated Enactment of Workflow Patterns . . . . . . . . . . . . . . . . . . . . . . . . . Gagarine Yaikhom, Chee Sun Liew, Liangxiu Han, Jano van Hemert, Malcolm Atkinson, and Amy Krause

317

A Distributed Approach to Detect Outliers in Very Large Data Sets . . . . Fabrizio Angiulli, Stefano Basta, Stefano Lodi, and Claudio Sartori

329

Topic 6: Grid, Cluster and Cloud Computing Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . K. Keahey, D. Laforenza, A. Reinefeld, P. Ritrovato, D. Thain, and N. Wilkins-Diehr

341

XXII


Deployment of a Hierarchical Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . Eddy Caron, Benjamin Depardon, and Frédéric Desprez

343

Toward Real-Time, Many-Task Applications on Large Distributed Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Sangho Yi, Derrick Kondo, and David P. Anderson

355

Scheduling Scientific Workflows to Meet Soft Deadlines in the Absence of Failure Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Kassian Plankensteiner, Radu Prodan, and Thomas Fahringer

367

A GPGPU Transparent Virtualization Component for High Performance Computing Clouds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Giulio Giunta, Raffaele Montella, Giuseppe Agrillo, and Giuseppe Coviello What Is the Price of Simplicity? A Cross-Platform Evaluation of the SAGA API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Mathijs den Burger, Ceriel Jacobs, Thilo Kielmann, Andre Merzky, Ole Weidner, and Hartmut Kaiser User-Centric, Heuristic Optimization of Service Composition in Clouds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Kevin Kofler, Irfan ul Haq, and Erich Schikuta A Distributed Market Framework for Large-Scale Resource Sharing . . . . Marian Mihailescu and Yong Meng Teo Using Network Information to Perform Meta-Scheduling in Advance in Grids . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Luis Tom´ as, Agust´ın Caminero, Blanca Caminero, and Carmen Carri´ on

379

392

405

418

431

Topic 7: Peer to Peer Computing Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Adriana Iamnitchi, Paolo Trunfio, Jonathan Ledlie, and Florian Schintke Overlay Management for Fully Distributed User-Based Collaborative Filtering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . R´ obert Orm´ andi, Istv´ an Heged˝ us, and M´ ark Jelasity Dynamic Publish/Subscribe to Meet Subscriber-Defined Delay and Bandwidth Constraints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Muhammad Adnan Tariq, Gerald G. Koch, Boris Koldehofe, Imran Khan, and Kurt Rothermel

444

446

458


Combining Hilbert SFC and Bruijn Graphs for Searching Computing Markets in a P2P System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Damia Castell` a, Hector Blanco, Francesc Giné, and Francesc Solsona Sampling Bias in BitTorrent Measurements . . . . . . . . . . . . . . . . . . . . . . . . . . Boxun Zhang, Alexandru Iosup, Johan Pouwelse, Dick Epema, and Henk Sips A Formal Credit-Based Incentive Model for Sharing Computer Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Josep Rius, Ignasi Barri, Fernando Cores, and Francesc Solsona

XXIII

471 484

497

Topic 8: Distributed Systems and Algorithms Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Pascal Felber, Ricardo Jimenez-Peris, Giovanni Schmid, and Pierre Sens

510

Improving Message Logging Protocols Scalability through Distributed Event Logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Thomas Ropars and Christine Morin

511

Value-Based Sequential Consistency for Set Objects in Dynamic Distributed Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Roberto Baldoni, Silvia Bonomi, and Michel Raynal

523

Robust Self-stabilizing Construction of Bounded Size Weight-Based Clusters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Colette Johnen and Fouzi Mekhaldi

535

Adaptive Conflict Unit Size for Distributed Optimistic Synchronization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Kim-Thomas Rehmann, Marc-Florian M¨ uller, and Michael Sch¨ ottner Frame Allocation Algorithms for Multi-threaded Network Cameras . . . . . José Miguel Piquer and Javier Bustos-Jiménez Scalable Distributed Simulation of Large Dense Crowds Using the Real-Time Framework (RTF) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ole Scharf, Sergei Gorlatch, Felix Blanke, Christoph Hemker, Sebastian Westerheide, Tobias Priebs, Christoph Bartenhagen, Alexander Ploss, Frank Glinka, and Dominik Meilaender

547

560

572

The x-Wait-Freedom Progress Condition . . . . . . . . . . . . . . . . . . . . . . . . . . . . Damien Imbs and Michel Raynal

584

Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

597

Table of Contents – Part II

Topic 9: Parallel and Distributed Programming Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Thilo Kielmann, Andrea Clematis, Sergei Gorlatch, and Alexey Lastovetsky

1

Transactional Mutex Locks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Luke Dalessandro, Dave Dice, Michael Scott, Nir Shavit, and Michael Spear

2

Exceptions for Algorithmic Skeletons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Mario Leyton, Ludovic Henrio, and José M. Piquer

14

Generators-of-Generators Library with Optimization Capabilities in Fortress . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Kento Emoto, Zhenjiang Hu, Kazuhiko Kakehi, Kiminori Matsuzaki, and Masato Takeichi User Transparent Task Parallel Multimedia Content Analysis . . . . . . . . . . Timo van Kessel, Niels Drost, and Frank J. Seinstra Parallel Simulation for Parameter Estimation of Optical Tissue Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Mihai Duta, Jeyarajan Thiyagalingam, Anne Trefethen, Ayush Goyal, Vicente Grau, and Nic Smith

26

38

51

Topic 10: Parallel Numerical Algorithms Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Patrick Amestoy, Daniela di Serafino, Rob Bisseling, Enrique S. Quintana-Ort´ı, and Marian Vajterˇsicx Scalability and Locality of Extrapolation Methods for Distributed-Memory Architectures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Matthias Korch, Thomas Rauber, and Carsten Scholtes CFD Parallel Simulation Using Getfem++ and Mumps . . . . . . . . . . . . . . . Michel Fournié, Nicolas Renon, Yves Renard, and Daniel Ruiz Aggregation AMG for Distributed Systems Suffering from Large Message Numbers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Maximilian Emans

63

65

77

89

XXVI


A Parallel Implementation of the Jacobi-Davidson Eigensolver and Its Application in a Plasma Turbulence Code . . . . . . . . . . . . . . . . . . . . . . . . . . . Eloy Romero and Jose E. Roman Scheduling Parallel Eigenvalue Computations in a Quantum Chemistry Code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Martin Roderus, Anca Berariu, Hans-Joachim Bungartz, Sven Kr¨ uger, Alexei Matveev, and Notker R¨ osch Scalable Parallelization Strategies to Accelerate NuFFT Data Translation on Multicores . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yuanrui Zhang, Jun Liu, Emre Kultursay, Mahmut Kandemir, Nikos Pitsianis, and Xiaobai Sun

101

113

125

Topic 11: Multicore and Manycore Programming Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Beniamino Di Martino, Fabrizio Petrini, Siegfried Benkner, Kirk Cameron, Dieter Kranzlm¨ uller, Jakub Kurzak, Davide Pasetto, and Jesper Larsson Tr¨ aff

137

JavaSymphony: A Programming and Execution Environment for Parallel and Distributed Many-Core Architectures . . . . . . . . . . . . . . . . . . . . Muhammad Aleem, Radu Prodan, and Thomas Fahringer

139

Scalable Producer-Consumer Pools Based on Elimination-Diffraction Trees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yehuda Afek, Guy Korland, Maria Natanzon, and Nir Shavit

151

Productivity and Performance: Improving Consumability of Hardware Transactional Memory through a Real-World Case Study . . . . . . . . . . . . . Huayong Wang, Yi Ge, Yanqi Wang, and Yao Zou

163

Exploiting Fine-Grained Parallelism on Cell Processors . . . . . . . . . . . . . . . Ralf Hoffmann, Andreas Prell, and Thomas Rauber

175

Optimized on-Chip-Pipelined Mergesort on the Cell/B.E. . . . . . . . . . . . . . Rikard Hultén, Christoph W. Kessler, and J¨ org Keller

187

Near-Optimal Placement of MPI Processes on Hierarchical NUMA Architectures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Emmanuel Jeannot and Guillaume Mercier Parallel Enumeration of Shortest Lattice Vectors . . . . . . . . . . . . . . . . . . . . . ¨ ur Dagdelen and Michael Schneider Ozg¨ A Parallel GPU Algorithm for Mutual Information Based 3D Nonrigid Image Registration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Vaibhav Saxena, Jonathan Rohrer, and Leiguang Gong

199 211

223


Multi-GPU and Multi-CPU Parallelization for Interactive Physics Simulations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Everton Hermann, Bruno Raffin, Fran¸cois Faure, Thierry Gautier, and Jérémie Allard Long DNA Sequence Comparison on Multicore Architectures . . . . . . . . . . Friman S´ anchez, Felipe Cabarcas, Alex Ramirez, and Mateo Valero Adaptive Fault Tolerance for Many-Core Based Space-Borne Computing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Mark James, Paul Springer, and Hans Zima Maestro: Data Orchestration and Tuning for OpenCL Devices . . . . . . . . . Kyle Spafford, Jeremy Meredith, and Jeffrey Vetter Multithreaded Geant4: Semi-automatic Transformation into Scalable Thread-Parallel Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Xin Dong, Gene Cooperman, and John Apostolakis Parallel Exact Time Series Motif Discovery . . . . . . . . . . . . . . . . . . . . . . . . . . Ankur Narang and Souvik Bhattacherjee Optimized Dense Matrix Multiplication on a Many-Core Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Elkin Garcia, Ioannis E. Venetis, Rishi Khan, and Guang R. Gao A Language-Based Tuning Mechanism for Task and Pipeline Parallelism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Frank Otto, Christoph A. Schaefer, Matthias Dempe, and Walter F. Tichy A Study of a Software Cache Implementation of the OpenMP Memory Model for Multicore and Manycore Architectures . . . . . . . . . . . . . . . . . . . . . Chen Chen, Joseph B. Manzano, Ge Gan, Guang R. Gao, and Vivek Sarkar Programming CUDA-Based GPUs to Simulate Two-Layer Shallow Water Flows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Marc de la Asunci´ on, José M. Mantas, and Manuel J. Castro

XXVII

235

247

260

275

287

304

316

328

341

353

Topic 12: Theory and Algorithms for Parallel Computation Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Christoph Kessler, Thomas Rauber, Yves Robert, and Vittorio Scarano

365

XXVIII


Analysis of Multi-Organization Scheduling Algorithms . . . . . . . . . . . . . . . . Johanne Cohen, Daniel Cordeiro, Denis Trystram, and Frédéric Wagner

367

Area-Maximizing Schedules for Series-Parallel DAGs . . . . . . . . . . . . . . . . . Gennaro Cordasco and Arnold L. Rosenberg

380

Parallel Selection by Regular Sampling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Alexander Tiskin

393

Ants in Parking Lots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Arnold L. Rosenberg

400

Topic 13: High Performance Networks Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . José Flich, Alfonso Urso, Ulrich Bruening, and Giuseppe Di Fatta An Efficient Strategy for Reducing Head-of-Line Blocking in Fat-Trees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jesus Escudero-Sahuquillo, Pedro Javier Garcia, Francisco J. Quiles, and Jose Duato

412

413

A First Approach to King Topologies for On-Chip Networks . . . . . . . . . . . Esteban Stafford, Jose L. Bosque, Carmen Mart´ınez, Fernando Vallejo, Ramon Beivide, and Cristobal Camarero

428

Optimizing Matrix Transpose on Torus Interconnects . . . . . . . . . . . . . . . . . Venkatesan T. Chakaravarthy, Nikhil Jain, and Yogish Sabharwal

440

Topic 14: Mobile and Ubiquitous Computing Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Gregor Schiele, Giuseppe De Pietro, Jalal Al-Muhtadi, and Zhiwen Yu

452

cTrust: Trust Aggregation in Cyclic Mobile Ad Hoc Networks . . . . . . . . . Huanyu Zhao, Xin Yang, and Xiaolin Li

454

Maximizing Growth Codes Utility in Large-Scale Wireless Sensor Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yao Zhao, Xin Wang, Jin Zhao, and Xiangyang Xue @Flood: Auto-Tunable Flooding for Wireless Ad Hoc Networks . . . . . . . . José Mocito, Lu´ıs Rodrigues, and Hugo Miranda

466

478


On Deploying Tree Structured Agent Applications in Networked Embedded Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Nikos Tziritas, Thanasis Loukopoulos, Spyros Lalis, and Petros Lampsas

XXIX

490

Meaningful Metrics for Evaluating Eventual Consistency . . . . . . . . . . . . . . Jo˜ ao Barreto and Paulo Ferreira

503

Caching Dynamic Information in Vehicular Ad Hoc Networks . . . . . . . . . . Nicholas Loulloudes, George Pallis, and Marios D. Dikaiakos

516

Collaborative Cellular-Based Location System . . . . . . . . . . . . . . . . . . . . . . . David Navalho and Nuno Pregui¸ca

528

Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

541

Distributed Systems and Algorithms Omer Rana1 , Giandomenico Spezzano1 , Michael Gerndt2 , and Daniel S. Katz2 1

Topic Chairs Members

2

Despite an impressive body of research, parallel and distributed computing remains a complex task prone to subtle software issues that can affect both the correctness and the performance of the computation. The increasing demand to distribute computing over large-scale parallel and distributed platforms, such as grids and large clusters, often combined with the use of hardware accelerators, overlaps with an increasing pressure to make computing more dependable. To address these challenges, the parallel and distributed computing community continuously requires better tools and environments to design, program, debug, test, tune, and monitor parallel programs. This topic aims to bring together tool designers, developers, and users to share their concerns, ideas, solutions, and products covering a wide range of platforms, including homogeneous and heterogeneous multi-core architectures. Contributions with solid theoretical foundations and experimental validations on production-level parallel and distributed systems were particularly valued. This year, we encouraged submissions proposing intelligent monitoring and diagnosis tools and environments, which can exploit behavior knowledge to detect programming bugs or performance bottlenecks and help ensure correct and efficient parallel program execution. This year nineteen papers were submitted to Topic 8 representing a wide spectrum of research areas in distributed computing. Of the nineteen submissions, seven have been accepted for publication. Each paper was reviewed by four reviewers (two papers received only three reviews). This year 20 papers were submitted to this topic. Each paper was reviewed by at least three reviewers and we selected 7 papers as definite accepts for the conference (35%). It was interesting to see papers focusing on emerging themes such as multi-core and GPUs, along with papers that covered more established themes such as program profiling, performance analysis, work ow management and application tuning. The paper entitled ”Starscheck: A tool to find errors in task-based parallel programs” received the highest review score and was accepted as a distinguished paper. This paper discusses a tool to debug Star Superscalar programs, producing a warning if a task or the main thread within a program generates an invalid access. We would like to thank the authors who submitted a contribution, the EuroPar Organizing Committee, and the referees who provided useful and timely comments.

P. D’Ambra, M. Guarracino, and D. Talia (Eds.): Euro-Par 2010, Part I, LNCS 6271, p. 1, 2010. c Springer-Verlag Berlin Heidelberg 2010

Starsscheck: A Tool to Find Errors in Task-Based Parallel Programs Paul M. Carpenter, Alex Ramirez, and Eduard Ayguade Barcelona Supercomputing Center, C/Jordi Girona, 31, 08034 Barcelona, Spain {paul.carpenter,alex.ramirez,eduard.ayguade}@bsc.es

Abstract. Star Superscalar is a task-based programming model. The programmer starts with an ordinary C program, and adds pragmas to mark functions as tasks, identifying their inputs and outputs. When the main thread reaches a task, an instance of the task is added to a run-time dependency graph, and later scheduled to run on a processor. Variants of Star Superscalar exist for the Cell Broadband Engine and SMPs. Star Superscalar relies on the annotations provided by the programmer. If these are incorrect, the program may exhibit race conditions or exceptions deep inside the run-time system. This paper introduces Starsscheck, a tool based on Valgrind, which helps debug Star Superscalar programs. Starsscheck verifies that the pragma annotations are correct, producing a warning if a task or the main thread performs an invalid access. The tool can be adapted to support similar programming models such as TPC. For most benchmarks, Starsscheck is faster than memcheck, the default Valgrind tool.

1

Introduction

There is a general need for verification and debug tools for parallel programming, to make it possible to develop reliable and correct parallel software. Debuggers are suitable for sequential and explicitly threaded applications. Data race detectors find concurrent accesses to shared memory, which may result in undefined behaviour. This paper presents Starsscheck, a tool that finds undefined behaviour in a task-based parallel program. Star Superscalar (StarSs) is a task-based programming model. The programmer starts with a sequential C program, and adds pragma annotations to mark tasks, identifying their inputs and outputs. Execution begins in the master thread on a single processor; when it reaches a task, an instance of that task is added to a run-time dependency graph, and later executed in a worker thread. The StarSs run-time system renames and tracks arrays in a similar way to register renaming in a superscalar processor, so the master thread does not have to wait before “forwarding” an output array on to a different task. There are several variants of StarSs: CellSs [1] supports the Cell Broadband Engine (CBE) [2], and SMPSs [3] targets SMP multicores. Figure 1 shows the bmod function from LU factorisation. The pragma declares the function to be a task, and specifies the direction of each array argument as P. D’Ambra, M. Guarracino, and D. Talia (Eds.): Euro-Par 2010, Part I, LNCS 6271, pp. 2–13, 2010. c Springer-Verlag Berlin Heidelberg 2010

Starsscheck: A Tool to Find Errors in Task-Based Parallel Programs

1 2 3 4 5 6 7 8 9 10

3

#pragma c s s task input (row , c o l ) inout ( inner ) void bmod( f l o a t row [ 3 2 ] [ 3 2 ] , float col [ 3 2 ] [ 3 2 ] , float inner [ 3 2 ] [ 3 2 ] ) { f o r ( i n t i =0; i ) { SQLQuery query = new SQLQuery; |- query -| => query.expression; |- resource -| => query.resource; return PE( => ); }

function PE5() PE( => ) { ImageToMatrixActivity itma = new ImageToMatrixActivity; MedianFilterActivity mfa = new MedianFilterActivity; itma.output => mfa.input; return PE( => ); }

Fig. 1. Example DISPEL functions

In DISPEL, processing element types represent abstract semantics; whereas, the concrete semantics is expressed as a directed graph of components, where the edges represent the flow of data between components. Each component of this directed graph is either a composite processing element (in which case, we have a higher-order hierarchical abstraction), or a concrete processing element with an implementation of a computational object. In the admire context, we use OGSA-DAI activities for the concrete implementations. Each of these composite and concrete processing elements are catalogued using wrapper functions. Thus, every abstraction is written as a DISPEL function, and every function returns either a processing element instance, or a composition of components. Figure 1 lists two example DISPEL functions: PE1 returns an OGSA-DAI SQL query activity; whereas, PE5 returns a composition with abstract components ImageToMatrixActivity and MedianFilterActivity.

320

2.3

G. Yaikhom et al.

Semantic Registry

Semantic registries are service providers with facilities to import, export, and search workflow patterns. All abstract semantics are defined and registered with a semantic registry. In addition to this, possible concrete semantics for each of these are also defined and registered as DISPEL functions. On the user front, semantic registries expose workflow patterns by exposing the processing element types. This allows users to search and re-use patterns during process engineering. In the prototype implementation, relationships between processing element types are captured using ontology; and the searches are carried out using the SPARQL query language1 . Semantic registries also play a crucial role during workflow execution. They provide various mechanisms for realising the abstract semantics of a workflow pattern by mapping them to suitable concrete semantic objects (either compositions or activities). This mapping is carried out by exporting from the registry the function definition that corresponds to the chosen concrete semantics, where the choice of a particular mapping is dictated by the properties associated with the pattern, and the execution parameters (e.g., environment features). 2.4

Enactment Gateways

In the admire architecture, the execution environments are not exposed for immediate access; instead, they are concealed behind a service provider called the enactment gateway. The enactment gateway has been assigned the task of interpreting and executing workflow compositions written in DISPEL.

B

D F

A C

E workflow composition

enactment gateway B A

F D C E

Fig. 2. Expansion of workflow patterns 1

SPARQL query language, http://www.w3.org/TR/rdf-sparql-query/

Federated Enactment of Workflow Patterns

321

In figure 2, we show how an enactment gateway expands patterns (shown in rectangles) into concrete semantic objects (shown in circles). The workflow composition received by this enactment gateway uses six different patterns: A, B, C, D, E, and F. The data dependencies between these patterns are given by the corresponding directed edges. Before interpreting this workflow composition, the enactment gateway gathers information about the execution environment that is currently available to it (resource availability, load, communication latency, etc.). This information is then used to query the semantic registry. Eventually, depending on the execution parameters received from the enactment gateway, the semantic registry returns the most appropriate pattern expansions as DISPEL functions. By executing each of these functions, recursively when higher-order abstractions are involved, the workflow composition is expanded to a form where all of the components are computational objects (e.g., OGSA-DAI activities). 2.5

Federated Execution

Higher-order abstraction leads to federated execution. A workflow composition submitted to an enactment gateway could be partitioned into sub-workflows, which are then delegated to a federation of enactment gateways. This facilitates execution of a complex workflow composition using various computational resources in parallel, thus increasing the throughput. Since we are interested in large-scale data intensive computations, the ability to partition a workflow composition presents an opportunity for significant performance improvement. Furthermore, since the primary enactment gateway (the enactment gateway which received the workflow composition) handles this internally, the entire federated execution is concealed from the domain experts. Let us consider the workflow expansion shown in figure 2. This expansion has six partitions, each of which can be submitted to a federation of enactment gateways associated with the primary gateway. This process of partition and submission continues until an enactment gateway is ready to begin the execution: instantiation of the services and establishment of communication links between these service instances. Optimisation. Since each enactment gateway receiving a partition can treat its sub-workflow independently of the remaining partitions, it has the freedom to interpret and execute the sub-workflow using a form which best suits the resources available to it. We can describe this by taking two commonly occurring patterns as examples: pattern B as a Map-Reduce; and pattern E as a Pipeline. The enactment gateway which receives partition B can decide the best mapping (multiplicity of the concurrent processes in the Map-Reduce) based on the number of parallel processors available to it. Similarly, the enactment gateway which receives partition E can decide whether the three stages of the pipeline should be mapped on the same processor using multi-threading (in cases of large data transfers between stages), or to map each stage to a separate processor (in cases where the data transfers are low but the stages are computation intensive).

322

G. Yaikhom et al.

SQL Query (PE1) SampleSplit (PE2)

......

Read Image File (PE3)

Image Rescale (PE4)

......

Image Rescale (PE4)

Image Denoise(PE5)

......

Image Denoise(PE5)

Feature Generation(PE6)

......

Feature Generation(PE6)

FS1

FS1 FS2 FS3

Feature Extraction(PE8)

Feature Selection (PE7)

Task Parallelism

Read Image File (PE3)

Data Parallelism

Data Subset n

Data Subset 1

Feature Extraction(PE8)

Classifier (PE9) Prediction Evaluation(PE10)

Fig. 3. Workflow composition for EURExpress-II experiment

Concrete example. To illustrate this with a concrete example, consider the workflow composition shown in figure 3. This is representative of the workflow used in our EURExpress-II experiment (discussed in section 3). The workflow uses several components, and we saw in figure 1 possible concrete semantics for two such components. Let us focus on the section identified as data parallel. In this section, the image processing part is composed of PE3, PE4, PE5, and PE6. In the actual DISPEL implementation, we define this as a composition (not shown in the figure). As part of the optimisation carried out by the enactment engine, we take advantage of this data parallelism by replicating the image processing composition. This is done when PE2 and PE7 are expanded. Based on the execution environment available, the enactment engine parametrises calls to function PE2 and PE7 (DISPEL functions returned by the registry) by supplying the number of times the image processing composition should be replicated. When further expansions have reached the computational activities, resources are assigned.


323

If we consider the replicated composition involving PE3, PE4, PE5, and PE6, we can see that it is in fact a pipeline. Hence, the enactment engine exploits the task parallelism by assigning each of these components to different resources if the pipeline components expand to computational activities. On the other hand, if these components are composite, the enactment engine might send them to different enactment gateways that are part of the federation. Thus, each of the enactment gateways determines how each of the computational activities in the pipeline stages are assigned to the available resources. In short, federated execution helps us achieve the full potential of a truly distributed computational platform. In the above example, the workflow partitioning is limited to top-down expansion, where each enactment gateway takes advantage of the available environment by parametrising the functional representations received from the semantic registry. We are currently investigating arbitrary partitioning and transformation (e.g., component sequence re-ordering) of the workflow using flow and cost-model analysis.

3

Case-Study: Data Mining and Integration

To demonstrate the ideas described in the previous sections, we shall now discuss a case-study from the life sciences. We have carried out a data mining and integration operation using the EURExpress-II [11] project data set. For this experiment, we have also implemented the required concrete semantic objects using the OGSA-DAI[3] framework. The EURExpress-II project2 builds an atlas database for a developing mouse embryo. It uses automated processes for in situ hybridisation on all of the genes from one stage of the developing embryo. The outcome of this automated process is a collection of images corresponding to the various sections of the embryo, with unique stains marking specific gene expressions. These images are then annotated by human curators, which involves tagging the images with anatomical terms from the ontology of a developing mouse embryo according to the gene expressions. The project has, so far, tagged 80% of the images manually, which amounts to 4 terabytes of data in total. The aim of our data mining and integration case-study is to automate the tagging process for the remaining 20%. This involves classifying 85,824 images using a vocabulary of 1,500 anatomical terms. 3.1

Workflow Composition

The automated image annotation through data mining requires three main stages. In the training stage, we train the automated classifier using training data sets taken from the human annotated collection. In the testing stage, we test the classifier against data sets from the human annotated collection that was not used during the training. Finally, in the deployment stage, we use the automated classifier to annotate the unannotated images. 2

EURExpress-II Project, http://www.eurexpress.org/ee/

324

G. Yaikhom et al.

The sub-tasks in each of the three stages are described as follows: – Image integration: Before the data mining process begins, image data located at various sources (as files) are integrated with annotation data from various databases. This produces images with annotations. – Image processing: Since the sizes of the images could vary, they are normalised to a standard size using image rescaling. The noise in these images is then reduced using median filtering. – Feature generation: From the stains in the embryo sections, which represent gene expression patterns, we generate features using wavelet transformations. For the images in the EURExpress-II data set, the features generated can be quite large; for instance, for an image with 300x200 pixels, the number of features could reach up to 60,000. – Feature selection and extraction: In order to build an automated classifier, we have to reduce the number of features that are taken into account. This is done by selecting the most significant feature subset. This subset is then applied during the feature extraction process, which transforms the original features into feature vectors that can be used by the classifier as input. – Classifier construction: The feature vectors are finally used to classify the unannotated images. All of the components, which correspond to these sub-tasks, are shown in figure 3. Due to the streaming parallelism inherent in the admire architecture, we observe two patterns of parallelism at the highest-level. For the image integration and image processing tasks, we exploit the data parallelism where image scaling and noise reduction are independent for each of the images. On the other hand, for the feature generation, selection, and extraction phases, we exploit task parallelism. For instance, feature generation in the testing stage and feature selection in the training stage are independent tasks. As discussed in section 2.5 for the embedded pipeline, we recursively exploit lower-level parallelism.

Table 1. The total execution time (in seconds), plotted using the number of computational nodes against the number of images

Node 1 2 3 4 5 6 7 8

800 70.680 36.183 23.675 18.364 15.313 12.925 11.588 10.220

1600 141.162 69.846 47.160 35.990 29.483 24.357 21.430 19.154

Number of images 3200 4800 6400 283.399 426.015 564.927 141.614 214.035 284.753 93.944 142.857 191.171 70.548 107.783 144.252 56.894 85.494 115.467 47.705 71.808 96.17 41.016 61.45 82.657 36.436 54.157 72.274

12800 1139.069 573.158 382.037 289.562 232.597 195.802 168.478 148.068

19200 1721.489 865.271 581.403 437.669 350.499 295.397 253.524 225.335


3.2

325

Experimental Results

The total workflow execution times (in seconds) and the corresponding speedups of the automated data mining experiments are shown in tables 1 and 2. Their graphical representations are shown in figures 4 and 5 respectively. These results are based on an experimental setup using nine dedicated machines that are connected over a local 100mbps network. A distributed computational framework (enactment platform) was created over this system using the OGSA-DAI framework 3.1, Java 1.6, Globus Toolkit 4.2 (web services core), and Jakarta Tomcat 5.5. Of the nine machines, a MacBook Pro with Intel 2GHz Core 2 Duo and 2GB RAM was configured as the enactment gateway, where the workflow composition was submitted. The remaining eight machines were workstations with Intel 2.4 GHz Core Duo with 2GB RAM each. Table 2. The speedup, plotted using the number of computational nodes against the number of images

Node 1 2 3 4 5 6 7 8

800 1.000 1.953 2.985 3.849 4.616 5.469 6.099 6.916

Number of images 3200 4800 6400 1.0000 1.0000 1.0000 2.0012 1.9904 1.9839 2.9763 2.9706 2.9678 3.9639 3.9490 3.9416 4.9586 4.9276 4.9123 5.9662 5.9102 5.8827 6.9924 6.9005 6.8555 8.0434 7.9022 7.8335

1600 1.000 2.021 2.993 3.922 4.788 5.796 6.587 7.370

12800 1.0000 1.9874 2.9637 3.9305 4.8894 5.8418 6.7891 7.7326

19200 1.0000 1.9895 2.9623 3.9268 4.8818 5.8283 6.7672 7.6995

1800

Workflow Execution Time (s)

1600

800 images 1600 images 3200 images 4800 images 6400 images 12800 images 19200 images

1400 1200 1000 800 600 400 200 0 0

1

2

3

4

5

6

7

8

9

Number of Computing Nodes

Fig. 4. Workflow execution times after exploiting streaming parallelism

326

G. Yaikhom et al. 9 8 8 7 7.5

Speedup

6 5

7 7

8

4

y=x 800 images 1600 images 3200 images 4800 images 6400 images 12800 images 19200 images

3 2 1 0 0

1

2

3

4

5

6

7

8

9

Number of Computing Nodes

Fig. 5. Speedup when using multiple computational nodes with streaming parallelism

The graphs in figures 4 and 5 show that the streaming parallelism inherent in the admire architecture can be exploited fruitfully to yield close-to-linear speedup; except for the case of 3200 images run on eight computational nodes, where we see super-linear speedup. This anomaly is possibly caused by caching in the nodes from previous experiments.

4

Related Work

Patterns, as a notion for the abstraction of complexity, permeate various fields of investigation involving structure and coordination. The idea of using patterns as a multi-purpose tool for software development is discussed in detail in [16]. Riehle and Z¨ ullighoven defined software patterns, their forms and context, and emphasised the need for cataloguing and referencing these patterns during different phases of software development. One such area of research, which is currently very active, is the usage of patterns to capture and abstract parallelism in both distributed and multi-core systems (see [14] for a survey of this field). With regard to workflow management systems, abstraction of parallelism is an important area of research since service-oriented architectures are primarily concurrent. Some of the most prominent systems in this category are Taverna [15], Kepler [2], and Pegasus [7]. Investigations into the structure of workflows, and coordination between primitive components, have previously been done using various formal tools such as Petri Nets and UML. For instance, Ellis and Nutt [10] investigated the modelling and enactment of workflows using Petri Nets; whereas, Dumas and Hofstede [8] used UML activity diagrams to express workflows. Evolution of workflows, and mechanisms to capture and specify those changes, was reported in [9] by Ellis et al. and in [12] by Joeris and Herzog. Following the survey [17] due to Rinderle et


327

al. on the correctness criteria of evolving workflow systems, Sun and Jiang [18] analysed dynamic changes to workflows using Petri Nets. The idea of using patterns as a tool to encapsulate these analyses began with [19], when Aalst et al. further advanced the level-of-abstraction by using workflow patterns to capture recurring structures of coordination between primitive components. In this paper, we have addressed an important aspect of workflow patterns which is lacking in these investigations—the high-order abstraction of patterns and the dynamic runtime assignment of concrete semantics. By introducing semantic registries, it is now possible to catalogue well-defined patterns based on their abstract semantics; while facilitating algorithm experts and computing engineers in providing various implementations of the concrete semantics. This freedom automatically facilitates parallelisation and resource-specific optimisations, since the expansion of patterns, inside an abstract workflow composition, takes place inside the enactment gateway, concealed from the domain experts.

5

Conclusions

We have addressed two research questions concerning workflows: 1) how do we abstract and catalogue recurring workflow patterns?; and 2) how do we facilitate optimisation of the mapping from workflow patterns to actual resources at runtime? To adapt to the practical reality of service provider multiplicity and infrastructure evolution, our approach relies on using well-defined patterns that can be mapped to concrete computational resources depending on the execution environment available. The novelty of our approach lies in our ability to separate abstract semantics (the functional aspects of a pattern) from the range of concrete semantics (the actual implementation using various middleware services); and the federated execution model, which allows execution platforms to share the workload transparently using a federation of distributed resources. Using a data mining case-study drawn from the life sciences, we have demonstrated experimentally the feasibility of this architecture. Acknowledgments. The work presented in this paper is funded by the European Commission under Framework 7 ICT 215024.

References 1. Agostini, A., Michelis, G.D.: Improving Flexibility of Workflow Management Systems. In: Business Process Management, Models, Techniques, and Empirical Studies, London, UK, pp. 218–234. Springer, Heidelberg (2000) 2. Altintas, I., Berkley, C., Jaeger, E., Jones, M., Ludascher, B., Mock, S.: Kepler: an extensible system for design and execution of scientific workflows. In: Proc. of 16th Intl. Conf. on Scientific and Statistical Database Management, pp. 423–424 (June 2004) 3. Antonioletti, M., Atkinson, M.P., Baxter, R.M., Borley, A., Chue Hong, N.P., Collins, B., Hardman, N., Hume, A.C., Knox, A., Jackson, M., Krause, A., Laws, S., Magowan, J., Paton, N.W., Pearson, D., Sugden, T., Watson, P., Westhead, M.: The design and implementation of grid database services in ogsa-dai. Concurrency - Practice and Experience 17(2-4), 357–376 (2005)

328

G. Yaikhom et al.

4. Atkinson, M.P., van Hemert, J.I., Han, L., Hume, A., Liew, C.S.: A distributed architecture for data mining and integration. In: Proc. of the Second Intl. Workshop on Data-Aware Distributed Computing, pp. 11–20. ACM, New York (2009), doi:10.1145/1552280.1552282 5. Casati, F., Ceri, S., Pernici, B., Pozzi, G.: Conceptual Modelling of WorkFlows. In: Proc. of the 14th Intl. Conf. on Object-Oriented and EntityRelationship Modelling, London, UK, pp. 341–354. Springer, Heidelberg (1995), doi:10.1007/BFb0020545 6. Cole, M.: Bringing skeletons out of the closet: a pragmatic manifesto for skeletal parallel programming. Parallel Comput. 30(3), 389–406 (2004) 7. Deelman, E., Singh, G., Su, M.-H., Blythe, J., Gil, Y., Kesselman, C., Mehta, G., Vahi, K., Berriman, G.B., Good, J., Laity, A.C., Jacob, J.C., Katz, D.S.: Pegasus: A framework for mapping complex scientific workflows onto distributed systems. Scientific Programming 13, 219–237 (2005) 8. Dumas, M., ter Hofstede, A.H.M.: UML Activity Diagrams as a Workflow Specification Language. In: Proc. of the 4th Intl. Conf. on The Unified Modeling Language, Modeling Languages, Concepts, and Tools, London, UK, pp. 76–90. Springer, Heidelberg (2001) 9. Ellis, C., Keddara, K., Rozenberg, G.: Dynamic change within workflow systems. In: Proc. of Conf. on Organizational Computing Systems, pp. 10–21. ACM, New York (1995) 10. Ellis, C.A., Nutt, G.J.: Modeling and enactment of workflow systems. In: Proc. of the 14th Intl. Conf. on Application and Theory of Petri Nets, London, UK, pp. 1–16. Springer, Heidelberg (1993) 11. Han, L., van Hemert, J.I., Baldock, R., Atkinson, M.: Automating gene expression annotation for mouse embryo. In: Huang, R., Yang, Q., Pei, J., Gama, J., Meng, X., Li, X. (eds.) Advanced Data Mining and Applications. LNCS, vol. 5678, pp. 469–478. Springer, Heidelberg (2009) 12. Joeris, G., Herzog, O.: Managing Evolving Workflow Specifications. In: Proc. of the 3rd IFCIS Intl. Conf. on Cooperative Information Systems, Washington, DC, USA, pp. 310–321. IEEE Computer Society, Los Alamitos (1998) 13. Kiepuszewski, B., ter Hofstede, A.H.M., Bussler, C.J.: On Structured Workflow Modelling. In: Wangler, B., Bergman, L.D. (eds.) CAiSE 2000. LNCS, vol. 1789, pp. 431–445. Springer, Heidelberg (2000) 14. Mattson, T.G., Sanders, B.A., Massingill, B.L.: Patterns for Parallel Programming. Addison-Wesley, Reading (2005) 15. Oinn, T., Addis, M., Ferris, J., Marvin, D., Senger, M., Greenwood, M., Carver, T., Glover, K., Pocock, M.R., Wipat, A., Li, P.: Taverna: a tool for the composition and enactment of bioinformatics workflows. Bioinformatics 20, 3045–3054 (2004) 16. Riehle, D., Z¨ ullighoven, H.: Understanding and using patterns in software development. Theor. Pract. Object Syst. 2(1), 3–13 (1996) 17. Rinderle, S., Reichert, M., Dadam, P.: Correctness criteria for dynamic changes in workflow systems: a survey. Data Knowl. Eng. 50(1), 9–34 (2004) 18. Sun, P., Jiang, C.: Analysis of workflow dynamic changes based on Petri net. Inf. Softw. Technol. 51(2), 284–292 (2009) 19. van der Aalst, W., ter Hofstede, A., Kiepuszewski, B., Barros, A.: Workflow Patterns. Distributed and Parallel Databases 14(1), 5–51 (2003), doi:10.1023/A:1022883727209

A Distributed Approach to Detect Outliers in Very Large Data Sets Fabrizio Angiulli1 , Stefano Basta2 , Stefano Lodi3 , and Claudio Sartori3 1

DEIS-UNICAL Via Pietro Bucci, 41C – 87036 Rende (CS), Italy [email protected],it 2 ICAR-CNR Via Pietro Bucci, 41C – 87036 Rende (CS), Italy [email protected] 3 DEIS-UNIBO Via Risorgimento, 2 – 40136 Bologna, Italy {stefano.lodi,claudio.sartori}@unibo.it

Abstract. We propose a distributed approach addressing the problem of distance-based outlier detection in very large data sets. The presented algorithm is based on the concept of outlier detection solving set ([1]), which is a small subset of the data set that can be provably used for predicting novel outliers. The algorithm exploits parallel computation in order to meet two basic needs: (i) the reduction of the run time with respect to the centralized version and (ii) the ability to deal with distributed data sets. The former goal is achieved by decomposing the overall computation into cooperating parallel tasks. Other than preserving the correctness of the result, the proposed schema exhibited excellent performances. As a matter of fact, experimental results showed that the run time scales up with respect to the number of nodes. The latter goal is accomplished through executing each of these parallel tasks only on a portion of the entire data set, so that the proposed algorithm is suitable to be used over distributed data sets. Importantly, while solving the distance-based outlier detection task in the distributed scenario, our method computes an outlier detection solving set of the overall data set of the same quality as that computed by the corresponding centralized method.

1

Introduction

Detecting outliers in large data sets, that is finding out examples considerably dissimilar, exceptional or inconsistent with respect to the remaining data [7], is an important research field that has practical applications in several domains such as fraud detection, network intrusion detection, data cleaning, medical diagnosis, and marketing segmentation. Unsupervised approaches to outlier detection are able to discriminate each datum as normal or exceptional when no training examples are available. Among the unsupervised approaches, distancebased outlier detection methods distinguish an object as outlier on the basis of P. D’Ambra, M. Guarracino, and D. Talia (Eds.): Euro-Par 2010, Part I, LNCS 6271, pp. 329–340, 2010. c Springer-Verlag Berlin Heidelberg 2010

330

F. Angiulli et al.

the distances to its nearest neighbors [9,13,5,3,1,14,6,2]. These approaches differ in the way the distance measure is defined, but in general, given a data set D of objects, an object p can be associated with a weight or score, that is a function of its k nearest neighbors distances. Intuitively, the weight/score measures how much an object is dissimilar from its neighbors. In this work we follow the definition provided in [3]. In particular, given a data set object p, the weight of p is the sum of the distances from p to its k nearest neighbors. Let w∗ be the n-th greatest weight of an object in D. A (top-n) distance-based outlier in D is an object having weight greater than or equal to w∗ . The outlier detection task can be very time consuming and recently there has been an increasing interest in parallel/distributed implementations of outlier detection methods. Hung and Cheung [8] presented a parallel version, called PENL, of the basic NL algorithm [9]. PENL is based on a definition of outlier employed in [9], that is, the distance-based outliers are those points for which there are less than k points within the distance δ in the input data set. This definition does not provide a ranking of outliers and needs to determine an appropriate value of the parameter δ. Moreover, PENL is not suitable for mining distributed data sets, because it requires that all the data set is transferred among all the network nodes. Lozano and Acuna [11] proposed a parallel version of Bay’s algorithm [5], which is based on a definition of distance-based outlier coherent with the one used here. However the method did not scale well in two out of the four experiments presented. Moreover, this parallel version does not deal with the drawbacks of the corresponding centralized version, which is sensitive to the order and to the distribution of the data set. Ghoting, Parthasarathy and Otey in [12] and Koufakou and Georgiopoulos in [10] proposed their strategies for distributed high-dimensional data sets. These methods are based on definitions of outlier which are completely different from the definition employed here, as that they are based on the concept of support rather than on that of distance in order to deal with mixed attribute data sets. More precisely in both works an anomaly score function is computed by taking into account the dependencies among the attributes in the data set and outliers are then the points that contradict these dependencies. In this work we present a distributed approach to detect distance-based outliers, called DistributedSolvingSet , based on the concept of outlier detection solving set [1]. The outlier detection solving set is a subset S of the data set D that includes a sufficient number of objects from D to allow considering only the distances among the pairs in S × D to obtain the top-n outliers. The solving set is a learned model that can be seen as a compressed representation of D. It has been shown that it can be used to predict if a novel object q is an outlier or not by comparing q only with the objects in S instead of considering all the objects in D. Since the solving set contains at least the top-n outliers, computing the solving set amounts to simultaneously solve the outlier detection task. As a starting point to assemble our distributed approach, we have used the concept of outlier detection solving set and the above described weight as distance function, since these definitions are easily extended to a parallel environment. In

A Distributed Approach to Detect Outliers in Very Large Data Sets

331

particular, as we will see in the Section 2, a single iteration of the main cycle of the sequential SolvingSet is efficiently translated according to a parallel/distributed implementation. The DistributedSolvingSet bases its effectiveness on two main features. The former one is represented by the parallel core of the computing procedure, which allows to get a great decrease in runtime with respect to the sequential algorithm proposed in [1] without losing quality in the retrieved solution. The latter feature is that each parallel task is designed to work on a subset of the data set. Then, this approach is suitable to be used both in the parallel and distributed scenario. Other than preserving the correctness of the result, the proposed schema exhibited excellent performances. As a matter of fact, experimental results showed that the run time scales up with respect to the number of nodes. Importantly, while solving the distance-based outlier detection task in the distributed scenario, our method computes an outlier detection solving set of the overall data set, which, as already recalled, is a small subset of the data set that can provably used for predicting novel outliers [1], of the same quality as that computed by the corresponding centralized method. The rest of the paper is organized as follows. In Section 2 we introduce our approach and provide the DistributedSolvingSet . Section 3 reports and discusses experimental results. Finally, Section 4 draws some conclusions.

2

Distributed Solving Set Algorithm

In this section we describe the DistributedSolvingSet algorithm by which it is possible to compute a solving set and the top n outliers for a data set. The DistributedSolvingSet algorithm is mainly based on the SolvingSet algorithm. Next we briefly recall the SolvingSet algorithm. SolvingSet algorithm. At each iteration (let us call j the iteration number), the SolvingSet algorithm compares each data set object with a selected small subset of the overall data set, called Cj (for candidate objects), and stores the k neighbors found so far with respect to the set C1 ∪ . . . ∪ Cj . From these stored neighbors, an upper bound to the true weight of each data set object can thus be obtained. Moreover, since the candidate objects have been compared with all the data set objects, their true weight is known. The objects having weight upper bound lower than the n-th greatest weight associated with a candidate object, are called non active (since these objects cannot belong to the top-n outliers), while the others are called active. At the beginning, C1 contains randomly selected objects from D, while, at each subsequent iteration j, Cj is built by selecting, among the active objects of the data set not already inserted in C1 , . . . , Cj−1 during the previous iterations, the objects having the maximum current weight upper bounds. During the computation, if an object becomes non active, then it will not be considered any more for insertion into the set of candidates, because it cannot be an outlier. As the algorithm processes new objects, more accurate weights are computed and the number of non active objects increases more

332

1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: 15: 16: 17: 18: 19: 20: 21: 22:

F. Angiulli et al. Algorithm DistributedSolvingSet begin DSS = ∅; OU T = ∅; d = li=1 di ; for each node Ni ∈ N NodeInit(m ddi , Ci ); C = li=1 Ci ; act = d; minOU T =0; while (C = ∅) { DSS = DSS ∪ C; for each node Ni ∈ N NodeComp(minOU T, C, act, LN N Ci , LCi , acti ); act = li=1 acti ; for each q ∈ C { N N C[q] = get k NN( li=1 LN N Ci [q]); UpdateMax(OU T, q, Sum(N N C[q]) ) } minOU T = Min(OU T ); C = ∅; for each p ∈ li=1 LCi C = C ∪ {p}; } end Fig. 1. The DistributedSolvingSet algorithm

quickly. The algorithm stops when no more objects have to be examined, i.e. when all the objects not yet selected as candidates are non active, and thus Cj becomes empty. The solving set is the union of the sets Cj computed during each iteration. DistributedSolvingSet. The DistributedSolvingSet algorithm follows the same strategy of the SolvingSet algorithm. It consists of a main cycle executed by a supervisor node, which iteratively schedules the following two tasks: (i) the core of the computation which is simultaneously carried out by all the other nodes; (ii) the synchronization of the partial results returned by each node after completing its job. Now we describe in detail the algorithm DistributedSolvingSet , which is shown in Figure 1. Assume a generic distributed environment consisting of l + 1 nodes is given, where N0 is the supervisor node and the remaining nodes N1 , . . . , Nl are called local nodes. Let D be a data set composed of the union of the l data sets Di , with each data set Di located at node Ni . We call D the global data set and the generic Di a local data set. Table 1 recalls some of the data structures and functions employed by the algorithm; the meaning of the items not described here should be made clear by the algorithm description.


333

Table 1. Data structures and functions of the algorithm DistributedSolvingSet LN N Ci N Ni NNC LCi OUT get k NN Sum UpdateMax

UpdateMin

Min

is an array of m heaps LN N Ci [q], each of which is associated with an object q of the current candidate set C and contains the distances separating q from its k nearest neighbors in the local data set Di is an array of di heaps N Ni [p], each of which is associated with an object p of the local data set Di and contains the distances separating p from its k nearest neighbors with respect to the so far seen candidate sets C is an array of m arrays N N C[q], each of which is associated with an object q of the current candidate set C and contains the distances separating q from its k nearest neighbors in the whole data set D is a heap storing mi pairs p, w, where p is an object of Di and w is the associated weight upper bound; it is employed to store the local objects to be employed as candidates in the next iteration is a heap of n pairs p, w, where p is an object of D and w is the associated true weight; it is employed to store the current top-n outliers of the whole data set D this function returns the k smallest distances among those received in input; it is employed to compute the true k nearest neighbors of the candidate objects this function computes the weight of a generic object by adding its k nearest neighbor distances the function call UpdateMax(OUT, q, Sum(N N C[q])) updates the heap OUT by substituting the pair p, w of OUT having associated the minimum weight w with the novel pair q, Sum(N N C[q]), provided that Sum(N N C[q]) > w the function call UpdateMin(LN N C[p], q, δ) updates the heap LN N Ci [p] by substituting the pair s, σ of LN N Ci [p] havin associated the maximum distance σ with the novel pair q, δ, provided that δ < σ the function call Min(OUT ) returns the smallest value w associated with a pair p, w stored in the heap OUT . Thus its value is a lower bound to the weight of the n-th outlier of D. This means that the objects of D having weight upper bound less than Min(OUT ) cannot be outliers.

The algorithm receives in input the number l of local nodes, the values di representing the sizes of the local data sets Di , a distance function dist on the objects in D, the number k of neighbors to consider for the weight calculation, the number n of top outliers to find, an integer m ≥ k, representing the number of objects to be added to the solving set at each iteration. It outputs the solving set DSS and the set OU T containing the top-n outliers in D. At the beginning of the execution of the algorithm, DSS and OU T are initialized to the empty set (lines 1-2), while the set of candidates C is initialized by picking at random m objects from the whole data set D (lines 3-6; refer to the procedure NodeInit for details). The main cycle (lines 9-22) will stop when the set C becomes empty. The points currently belonging to C are added to the solving set DSS (line 10). At the beginning of each iteration, the set of candidates C is sent to the procedures NodeCompi running at each local node (the instance NodeCompi runs at node Ni , for i = 1, 2, . . . , l), together with the value minOU T , representing a lower bound to the weight of the top-n outlier, and the total number act of active objects. Each NodeCompi returns: – the data structure LLN Ci containing the k distances to the nearest neighbors in the local data set Di of the candidate objects in C; – the updated number acti of active objects in the local data set Di , – the data structure LCi containing mi objects coming from the local data set Di to be used to build the set of candidates C for the next iteration;

334

F. Angiulli et al.

the number mi represents the percentage of the active objects in Di , and i is defined as mi = m act (note that the structure LCi returned to the act supervisor node does not include the weights associated with the objects there contained). After having collected all the results of the procedures NodeCompi , the true weight associated with the candidate objects in the set C can be computed (lines 14-17). The k nearest neighbors’ distances in the whole data set of each candidate object q are obtained from the distances stored in the data structures LN N Ci [q] (line 15); in fact, the k lowest distances extracted from the union of LN N Ci [q] for each node i represent the distances from q to its k nearest neighbors in the global data set. Then the heap OU T containing the current top-n outliers is updated (line 16). We point up that by computing the true weight for each candidate object, the distributed algorithm finds the same outliers as those detected by the centralized one, and therefore the distributed solution is correct w.r.t. the centralized one. Finally, the lower bound to the weight of the nth top outlier is updated (line 18), and the novel set of candidate objects C is built (lines 19-21). We now give the details of the two procedures NodeInit and NodeComp. NodeInit. The procedure NodeIniti (see Figure 2) runs at the local node Ni . It receives in input an integer value mi and returns a randomly selected set Ci of mi data points belonging to the local data set. The variable acti , that is the number of the active data points in the local data set, is set to the local data set size. Finally, both the variable acti and the set Ci are stored in node storage memory. NodeComp. The procedure NodeCompi , shown in Figure 2, runs at local node Ni . First of all, the value acti and the set of local candidates Ci (computed either by NodeIniti or during the previous execution of NodeCompi ) are loaded from the local memory (line 27). Then, the objects in Ci are removed from the local data set (line 27) and the number acti of local active objects is consequently updated (line 28). Before starting the comparison of the local objects with the current candidate objects, the heap LCi is dimensioned (by the procedure init) to accommodate mi objects (line 29). Moreover, the heaps LN N Ci [p] referring to the local candidate objects p are initialized with the corresponding heaps N Ni (lines 30-31), while the other heaps LN N Ci [p] are initially empty. Thus, only the local node that generated the local candidate p is aware of the distances to the nearest neighbors of p with respect to all the previous sets of candidates (which are stored in the heap N Ni [p]). By adopting this strategy, we achieved vast communication savings. The supervisor node will then take care of selecting the true nearest neigbhor distances of p among the distances stored in all the heaps LN N Ci [p] (i = 1, . . . , l). At this point, the weights of the objects in Ci is computed by comparing each object in Ci with all the objects in the local data set. This computation is split into three steps in order to avoid duplicated distance computations (lines


23: 24: 25:

procedure NodeIniti (mi , Ci ) { Ci = RandomSelect(Di , mi ); acti =|Di |; store (acti , Ci ); }

26: 27: 28: 29: 30: 31: 32: 33: 34: 35: 36: 37: 38: 39: 40: 41: 42: 43: 44: 45: 46: 47: 48: 49: 50: 51: 52: 53: 54: 55: 56:

procedure NodeCompi (minOU T, C, act, LN N Ci , LCi , acti ) { load (acti , Ci ); D = D\Ci ; acti = acti − |Ci |; i init(LCi ,m act ); act for each (p in Ci ) LN N Ci [p]=N N [p]; for each (pj in Ci = {p1 , . . . , p|Ci | }) for each (q in {pj , . . . , p|Ci | }) { δ = dist(pj , q); UpdateMin(LN N Ci [pj ], q, δ ); if (pj = q) UpdateMin(LN N Ci [q], pj , δ ); } for each (p in Ci ) for each (q in (C\Ci ) { δ = dist(p, q); UpdateMin(LN N Ci [p], q, δ ); } for each (p in Di ) { for each (q in C) if (max{Sum(N Ni [p]), Sum(LN N Ci [q])} ≥ minOU T ) { δ = dist(p, q); UpdateMin(N Ni [p], q, δ ); UpdateMin(LN N Ci [q], p, δ ); } if (Sum(N Ni [p]) < minOU T ) acti = acti − 1; else UpdateMax(LCi , p, Sum(N Ni [p]) ); } Ci = objects(LCi ); store (acti , Ci ); }

335

Fig. 2. The procedures employed in the DistributedSolvingSet algorithm

32-54). Thus, the first double cycle compares the objects of Ci with themselves and updates the associated heaps (lines 32-37). The second double cycle compares the objects of Ci with the other objects of C (lines 38-42). Finally, the third double cycle compares the objects of Di with the objects of C (lines 43-54). In particular, a pair of objects (p, q), with p ∈ Di and q ∈ C, is compared only if at least one of the two objects could be an outlier (that is, if the maximum between

336

F. Angiulli et al.

their weight upper bounds Sum(N Ni [p]) and Sum(LN N Ci [q]) is greater than the lower bound minOU T ). During the last double cycle, if the weight upper bound of the local object p becomes smaller than minOU T , the number acti of local active objects is decremented (lines 50-51). Otherwise, the heap LCi is updated with p (lines 52-53). Finally, Ci stores the objects in LCi (line 55) and both acti and Ci are saved in the local memory (line 56). We now trace the analysis of the transferred data during the execution of the algorithm. Communication analysis. The communication among the supervisor node and the local nodes is carried out by the procedures NodeInit and NodeComp. NodeInit is launched for each local node only one time and requires that one integer value (that is the first operand) is sent to local nodes and mi data objects are transferred from the local node i to the supervisor one. NodeComp is executed for each local node one time for each iteration of the basic cycle of the DistributedSolvingSet ; at each execution, it requires that one float number, m data objects, and one integer value (that is, the first three operands respectively) are sent from the supervisor node to the local ones and that mk distances, mi data objects and one integer value (that is, the remaining operands respectively) are returned from each local node to the coordinator one. Let us call d the number of attributes of a data object and niter the number of iterations, then the total amount of transferred data in terms of quantity of float numbers (for the sake of simplicity, we assume that integer numbers have the same size as the float ones) is T D = li=1 (1 + mi d) + niter (1 + md + 1 + li=1 (mk + mi d + 1)) l Given that i=1 mi = m and niter m = |DSS|, then T D = l + md + 2niter + |DSS|d + |DSS|lk + |DSS|d + niter l We note that m |DSS|, then dm |DSS|d and also that all the terms l, 2niter , and 2niter l are negligible. Therefore, we can approximate T D ≈ |DSS|(lk + 2d). Now we relate the amount T D of trasferred data to the size |D|d of the data set. Let |DSS| = ρ|D|, then T D% =

ρlk TD ≈ . |D|d d

Thus, T D% is directly proportional to relative size ρ = |DSS| of the dis|D| tributed solving set. To this aim, we notice that the value of ρ observed in the experiments is of the order of 0.01. Importantly, for n and k fixed, ρ decreases more than linearly as the size |D| increases, since the size |DSS| of DSS tends to stabilize [1]. From the analysis above, we can derive the following considerations: – even if the amount of transferred data is great (for large values of lk), the computational gain due to computation distribution remains remarkable;


337

– the data objects transmitted among the nodes are only those that belong to the DSS and they represent a small percentage of the data objects in D; – most of the transmitted data are not data points, but distances, this accounts for the inverse proportionality of the percentage of transmitted data w.r.t. the dimensionality of the data set; – more sophisticated strategies could lead to the transmission of a reduced number of distances for each node, say kd , therefore substituting the term lk with something less, such that lkd is O(k), thus removing the dependence of the amount of data transmitted from l; this will be tackled in further studies.

3

Experiments

To check the effectiveness of the proposed approach, we evaluated the performance of the algorithm through several experiments over large data sets. In order to guarantee a great level of generality, the algorithm is written in Java and supports communication through the Java libraries implementing the TCP sockets. We used a network of 11 workstations, each equipped with a Intel 2.26 GHz processor and 4GB of RAM. In the experiments reported in the following, the default values for the parameters are n = 10, k = 50, and m set to the best performance of the centralized algorithm. We note that we considered also other combinations of values for the parameters n and k, that we do not report due to space limitations, and that the method exhibited always a behavior similar to that showed in the following. We considered the following data sets: – G3d is a synthetic data set containing 500,000 3D vectors of reals, obtained by the union of three subsets each generated from a normal distribution having the same standard deviation and a different mean; – Covtype is a projection on the quantitative attributes of the real data set Covertype available at the Machine Learning Repository at the University of California, Irvine [4]; it consists of 581,012 instances and 10 attributes; – G2d is a synthetic data set representing a collection of 1,000,000 points of the plane generated from a normal distribution having unit standard deviation; – Poker is obtained form the real data set PokerHands, available at UCI repository, by removing the class label; then P oker consists of 1,000,000 instaces and 10 attributes. Now we present the results of the experiments. We note that when the number of the nodes p is set to 1, then a centralized version of the DistributedSolvingSet algorithm, which is equivalent to the SolvingSet algorithm, is executed. In the experiments we took into account also a modified version of the G2d data set, called G2d in the following. The G2d data set contains the same data as G2d. However, in the G2d data set the partitioning of the points on the local nodes is intentionally biased, since all the data set outliers have been allocated to a unique local node. The G2d data set is representative of a worst case scenario for the distributed outlier detection task. Thus, by testing the

F. Angiulli et al. (a)

(b) Communication time / Execution time

10 9 8 Speedup

7

G3d Covtype Poker G2d G2d‘

6 5 4 3 2 1

2

4 6 Number of nodes

8

10

(c)

0.06

0.03 Supervisor time / Execution time

338

0.05 0.04


0.03 0.02 0.01 0 0

2

4 6 Number of nodes

8

10

0.025 0.02


0.015 0.01 0.005 0 0

2

4 6 Number of nodes

8

10

Fig. 3. (a) Speedup (b) Communication time (c) Supervisor time

algorithm DistributedSolvingSet on it, we have the opportunity to check whether it is influenced by an unfavourable distribution of the outliers, or not. Figure 3(a) shows the speedup Sp (T1 /Tp ) obtained by using the DistributedSolvingSet algorithm. Importantly, for all the considered data sets, the algorithm scaled very well, exhibiting a speedup close to linear. These good performances can be explained by analyzing the communication time and the supervisor node execution time. As far as the communication time is concerned, the time spent to transfer data from the local nodes to the supervisor node during the computation is always a small portion of the whole execution time, as reported in Figure 3(b). As for the supervisor node execution time, also the time spent by the supervisor node to put together the partial results returned by the various nodes is small with respect the overall execution time, as reported in Figure 3(c). It can be concluded that the overall execution time is mainly determined by the time spent to execute the core of the computation, which is represented by the procedure NodeComp. As described in Section 2, the dominating operation of the procedure NodeComp is the computation of the distance between two data points. By observing Figure 4(a), it can be noted that the total number of distances computed is sligthly increasing with the number of nodes. Since the total number of distances computed is little sensitive to the number of nodes, then the average number of distances per node scales near linearly. If the load is balanced, that is, if the average number of distances per node corresponds to the actual number of distances computed by each single node, then, from the analysis of the communication and supervisor node time, the speedup of the algorithm should exhibit near linear behavior. It must be noted that each single iteration terminates only after every node has completed its computation. Hence, the actual number of distances per node is related to the maximum node load, that is to the maximum number of distances to be computed by a single local node (this number is related to the number of active objects in the local node). Thus, in order to quantify the deviation from the ideal behavior, we employed the equivalent distances measure, defined as i maxj {di,j }, where j represents a generic iteration of the algorithm, i represents a generic local node, and di,j is the number of distances computed by node j during the ith iteration.



5

2

4 6 Number of nodes

8

10

1.035

500

1.03

Number of iterations

10

0 0

600

1.04 Relative equivalent distances

Computed distances

x 10

(c)

(b)

(a)

9

15


1.025 1.02 1.015 1.01

400


300 200 100

1.005 1 0

339

2

4 6 Number of nodes

8

10

0 0

2

4 6 Number of nodes

8

10

Fig. 4. (a) Computed distances (b) Relative equivalent distances (c) No. of Iterations

Figure 4(b), comparing the ideal behavior (that is, the average number of distances per node) with the actual one (that is, the equivalent distances measure), shows that during the execution of the algorithm the load is indeed almost balanced (the relative equivalent distances is the ratio between the equivalent distances and the average number of distances per node). Interestingly, the analysis on the G2d dataset reveals that a biased distribution of the outliers does not affect the load distribution. Finally, we study the number of iterations of the algorithm versus the number of local nodes. Figure 4(c) shows that the number of iterations necessary to reach the algorithm termination is practically the same one as the sequential case that corresponds to the execution for p = 1. Since at each iteration at most m points are added to the solving set, the size of the distributed solving set does not vary with the number of local nodes. It can be concluded that the solving set computed by the parallel algorithm are of the same quality of that computed by the sequential algorithm, therefore it can be usefully exploited for the outlier prediction task.

4

Conclusions and Future Work

We presented the DistributedSolvingSet algorithm, a distributed method for computing an outlier detection solving set and the top-n distance-based outliers according to definitions given in [3,1]. We proved that the original centralized algorithm can be adapted to work in distributed environments and that the proposed solution (i) produces an overall speed-up close to linear w.r.t. the number of computing nodes and (ii) scales well for increasing number of nodes w.r.t. both the computation in the coordinator node and the data transmission. For this reason we claim that the solution is suitable when data reside on distributed nodes and also when data are distributed for load balance reasons. As future work, we are planning (i) to further reduce the communication overhead by optimizing the amount of information sent in order to determine the true nearest neighbors of the current candidate objects, (ii) to conduct a more extensive experimental campaign, which will take into account also other parallel/distributed scenarios, and (iii) to design an alternative approximate

340

F. Angiulli et al.

strategy based on computing local solutions at each node and then obtaining the global one by a suitable merge operator. Acknowledgements. We would like to thank D. Tiraferri for his work on the implementation of the DistributedSolvingSet algorithm.

References 1. Angiulli, F., Basta, S., Pizzuti, C.: Distance-based detection and prediction of outliers. TKDE 18(2), 145–160 (2006) 2. Angiulli, F., Fassetti, F.: Dolphin: An efficient algorithm for mining distance-based outliers in very large datasets. TKDD 3(1) (2009) 3. Angiulli, F., Pizzuti, C.: Outlier mining in large high-dimensional data sets. TKDE 2(17), 203–215 (2005) 4. Asuncion, A., Newman, D.: UCI machine learning repository (2007) 5. Bay, S.D., Schwabacher, M.: Mining distance-based outliers in near linear time with randomization and a simple pruning rule. In: Proc. KDD (2003) 6. Ghoting, A., Parthasarathy, S., Otey, M.E.: Fast mining of distance-based outliers in high-dimensional datasets. Data Min. Knowl. Discov. 16(3), 349–364 (2008) 7. Han, J., Kamber, M.: Data Mining, Concepts and Technique. Morgan Kaufmann, San Francisco (2001) 8. Hung, E., Cheung, D.W.-L.: Parallel mining of outliers in large database. Distributed and Parallel Databases 12(1), 5–26 (2002) 9. Knorr, E., Ng, R.: Algorithms for mining distance-based outliers in large datasets. In: Proc. Int. Conf. on Very Large Databases (VLDB 1998), pp. 392–403 (1998) 10. Koufakou, A., Georgiopoulos, M.: A fast outlier detection strategy for distributed high-dimensional data sets with mixed attributes. Data Mining and Knowledge Discovery (November 11, 2009) (published online) 11. Lozano, E., Acu˜ na, E.: Parallel algorithms for distance-based and density-based outliers. In: ICDM, pp. 729–732 (2005) 12. Otey, M.E., Ghoting, A., Parthasarathy, S.: Fast distributed outlier detection in mixed-attribute data sets. Data Min. Knowl. Discov. 12(2-3), 203–228 (2006) 13. Ramaswamy, S., Rastogi, R., Shim, K.: Efficient algorithms for mining outliers from large data sets. In: Proc. Int. Conf. on Managment of Data (SIGMOD 2000), pp. 427–438 (2000) 14. Tao, Y., Xiao, X., Zhou, S.: Mining distance-based outliers from large databases in any metric space. In: KDD, pp. 394–403 (2006)

Grid, Cluster and Cloud Computing K. Keahey1 , D. Laforenza1 , A. Reinefeld2 , P. Ritrovato2 , D. Thain2 , and N. Wilkins-Diehr2 1


2

Grid computing is a major research area with strong involvement from both academia and the computing industry. The common vision is that grid computing represents the culmination of truly general distributed computing across various resources in a ubiquitous, open-ended infrastructure to support a wide range of different application areas. Although significant progress has been made in the design and deployment of grids, many challenges still remain before the goal of a user-friendly, efficient, and reliable grid can be realized. Grid research issues cover many areas of computer science to address the fundamental capabilities and services that are required in a heterogeneous environment, such as adaptability, scalability, reliability and security, and to support applications as diverse as ubiquitous local services, enterprise-scale virtual organizations, and internet-scale distributed supercomputing. Cloud computing is also emerging as an alternate platform for large-scale distributed applications where resources are typically provided by a single administrative domain in a pay per-use mode. To some, cloud computing is a natural evolution of grid computing, to others, it is a complementary and perhaps competing technology. Grid and cloud research will greatly benefit from interactions with the many related areas of computer science, making Euro-Par an excellent venue to present results and discuss issues. This year, 29 papers discussing some those issues were submitted to this topic. Each paper was reviewed by at least three reviewers and, finally, we were able to select 8 regular papers (28% acceptance rate). The accepted papers discuss interesting challenges arising in grid and cloud computing. In particular, in the paper ”What Is the Price of Simplicity? A Cross-platform Evaluation of the SAGA API” M. den Burger et al. describe the effects on expressiveness and ease of programming using abstractions defined by the SAGA API (Simple API for Grid Applications) and shows what price is paid for the abstraction in terms of performance. Two papers focus on examining the distribution challenge in grids and clouds. ”Using Network Information to Perform Meta-scheduling in Advance in Grids” by L. Toms et al. discusses the challenges of providing quality of service (QoS) in the grid environment and proposes a system that provides it through networkaware job scheduling in advance while “Deployment of a hierarchical middleware” by E. Caron et al. proposes a hierarchical model for gird middleware and an algorithm for automatically and efficiently deploying it is a distributed environment taking into account throughput ratios. P. D’Ambra, M. Guarracino, and D. Talia (Eds.): Euro-Par 2010, Part I, LNCS 6271, pp. 341–342, 2010. c Springer-Verlag Berlin Heidelberg 2010

342

K. Keahey et al.

Investigating the properties of workflow systems continues to be a significant challenge in the grids. In ”Toward Real-time, Many-Task Applications on Large Distributed Systems”, S. Yi et al. design and implement a real-time task management system for many-task computing - the system builds on the BOINC platform but provides impressive O(1) worst-case execution time for task management operations. ”Scheduling Scientific Workflows to Meet Soft Deadlines in the Absence of Failure Models” by K. Plankensteiner et al., proposes a dynamic execution and scheduling heuristic able to schedule workflow applications with a high degree of fault tolerance, while taking into account soft deadlines. And finally ”User-centric, Heuristic Optimization of Service Composition in Clouds” by K. Kofler at al. proposes a system to map costumer requirements onto functional and non-functional attributes of the services and evaluates the resulting system in the context of the Kepler workflow tool. Challenges arising in cloud computing have been investigated all throughout the programming stack. The paper ”A GPGPU transparent virtualization component for high performance computing clouds” by G. Giunta et al. focuses on the minutiae of virtualization and explains how a virtual machine can access GPGPUs in a transparent way, with an overhead only slightly greater than a real machine/GPGPU setup whereas the paper ”A Distributed Market Framework for Large-scale Resource Sharing” by M. Mihailescu et al. addresses things on a higher level and presents scalable distributed market framework for the allocation of shared resources in large distributed systems. We would like to take the opportunity of thanking all authors who shared their results with EuroPar community as well as the Euro-Par Organizing Committee, and the referees for their hard work and insightful comments, whose efforts have made this conference and this topic possible.

Deployment of a Hierarchical Middleware Eddy Caron, Benjamin Depardon, and Frédéric Desprez University of Lyon, LIP Laboratory, UMR CNRS - ENS Lyon INRIA - UCBL 5668, France {Eddy.Caron,Benjamin.Depardon,Frederic.Desprez}@ens-lyon.fr

Abstract. Accessing the power of distributed resources can nowadays easily be done using a middleware based on a client/server approach. Several architectures exist for those middleware. The most scalable ones rely on a hierarchical design. Determining the best shape for the hierarchy, the one giving the best throughput of services, is not an easy task. We first propose a computation and communication model for such hierarchical middleware. Our model takes into account the deployment of several services in the hierarchy. Then, based on this model, we propose an algorithm for automatically constructing a hierarchy. This algorithm aims at offering the users the best obtained to requested throughput ratio, while providing fairness on this ratio for the different kind of services, and using as few resources as possible. Finally, we compare our model with experimental results on a real middleware called Diet. Keywords: Hierarchical middleware, Deployment, Modelization.

1

Introduction

Using distributed resources to solve large problems ranging from numerical simulations to life science is nowadays a common practice. Several approaches exist for porting these applications to a distributed environment; examples include classic message-passing, batch processing, web portals and GridRPC systems. In this last approach, clients submit computation requests to a meta-scheduler (also called agent) that is in charge of finding suitable servers for executing the requests within the distributed resources. Scheduling is applied to balance the work among the servers. A list of available servers is sent back to the client; which is then able to send the data and the request to one of the suggested servers to solve its problem. There exists several grid middleware to tackle the problem of finding services available on distributed resources, choosing a suitable server, then executing the requests, and managing the data. Several environments, called Network Enabled Servers (NES) environments, have been proposed. Most of them share a common characteristic which is that they are built with broadly three main components: clients which are applications that use the NES infrastructure, agents which are in charge of handling the clients’ requests (scheduling them) and of finding suitable servers, and finally computational servers which provide computational P. D’Ambra, M. Guarracino, and D. Talia (Eds.): Euro-Par 2010, Part I, LNCS 6271, pp. 343–354, 2010. c Springer-Verlag Berlin Heidelberg 2010

344

E. Caron, B. Depardon, and F. Desprez

power to solve the requests. Some of the middleware only rely on basic hierarchies of elements, a star graph, such as Ninf-G [1] and NetSolve [2]. Others, in order to divide the load at the agents level, can have a more complicated hierarchy shape: WebCom-G [3] and Diet [4,5]. In this latter case, a problem arises: what is the best shape for the hierarchy? In this paper, we will mainly focus on one particular hierarchical NES: Diet (Distributed Interactive Engineering Toolbox). The Diet component architecture is structured hierarchically as a tree to obtain an improved scalability. Such an architecture is flexible and can be adapted to diverse environments, including arbitrary heterogeneous computing platforms. Diet comprises several components. Clients that use Diet infrastructure to solve problems using a remote procedure call (RPC) approach. SeDs, or server daemons, act as service providers, exporting functionalities via a standardized computational service interface; a single SeD can offer any number of computational services. Finally, agents facilitate the service location and invocation interactions of clients and SeDs. Collectively, a hierarchy of agents provides higher-level services such as scheduling and data management. These services are made scalable by distributing them across a hierarchy of agents composed of a single Master Agent (MA) (the root of the hierarchy) and several Local Agents (LA) (internal nodes). Deploying applications on a distributed environment is a problem that has already been addressed. We can find in the literature a few deployment software: DeployWare [6], Adage [7], TUNe [8], and GoDiet [9]. Their field of action ranges from single deployment to autonomic management of applications. However, none include intelligent deployment mapping algorithms. Either the mapping has to be done by the user, or the proposed algorithm is random or round-robin. Some algorithms have been proposed in [10,11] to deploy a hierarchy of schedulers on clusters and grid environments. However, a severe limitation in these works is that only one kind of service could be deployed in the hierarchy. Such a constraint is of course not desirable, as nowadays many applications rely on workflows of services. Hence, the need to extend the previous models and algorithms to cope with hierarchies supporting several services. The contribution of this paper is twofold. We first present a model for predicting the performance of a hierarchical NES on a homogeneous platform. As we will see this model can easily be applied to a computation heterogeneous platform. Secondly, we present an algorithm for automatically determining the best shape for the hierarchy, i.e., the number of servers for each services, and the shape of the hierarchy supporting these servers. We first present in Section 2 the hypotheses for our model, then the model itself in Section 3 for both agents and servers. Then, we explain our approach to automatically build a suitable hierarchy in Section 4, and finally compare the theoretical results with experimental results in Section 5, before concluding.

2

Model Assumptions

Request definition. Clients use a 2-phases process to interact with a deployed hierarchy: they submit a scheduling request to the agents to find a suitable

Deployment of a Hierarchical Middleware

345

server in the hierarchy (the scheduling phase), and then submit a service request (job) directly to the server (the service phase). A completed request is one that has completed both the scheduling and service request phases and for which a response has been returned to the client. We consider that a set R of services have to be available in the hierarchy. And that for each service i ∈ R, the clients aim at attaining a throughput ρ∗i of completed requests per seconds. Resource architecture. In this paper we will focus on the simple case of deploying the middleware on a fully homogeneous, fully connected platform G = (V, E, w, B), i.e., all nodes’ processing power are the same: w in M f lops/s, and all links have the same bandwidth: B in M b/s. We do not take into account contentions in the network. Deployment assumptions. We consider that at the time of deployment we do not know the clients locations or the characteristics of the clients’ resources. Thus, clients are not considered in the deployment process and, in particular, we assume that the set of computational resources used by clients is disjoint from V . A valid deployment will always include at least the root-level agent and one server per service i ∈ R. Each node v ∈ V can be assigned either as a server for any kind of service i ∈ R, or as an agent, or left idle. Thus with |A| agents, |S| servers, and |V | total resources, |A| + |S| ≤ |V |. Objective. As we have multiple services in the hierarchy, our goal cannot be to maximize the global throughput of completed requests regardless of the kind of services, this would lead to favor services requiring only small amount of computing power for scheduling and solving phases, and with few communications. Hence, our goal is to obtain for each service i ∈ R a throughput ρi such that all services receive almost the same obtained throughput to requested throughput ratio: ρρ∗i , while having as few agents in the hierarchy as possible, so as not to i use more resources than necessary.

3 3.1

Hierarchy Model Overall Throughput

For each service i ∈ R, we define ρschedi to be the scheduling throughput for requests of type i offered by the platform, i.e., the rate at which requests of type i are processed by the scheduling phase. We define as well ρservi to be the service throughput. Lemma 1. The completed request throughput ρi of type i of a deployment is given by the minimum of the scheduling and the service request throughput ρschedi and ρservi . ρi = min {ρschedi , ρservi } Lemma 2. The service request throughput ρservi for service i increases as the number of servers included in a deployment and allocated to service i increases.

346

3.2


Hierarchy Elements Model

We now precise the model of each element of the hierarchy. We consider that a request of type i is sent down a branch of the hierarchy, if and only if service i is present in this branch, i.e., if at least a server of type i is present in this branch of the hierarchy. Thus a server of type i will never receive a request of type j = i. Agents will not receive a request i if no server of type i is present in its underlying hierarchy, nor will it receive any reply for such a type of request. This is the model used by Diet. Server model. We define the following variables for the servers. wprei is the amount of computation in M F lops needed by a server of type i to predict its own performance when it receives a request of type i from its parent. Note that a server of type i will never have to predict its performance for a request of type j = i as it will never receive such a request. wappi is the amount of computation in M F lops needed by a server to execute a service. mreqi is the size in M b of the messages forwarded down the agent hierarchy for a scheduling request, and mrespi the size of the messages replied by the servers and sent back up the hierarchy. Since we assume that only the best server is selected at each level of the hierarchy, the size of the reply messages does not change as they move up the tree. Server computation model. Let’s consider that we have ni servers of type i, and that ni requests of type i are sent. On the whole, the ni servers of type i require ni .wprei +wappi time unit to serve the ni requests: each server has to compute the w performance prediction ni times, and serve one request. Hence, on average, the time to compute one request of type i is given by Equation (1). server = Tcomp i

wprei +

wappi ni

(1)

w

Server communication model. A server of type i needs, for each request, to receive the request, and then to reply. Hence Equations (2) and (3) represent respectively the time to receive one request of type i, and the time to send the reply to its parent. server Trecv = i

mreqi B

(2)

server Tsend = i

mrespi B

(3)

Service throughput. Concerning the machines model, and their ability to compute and communicate, we consider the following models: – Send or receive or compute, single port: a node cannot do anything simultaneously. 1 ρservi = server (4) server server Trecvi + Tsendi + Tcomp i


347

– Send or receive, and compute, single port: a node can simultaneously send or receive a message, and compute. 1 1 ρservi = min , (5) server + T server T server Trecv compi sendi i – Send, receive, and compute, single port: a node can simultaneously send and receive a message, and compute. 1 1 1 ρservi = min , , (6) server T server T server Trecv compi sendi i Agent model. We define the following variables for the agents. wreqi is the amount of computation in M F lops needed by an agent to process an incoming request of type i. For a given agent Aj ∈ A, let Chldji be the set of children of Aj having service i in their underlying hierarchy. Also, let δij be a Boolean variable equal to 1 if and only if Ajhas at least one child which knows service j i in its underlying hierarchy. wrespi Chldi is the amount of computation in M F lops needed to merge the replies of type i from its Chldji children. This amount grows linearly with the number of children. Our agent model relies on the underlying servers throughput. Hence, in order to compute the computation and communication times taken by an agent Aj , we need to know both the servers throughput ρservi for each i ∈ R, and the children of Aj . Agent computation model. The time for an agent Aj to schedule a request it receives and forwards is given by Equation (7). ρservi .δij .wreqi + i∈R ρservi .wrespi Chldji i∈R agentj (7) Tcomp = w Agent communication model. Agent Aj needs, for each request of type i, to receive the request and forwards it to the relevant children, then to receive the replies and forward the aggregated result back up to its parent. Hence Equations (8) and (9) present the time to receive and send all messages when the servers provide a throughput ρservi for each i ∈ R. j j ρ .δ .m + ρ . Chld serv req serv i i i i i .mrespi i∈R i∈R agentj Trecv = (8) B j j ρ ρ serv serv i .δi .mrespi + i . Chldi .mreqi i∈R i∈R agent Tsend j = (9) B We combine (7), (8), and (9) according to the chosen communication / computation model (similar to Equations (4), (5), and (6)).

348


Lemma 3. The highest throughput a hierarchy of agents is able to serve is limited by the throughput an agent having only one child of each kind of service can support.

4

Automatic Planning

Given the models presented in the previous section, we propose a heuristic for automatic deployment planning. The heuristic comprises two phases. The first step consists in dividing N nodes between the services, so as to support the servers. The second step consists in trying to build a hierarchy, with the |V | − N remaining nodes, which is able to support the throughput generated by the servers. In this section, we present our automatic planning algorithm in three parts. In Section 4.1 we present how the servers are allocated nodes, then in Section 4.2 we present a bottom-up approach to build a hierarchy of agents, and finally in Section 4.3 we present the whole algorithm. 4.1

Servers’ Repartition ρ

i Our goal is to obtain for all services i ∈ R the same ratio serv . As we try to ρ∗ i build a hierarchy of agents being able to support the throughput offered by the servers, in the end we will have ρi = ρservi . Algorithm 1 presents a simple way of dividing the available nodes to the different services. We iteratively increase the number of assigned nodes per services, starting by giving nodes to the service ρ i with the lowest serv ratio. ρ∗ i

Algorithm 1. Servers’ repartition Require: N : number of available nodes Ensure: n: number of nodes allocated to the servers 1: S ← list of services in R 2: n ← 0 3: repeat 4: i ← first service in S 5: Assign one more node to i, and compute the new ρservi 6: n ← n+1 7: if ρservi ≥ ρ∗i then 8: ρservi ← ρ∗i 9: S ← S − {i} ρ i 10: S ← Sort services by increasing serv ρ∗ i 11: until n = N or S = ∅ 12: return n

4.2

Agents’ Hierarchy

Given the servers’ repartition, and thus, the services’ throughput ρservi , for all i ∈ R, we need to build a hierarchy of agents that is able to support the throughput offered by the servers. Our approach is based on a bottom-up construction:


349

we first distribute some nodes to the servers, then with the remaining nodes we iteratively build levels of agents. Each level of agents has to be able to support the load incurred by the underlying level. The construction stops when only one agent is enough to support all the children of the previous level. In order to build each level, we make use of an integer linear program (ILP): (L1 ). We first need to define a few more variables. Let k be the current level: k = 0 corresponds to the server level. For i ∈ R let ni (k) be the number of elements (servers or agents) obtained at step k, which know service i. For k ≥ 1, we recursively define new sets of agents. k−1We define by Mk the number of available resources at step k: Mk = M1 − l=1 ni (l). For 1 ≤ j ≤ Mk we define aj (k) ∈ {0, 1} to be a Boolean variable stating whether or not node j is an agent in step k. aj (k) = 1 if and only if node j is an agent in step k. For 1 ≤ j ≤ Mk , ∀i ∈ R, δij (k) ∈ {0, 1} defines whether of not node j has service i in its underlying hierarchy in step k. For the servers, k = 0, 1 ≤ j ≤ M0 , ∀i ∈ R, δij (0) = 1 if and only if server j is of type i, otherwise δij (0) = 0. Hence, we have the following Mk j relation: ∀i ∈ R, ni (k) = j=1 δi (k). For 1 ≤ j ≤ Mk , ∀i ∈ R, Chldji (k) ∈ N is as previously the number of children of node j that know service i. Finally, for 1 ≤ j ≤ Mk , 1 ≤ l ≤ Mk−1 let cjl (k) ∈ {0, 1} be a Boolean variable stating that node l in step k − 1 is a child of node j in step k. cjl (k) = 1 if and only if node l in step k − 1 is a child of node j in step k. Using integer program (L1 ), we can recursively define the hierarchy of agents, starting from the bottom of the hierarchy. Let’s have a closer look at (L1 ). Lines (1), (2) and (3) only define the variables. Line (4) states that any element in level k − 1 has to have exactly 1 parent in level k. Line (5) counts, for each element at level k, its number of children that know service i. Line (6) states that the number of children of j of type i cannot be greater than the number of elements in level k − 1 that know service i, and has to be 0 if δij (k) = 0. The following two lines, (7) and (8), enforce the state of node j: if a node has at least a child, then it has to be an agent (line (7) enforces aj (k) = 1 in this case), and conversely, if it has no child, then it has to be unused (line (8) enforces aj (k) = 0 in this case). Line (9) states that at least one agent has to be present in the hierarchy. Line (10) is the transposition of the agent model in the send or receive or compute, single port model. Note that the other models can easily replace this model in ILP (L1 ). This line states that the time required to deal with all requests going through an agent has to be lower than or equal to one second. Finally, our objective function is the minimization of the number of agents: the equal share of obtained throughput to requested throughput ratio has already been cared of when allocating the nodes to the servers, hence our second objective that is the minimization of the number of agents in the hierarchy has to be taken into account. Remark 1. In order to improve the converge time to an optimal solution for integer program (L1 ), we can add the following constraint: a1 (k) ≥ a2 (k) · · · ≥ aMk (k)

(10)

350


Minimize

Mk

aj (k)

j=1

Subject to ⎧ (1) 1 ≤ j ≤ Mk ⎪ ⎪ ⎪ ⎪ ⎪ (2) 1 ≤ j ≤ Mk , ∀i ∈ R ⎪ ⎪ ⎪ ⎪ ⎪ (3) 1 ≤ j ≤ Mk , ⎪ ⎪ ⎪ ⎪ 1 ≤ l ≤ Mk−1 ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ (4) 1 ≤ l ≤ Mk−1 ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ (5) 1 ≤ j ≤ Mk , ∀i ∈ R ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ (6) 1 ≤ j ≤ Mk , ∀i ∈ R ⎪ ⎪ ⎪ ⎨ (7) 1 ≤ j ≤ Mk , i ∈ R ⎪ (8) 1 ≤ j ≤ Mk ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ (9) ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ (10) 1 ≤ j ≤ Mk ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎩

aj (k) ∈ {0, 1}

δij (k) ∈ {0, 1} and Chldji (k) ∈ N cjl (k) ∈ {0, 1} Mk cjl (k) = 1 j=1

k−1 M j cjl (k).δil (k − 1) Chldi (k) = l=1 j j Chldi (k) ≤ δi (k).ni (k − 1)

(L1 )

δij (k)

≤ a j (k) aj (k) ≤ δij (k) Mk

i∈R

aj (k) ≥ 1

j=1

ρservi ×

i∈R δij (k).wreqi + wrespi Chldji (k) + w δij (k).mreqi + Chldji (k) .mrespi + B j j δi (k).mrespi + Chldi (k) .mreqi ≤1 B

This states that only the first nodes can be agents. This prevents the solver from trying all swapping possibilities when searching a solution. We can safely add this constraint, as we suppose that we have a homogeneous platform. 4.3

Building the Whole Hierarchy

So far, we did not talk about the repartition of the available nodes between agents and servers. We will now present the whole algorithm for building the hierarchy. Maximum attainable throughput per service. Whatever the expected throughput for each service is, there is a limit on the maximum attainable throughput. Given Equations (7), (8) and (9), and the fact that a hierarchy must end at the very top by only one agent, the maximum throughput attainable by an agent serving all kinds of services (which is the case of the root of the hierarchy), is attained when the agent has only one child of each service (see Lemma 3). Hence, the maximum attainable throughput for each service, when all services receive the same served to required throughput ratio, from the agents’ point of view is given by integer program (L2 ) which computes ρmax servi for i ∈ R, the maximum attainable throughput for each service i that an agent can offer under the assumption that all services receive an equal share.


Maximize μ Subject to ⎧ ⎪ ⎪ (1) ∀i ∈ R ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎨ (2) ∀i, i ∈ R ⎪ ⎪ (3) 1 ≤ j ≤ Mk ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎩

ρmax servi ∗ and μ ∈ [0, 1], ρmax servi ∈ [0, ρi ] ρ∗i ρmax ρmax servi servi = ∗ ρ ρ∗i i max ρ i∈R servi × wreqi + wrespi 2.mreqi + 2.mrespi + ≤1 w B

351

μ≤

(L2 )

When building the hierarchy, there is no point in allocating nodes to a service i if ρservi gets higher than ρmax servi . Hence, whenever a service has a throughput max higher than ρmax servi , then we consider that its value is ρservi when building the hierarchy. Thus, lines 7 and 8 in Algorithm 1 become: ∗ max 7: if ρcomp then servi ≥ min ρi , ρservi 8: ρservi ← min ρ∗i , ρmax servi

Building the hierarchy. Algorithm 2 presents how to build a hierarchy, it proceeds as follows. We first try to give as many nodes as possible to the servers (line 4 to 7), and we try to build a hierarchy on top of those servers with the remaining nodes (line 8 to 24). Whenever building a hierarchy fails, we reduce the number of nodes available for the servers (line 24, note that we can use a binary search to reduce the complexity, instead of decreasing by one the number of available nodes). Hierarchy construction may fail for several reasons: no more nodes are available for the agents (line 10), (L1 ) has no solution (line 12), or only chains of agents have been built, i.e., each new agent has only one child (line 20). If a level contains agents with only one child, those nodes are set as available for the next level, as having chains of agents in a hierarchy is useless (line 23). Finally, either we return a hierarchy if we found one, or we return a hierarchy with only one child of each type i ∈ R, as this means that the limiting factor is the hierarchy of agents. Thus, only one server of each type of service is enough, and we cannot do better than having only one agent. Correcting the throughput. Once the hierarchy has been computed, we need to correct the throughput for services that were limited by the agents. Indeed, the throughput computed using (L2 ) may be too restrictive for some services. The values obtained implied that we had an equal ratio between obtained throughput over requested throughput for all services, which may not be the case if a service requiring lots of computation is deployed alongside a service requiring very few computation. Hence, once the hierarchy is created, we need to compute what is really the throughput that can be obtained for each service on the hierarchy. To do so, we simply use our agent model, with fixed values for ρservi for all i ∈ R such that the throughput of i is not limited by the agents, and we try to maximize the values of ρservi for all services that are limited by the agents.

352


Algorithm 2. Build hierarchy 1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: 15: 16: 17: 18: 19: 20: 21: 22: 23: 24: 25: 26: 27: 28:

N ← |V | − 1 // One node for an agent, |V | − 1 for the servers Done ← false while N ≥ |R| and not Done do Use Algorithm 1 to find the server repartition with N nodes nbU sed ← number of nodes used by Algorithm 1 M0 ← nbU sed Set all variables: ni (0), aj (0), δij (0), Chldji (0) and cjl (0) k←1 Mk ← |V | − nbU sed while Mk > 0 and not Done do Compute level k using integer program (L1 ) if level k could not be built (i.e., (L1 ) failed) then break nbChains ← count number of agents having only 1 child availN odes ← Mk k Mk ← M j=1 aj (k) // Get the real number of agents if Mk == 1 then Done ← true // We attained the root of the hierarchy break if nbChains == Mk−1 then break// This means we added 1 agent over each element at level k − 1 k ←k+1 Mk ← availN odes − Mk−1 + nbChains N ← nbU sed − 1 if Done then return the hierarchy built with (L1 ) without chains of agents else return a star graph with one agent and one server of each type i ∈ R

Extending the model to heterogeneous machines. The model and the algorithms can easily be extended to support the case where each machine has a different computing power wj , but are still connected with the same bandwidth B. Indeed, we only need to replace w by wj in all the previous agents’ equations, replace w +|S |.w Equation (1) by appi i wjprei (with Si the set of servers of type i), and modify j∈Si

Algorithm 1 so as to take into account the power of the nodes (for example by sorting the nodes by increasing power) to be able to deal with heterogeneous machines interconnected with a homogeneous network. Note that in this model Remark 1 is no longer relevant.

5


In order to validate our model, we conducted experiments with Diet on the French experimental testbed Grid’5000 [12]. After a phase of benchmarking for the Diet elements, the services (dgemm [13] and computation of the Fibonacci


353

Table 1. Comparison between theoretical and experimental throughput (number of requests per second) No. nodes Services Theoretical 3 5 10 20

dgemm Fibonacci dgemm Fibonacci dgemm Fibonacci dgemm Fibonacci

273.0 238.3 534.8 470.1 1027.8 915.4 1699.1 1738.6

Experimental Mean Median Std. 278.0 278 242.5 242 543.2 544 476.1 477 984.9 995 912.9 922 1624.4 1666 1699.0 1735

Dev. 4.4 11.2 6.1 10.7 49.5 52.1 114.7 114.0

Relative Error 1.87% 1.76% 1.58% 1.26% 4.17% 0.26% 4.39% 2.28%

number using a naive algorithm) and the platform, we generated hierarchies for a number of nodes ranging from 3 to 50 (even though the algorithm is based on an ILP, it took only a few seconds to generate all the hierarchies). Our goal here is to stress Diet, so we use relatively small services: dgemm on 100 × 100 matrices and computing the Fibonacci number for n = 30. With more than 20 nodes the generated hierarchy was always the same as with 20 nodes. We ran as many clients as necessary to obtain the best throughput at the same time for all services. From 3 to 10 nodes, the hierarchy has only one agent, for 20 nodes it has 2 levels of agents, with 3 agents on the second level. Results are given in Table 1. As can be seen, the experimental results closely follow the theoretical predictions. Due to lack of space we do not present all the conducted experiments1 , more results can be found in our research report [14].

6

Conclusion

In this paper we presented a computation and communication model for hierarchical middleware, when several services are available in the middleware. We proposed an algorithm to find a hierarchy that gives the best obtained throughput to requested throughput ratio for all services. The algorithm uses a bottomup approach, and is based on an ILP to successively determine levels of the hierarchy. Our experiments on a real middleware, Diet, show that the obtained throughput closely follows what our model predicts. As future works, we intend to run experiments on larger platforms, with “bigger” services. Deployment on homogeneous machines is only the first step that allowed us to validate our model, we intend to extend our model and algorithms to fully heterogeneous platforms.

1

Experiments presented in this paper were carried out using the Grid’5000 experimental testbed, being developed under the INRIA ALADDIN development action with support from CNRS, RENATER and several Universities as well as other funding bodies (see https://www.grid5000.fr)

354


References 1. Tanaka, Y., Nakada, H., Sekiguchi, S., Suzumura, T., Matsuoka, S.: Ninf-g: A reference implementation of RPC-based programming middleware for grid computing. Journal of Grid Computing 1(1), 41–51 (2003) 2. Casanova, H., Dongarra, J.: Netsolve: a network server for solving computational science problems. In: Supercomputing 1996: Proceedings of the 1996 ACM/IEEE Conference on Supercomputing (CDROM), Washington, DC, USA, p. 40. IEEE Computer Society, Los Alamitos (1996) 3. Morrison, J.P., Clayton, B., Power, D.A., Patil, A.: Webcom-G: grid enabled metacomputing. Neural, Parallel Sci. Comput. 12(3), 419–438 (2004) 4. Caron, E., Desprez, F.: DIET: A scalable toolbox to build network enabled servers on the grid. International Journal of High Performance Computing Applications 20(3), 335–352 (2006) 5. Amar, A., Bolze, R., Caniou, Y., Caron, E., Depardon, B., Gay, J.S., Le Mahec, G., Loureiro, D.: Tunable scheduling in a GridRPC framework. Concurrency and Computation: Practice and Experience 20(9), 1051–1069 (2008) 6. Flissi, A., Merle, P.: A generic deployment framework for grid computing and distributed applications. In: Proceedings of the 2nd International OTM Symposium on Grid computing, high-performAnce and Distributed Applications (GADA 2006). LNCS, vol. 4279, pp. 1402–1411. Springer, Heidelberg (November 2006) 7. Lacour, S., Pérez, C., Priol, T.: Generic application description model: Toward automatic deployment of applications on computational grids. In: 6th IEEE/ACM International Workshop on Grid Computing (Grid 2005), Seattle, WA, USA. Springer, Heidelberg (November 2005) 8. Broto, L., Hagimont, D., Stolf, P., Depalma, N., Temate, S.: Autonomic management policy specification in tune. In: SAC 2008: Proceedings of the 2008 ACM Symposium on Applied Computing, pp. 1658–1663. ACM, New York (2008) 9. Caron, E., Chouhan, P.K., Dail, H.: GoDiet: A deployment tool for distributed middleware on grid’5000. In: IEEE (ed.) EXPGRID Workshop. Experimental Grid Testbeds for the Assessment of Large-Scale Distributed Apllications and Tools. In Conjunction with HPDC-15, Paris, France, June 19, pp. 1–8 (2006) 10. Caron, E., Chouhan, P., Legrand, A.: Automatic deployment for hierarchical network enabled servers. In: Proceedings of 18th International Parallel and Distributed Processing Symposium, p. 109 (April 2004) 11. Chouhan, P.K., Dail, H., Caron, E., Vivien, F.: Automatic middleware deployment planning on clusters. Int. J. High Perform. Comput. Appl. 20(4), 517–530 (2006) 12. Bolze, R., Cappello, F., Caron, E., Daydé, M., Desprez, F., Jeannot, E., Jégou, Y., Lanteri, S., Leduc, J., Melab, N., Mornet, G., Namyst, R., Primet, P., Quetier, B., Richard, O., Talbi, E.G., Irena, T.: Grid’5000: a large scale and highly reconfigurable experimental grid testbed. International Journal of High Performance Computing Applications 20(4), 481–494 (2006) 13. Dongarra, J., et al.: Basic linear algebra subprograms technical forum standard. International Journal of High Performance Applications and Supercomputing 16(1), 1–111 (2002) 14. Caron, E., Depardon, B., Desprez, F.: Modelization for the Deployment of a Hierarchical Middleware on a Homogeneous Platform. Technical report, Institut National de Recherche en Informatique et en Automatique, INRIA (2010)

Toward Real-Time, Many-Task Applications on Large Distributed Systems Sangho Yi1 , Derrick Kondo1 , and David P. Anderson2 1

INRIA, France {sangho.yi,derrick.kondo}@inrialpes.fr 2 UC Berkeley, USA [email protected]

Abstract. In the age of Grid, Cloud, volunteer computing, massively parallel applications are deployed over tens or hundreds of thousands of resources over short periods of times to complete immense computations. In this work, we consider the problem of deploying such applications with stringent real-time requirements. One major challenge is the server-side management of these tasks, which often number in tens or hundreds of thousands on a centralized server. In this work, we design and implement a real-time task management system for many-task computing, called RT-BOINC. The system gives low O(1) worst-case execution time for task management operations, such as task scheduling, state transitioning, and validation. We implement this system on top of BOINC, a common middleware for volunteer computing. Using micro and macro-benchmarks executed in emulation experiments, we show that RT-BOINC provides significantly lower worst-case execution time, and lessens the gap between the average and the worst-case performance compared with the original BOINC implementation.

1

Introduction

Workloads on Grid, Cloud, and Volunteer Computing platforms often consist of tens or hundreds of thousands of parallel tasks that must be processed in short amounts of time on the order of hours or days [1]. In this work, we focus on real-time applications of similar size where the deadline per task is on the order of seconds or tens of seconds. Important applications include online strategy games (such as Go [2] or Chess [3]), interactive visualization [4] (possibly with precedence constraints), and real-time digital forensics [5]. Our aim to enable the execution of real-time applications on large (on the order of 10,000 nodes) distributed systems, such as volunteer computing platforms. Volunteer computing platforms use the free resources in Intranet and Internet environments for large distributed computation, and currently provide over 8 PetaFLOPS of computing power for over 50 applications. However, these applications are limited to mainly high-throughput jobs or large batch jobs.

This paper was submitted to Euro-Par 2010.

P. D’Ambra, M. Guarracino, and D. Talia (Eds.): Euro-Par 2010, Part I, LNCS 6271, pp. 355–366, 2010. c Springer-Verlag Berlin Heidelberg 2010

356

S. Yi, D. Kondo, and D.P. Anderson

There are three main challenges for supporting real-time applications. First, one must ensure or predict the availability of volunteers. This has been the subject of recent work described in [6,7,8]. Second, one must bound network latency between the server and clients. Much work also exists on network distance that one could leverage [9]. Third, management of hundreds of thousands of tasks on the server must be efficient and in particular, have bounded execution time. In this work, we focus on the third challenge. Server-side management includes task generation, the transition of task and result states, and scheduling. These management functions can incur significant overheads, given that the desired job makespans are on the order of seconds and the number of tasks per job in on the order of tens of thousands. The main contribution of this work is the design and implementation of a real-time management system, based on a popular middleware for volunteer computing called BOINC. Our system RT-BOINC gives low worst-case bounds on server-side task management, while minimizing the gap between worst-case and average execution time. Our approach is to use novel data structures (in particular multi-level lookup tables) and functions that ensure O(1) worst-case complexity. In emulation experiments with our prototype, we show performance improvements of often 2 orders of magnitude compared to the original BOINC. The remainder of this paper is organized as follows. Section 2 describes related work in volunteer computing environments. Section 3 presents the design and internal structures, and implementation of RT-BOINC. Section 4 evaluates performance of RT-BOINC and the original BOINC in terms of both the average and worst-case execution time. Finally, Section 5 presents conclusions and future work.

2

Related Works

Volunteer computing systems, such as XtremWeb and Condor, are tailored for maximizing task throughput, not minimizing latency on the order of seconds. For instance, in [1], Silberstein et al. proposed GridBot, which provides efficient execution of bags-of-tasks on heterogeneous collections of computing platforms including grid, volunteer, and cluster computing environments virtualized as a single computing host. While the system uses a hierarchical task management system, the system cannot provide the task-level guarantees of execution time. Hierarchical systems can improve server performance but they still often do not provide any guarantee of performance in terms of worst-case execution time. For instance, in [10], Kacsuk et al. proposed a SZTAKI desktop grid, which is a hierarchical system developed on top of BOINC server structures. They modified the original BOINC server to have multiple levels of workunit distribution. By doing this, SZTAKI can reduce load on the primary BOINC server by using the second and third-level BOINC servers. But, each level of BOINC servers still has the same characteristics of the original BOINC, which performance is not guaranteed. Dedicated supercomputers can run real-time tasks, but volunteer computing could be an low-cost alternative if it could support real-time guarantees. In the domain of complex strategy games, Deep Blue [3] was the first machine defeat

Toward Real-Time, Many-Task Applications on Large Distributed Systems

357

the human world champion in 1996. IBM developed a dedicated server system for Deep Blue, and the server achieved about 11.38 GFLOPS on the LINPACK benchmark. Since 2006, several researchers in the world have been developed MoGo, which is software to find the next move in the game of Go. They adapted Monte-Carlo-based algorithms, and now, they are as strong as the professional Go players in the 9 × 9 small board based on the cluster computing machines[2]. Grid gaming middleware [11] have been developed and address issues such as adaptive redirection of communication or computation given variable load. They also address issues such as high-level easy-to-use programming interfaces, and monitoring tools for capacity planning. We believe our work on giving worst-case bounds on execution time is complementary with those methods. For example, our techniques guarantee performance given that the data can be store entirely in the server’s memory; this in turn could be used with capacity planning tools to determine when to replicate a server.

3

Design and Implementation of RT-BOINC

In this section, we briefly describe the internal structures of BOINC, and we present some requirements for computing real-time and interactive tasks in volunteer computing environments. Then, we present the design and implementation of RT-BOINC in detail. 3.1

The Original BOINC

BOINC server consists of two main parts, namely the main database, and server daemon processes (feeder, transitioner, assimilator, validator, file-deleter, workgenerator, and scheduler ). When the project-manager sends work for the BOINC server, the workgenerator creates several workunits on the main database. Then, transitioner makes workunit-results pairs on the database. The pairs are fed to the scheduler

Fig. 1. Internal structures of BOINC server

358


by the feeder, and they are distributed to multiple BOINC hosts. When each host returns its result, the scheduler reports it to the validator. When the validation is completed, the data will be processed and finalized by the assimilator. Finally, the project-manager can get the assimilated results from BOINC server. BOINC projects use the same structures described in Fig. 1. Each project should be aware of the performance bottlenecks in the BOINC projects if we need guaranteed performance. BOINC is geared towards large and long-term computation. The execution time of each workunit is relatively long enough, so that the BOINC server performs a relatively small amount of work distribution and reporting at the same time. Existing BOINC projects handle about 1 ∼ 10 workunits per second [12,13]. However, in the case of computing highly-interactive and short-term tasks with deadlines, the BOINC server must perform a relatively large number of transactions per period to guarantee the worst-case performance. In Fig. 1, most of the daemon processes read/write the main database. This means that the execution time of each daemon process depends on the number of records n in the database storing application, workunit, and result data, for example. MySQL in particular has O(log n) ∼ O(n2 ) time complexity[14]. In addition, we found that daemon processes have at least linear, and up to polynomial complexity1 . This makes it hard to provide relatively low worst-case execution time compared with the average execution time for all data-related operations and processes.

Fig. 2. An example of interactive, short-term, and real-time parallel tasks

3.2

Requirements for Interactive, Short-Term, and Real-Time Parallel Applications

We describe the real-time requirements of the online game of Chess. Figure 2 shows an example calculating the next move. In this example, the Chess player wants to determine the best move by using the volunteer computing environment. If we assume that the number of volunteer hosts is about 10,000, and each move should be calculated within 10 seconds, then the expected number of transactions between hosts and the server is 1,000 per second. This means that 1

Complexities of processes; feeder : O(nw ), assimilator : O(nw ), validator : O(nw · n2r ), transitioner : O(nw · nr ), file-deleter : O(nr ), work-generator : O(nw ), and scheduler : O(nr ) where nw is the number of workunits and nr is the number of results.


359

Fig. 3. Internal structures of RT-BOINC server

the server should finish each transaction within 1 ms. If the applications need guarantees of real-time execution, the worst-case execution time on the serverside should be less than 1 ms for each transaction. To provide such a low bounded execution time, the internal server structures should be designed to limit the gap between the average and the worst-case. 3.3

Design of RT-BOINC

RT-BOINC was designed to provide guaranteed real-time performance for distributing work and reporting their results in the BOINC server. To do this, we modified several components of BOINC server (see Figure 3), and added new data structures and interfaces for retrieving them. The major difference between the original BOINC and RT-BOINC is the management of data records. RT-BOINC does not use the database, and uses instead only in-memory data structures shared among daemon processes. We also modified the internal structures of the server daemon processes to reduce their complexity. The original BOINC uses MySQL as the main DBMS, and this widens the gap between the average and the worst-case execution time for reasons discussed in Section 3.1. In RT-BOINC, we replaced the database with in-memory data structures, which provide O(1) lookup, insertion, and deletion of data records. The data structures are shared by several daemon processes via shared memory IPC. Data Structures for Real-time Operations. Figure 4 shows an example of retrieving data from the shared memory data structures. In this example, we retrieve one result that has workunitid = 0x1234 from the result table, where workunitid is a field of the result. In Fig. 4, two-level lookup tables are used to reduce the maximum length of a list. In the worst-case, 256 entries will be scanned in search of the workunit ID.

360


Fig. 4. An example of retrieving result in RT-BOINC

Fig. 5. Data insertion and deletion examples in RT-BOINC

Figure 5(a) shows the case of inserting a new result to the data structures. First, we need to find a place to store the new result. To do this, we use a lookup pool for available (free) results. We can get an available result field’s ID at the end of list, and remove the entry from list. Then, we can insert the result to the data structure with constant time2 . In Fig. 5(b), we see how to delete the existing result from the data structures. If we want to delete the result which has id = 1234, we insert the value 1234 to the end of lookup list. Then, we can invalidate the result by removing the valid-flag of the result data3 . Server Processes in RT-BOINC. As we mentioned, each daemon process in BOINC has at-least linear time complexity for handling workunits and their results. To reduce complexity by orders of magnitude, we modified the internal structures of the server processes. We replaced all BOINC database-related code with O(1) lookup, insertion, and deletion code. We removed unnecessary loops and redundant code from the remaining parts of the server processes. 2 3

At the same time, we need to manage other lookup tables such as workunitid, which are presented in Fig. 4. If we have active lookup records for the result in other data structures, we should delete them also.


3.4

361

Prototype Implementation

We implemented RT-BOINC on top of the BOINC server source code4 . The full source code of RT-BOINC (prototype implementation) are available at the following website: http://rt-boinc.sourceforge.net/ Data Format Compaction. We reduced significantly the memory consumption of the original data types of BOINC. We reduced unnecessary large blocks from workunit, result, user, team, and many other data types. For example, one result data has three huge fields to store XML code and standard output messages. The size of each field is 64KB, which corresponds to the MySQL BLOB data type. Then, each result record consumes more than 192KB, and if we have 10, 000 records on the server, this will consume more than 1.92GB of space for just handling the result records. However RT-BOINC does not use the database nor BLOB data types, so it does not need to consume as much space. We reduced the size of each field by a factor of 8, and we also made similar reductions for the other data types. Detailed information of data format compaction is available at the following website: http://rt-boinc.sourceforge.net/dataformat.html Data Structures and Interfaces. We implemented the data structures using shared memory IPC among several daemon processes. The prototype implementation of RT-BOINC supports up to 64K active hosts, which is reasonable based on the size of most BOINC projects [15]. To provide O(1) lookup, insertion, and deletion operations on the data structures, we used two-level lookup tables and fixed-size list structures (see Figs. 4 and 5). We used a 4-bit lookup table for each level, thus each lookup table has 24 = 16 fields (same as Fig. 4). Also, we made a few limitations for the prototype version of RT-BOINC. We assume that a workunit has a one-to-one relationship with a result. In our prototype, the memory space overhead for the O(1) data structures is about 38.6% of the total memory usage (where the total memory requirement is 1.09GB for 10, 000 hosts). 3.5

Compatibility with the Original BOINC

All of our modifications of the BOINC server source code preserve compatibility with the original BOINC implementation and components (such as the client). In Figs. 1 and 3, we can observe that the main components of RT-BOINC are exactly the same as that of BOINC. RT-BOINC has the same set of server processes, and the flow of work distribution and reporting is the same as the original BOINC. Therefore, most of RT-BOINC server components are compatible with the existing project configurations and their applications.

4

Performance Evaluation

We made micro and macro-level benchmarks to determine the performance of both the low-level operations (such as insertion, update, and remove) and 4

We used the server stable version of BOINC in November 2009.

362


high-level server processes (such as the feeder, transitioner, and validator). We measured performance in terms of the average, and the worst-case execution time of these operations and processes in BOINC and RT-BOINC. For the micro-benchmarks, we implemented a program that generates every possible key value for the server-side operations. Key values correspond to user, host, workunit, result id’s , and other id’s related to the performance. For the macro-benchmarks, we implemented an emulator of the BOINC client, which typically runs on each volunteered host. The emulated client uses the identical protocol of the BOINC client for requesting workunits, and returning results. Moreover, it uses the server processes in the same way as the BOINC client itself. The emulated client generates server requests for all possible key values. Table 1. Specification of the base server platform used in this paper Component Processor Main memory Secondary storage Operating system

Description 1.60GHz, 3MB L2 cache 3GB (800MHz) 64GB Solid State Drive Ubuntu 9.10

Notes Intel Core-2 Duo with VT Dual-channel DDR3 SLC type Linux kernel 2.6.31-19

Fig. 6. Micro benchmark results of BOINC and RT-BOINC


363

With regard to the workunits, we used the uppercase application, which is a synthetic application that converts contents of a text file to uppercase. The setup in Table 1 allows us to measure the real performance in generalpurpose, off-the-shelf server system. In Fig. 6(a), Y-axis is in log-scale, and most of execution times of operations have more than a 2-step difference in terms of average execution time. In the worst-case, most times have a 3-step difference (in Fig. 6(b)). This means that RT-BOINC has improvements of more than 100-times for the average case, and almost 1, 000-times for the worst-case (which is presented in Fig. 6(d)). In addition, Fig. 6(c) shows that RT-BOINC’s performance gap between the worst and average case is much lower than that of the original BOINC. Based on these results, we can observe that RT-BOINC provides low worst-case execution time compared to the original BOINC for each operation. Figure 7 shows the results of each server process in BOINC and RT-BOINC when the server has low-load conditions. In Fig. 7(a) and (c), the results show almost a 1-step difference, and Fig. 7(b) shows almost a 2-step difference between BOINC and RT-BOINC. The results show that RT-BOINC provides almost 100times lower worst-case execution time than BOINC. In Fig. 7(d), we can observe that the performance gap of RT-BOINC is much lower than that of BOINC. Figure 8 shows the same set of results when the server has conditions of high-load. In these results, we observe almost a 1-step difference in the averagecase, and almost a 2.5-steps difference in the worst-case. Fig. 8(d) shows that BOINC has a significant difference between worst-case and average-cases, which RT-BOINC improves immensely. From the results in Fig. 8, we found that the “end-to-end” transaction time of one workunit going through all server processes in RT-BOINC is 4.1 ms for

Fig. 7. Macro benchmark results of BOINC and RT-BOINC (low-load: 1 host)

364


Fig. 8. Macro benchmark results of BOINC and RT-BOINC (high-load: 10,000 host)

Fig. 9. Worst-case execution time difference between low and high load conditions

average, and 31.2 ms for the worst-case when we do not consider pipe-lined parallel execution of the server processes. If the server can execute multiple threads at the same time, the transaction time of each workunit decreases significantly. For instance, in the average case, the server can perform 1, 000 workunits per second with only 4 parallel threads of execution. The hardware setup in Table 1 supports this degree of parallelism. If the server supports 32 threads (as on a standard dual quad-core processor with hyperthreading) with the same performance presented in Fig. 8, the server can perform almost 1, 000 workunits per second even in the worst-case. In Fig. 9, BOINC shows a big difference in performance than RT-BOINC under different load conditions wrt the number of hosts. This result has significant meaning on both predictability and scalability of the server system. RT-BOINC


365

provides almost the same worst-case execution time, even if the server has significant change of host load. Based on these results, we can observe that RTBOINC provides significantly low worst-case execution time compared with the original BOINC.

5


In this paper we proposed RT-BOINC, which is a platform for real-time and highly interactive computing. This system can be used on any large distributed system, such as Clouds, Grids, or Volunteer Computing platforms, where workloads consists of tens of thousands of real-time tasks. In RT-BOINC, every component provides bounded execution time. Thus it helps to provide the guaranteed execution of real-time applications. We implemented it based on the original BOINC, and our evaluation results show that RT-BOINC has low worst-case execution time and reasonable memory space usage compared with BOINC. For future work, we are interested in the following issues for RT-BOINC. First, we are interested in conducting dynamic shared memory management when the server system does not have enough main-memory space. Second, we are interested in studying the trade-offs between time and space of our data structures.

Acknowledgments This research was supported by supported the ALEAE project (INRIA ARC), and the ANR Clouds@home project (contract ANR-09-JCJC-0056-01).

References 1. Silberstein, M., Sharov, A., Geiger, D., Schuster, A.: Gridbot: Execution of bags of tasks in multiple grids. In: SC 2009: Proceedings of the 2009 ACM/IEEE Conference on Supercomputing. ACM, New York (2009) 2. Lee, C.S., Wang, M.H., Chaslot, G., Hoock, J.B., Rimmel, A., Teytaud, O., Tsai, S.R., Hsu, S.C., Hong, T.P.: The Computational Intelligence of MoGo Revealed in Taiwan’s Computer Go Tournaments. IEEE Transactions on Computational Intelligence and AI in Games (2009) 3. Deep Blue (chess computer), http://en.wikipedia.org/wiki/Deep_Blue_ (chess_computer) 4. Lopez, J., Aeschlimann, M., Dinda, P., Kallivokas, L., Lowekamp, B., O’Hallaron, D.: Preliminary report on the design of a framework for distributed visualization. In: Proceedings of the International Conference on Parallel and Distributed Processing Techniques and Applications (PDPTA 1999), pp. 1833–1839 (1999) 5. Capsicum Group: Digital Forensics, http://www.capsicumgroup.com/ content-pages/services/digital-forensics.html 6. Dinda, P.: A Prediction-Based Real-Time Scheduling Advisor. In: Proceedings of the International Parallel and Distributed Processing Symposium, IPDPS 2002 (2002)

366


7. Sonnek, J.D., Nathan, M., Chandra, A., Weissman, J.B.: Reputation-based scheduling on unreliable distributed infrastructures. In: ICDCS, p. 30 (2006) 8. Andrzejak, A., Kondo, D., Anderson, D.P.: Ensuring collective availability in volatile resource pools via forecasting. In: De Turck, F., Kellerer, W., Kormentzas, G. (eds.) DSOM 2008. LNCS, vol. 5273, pp. 149–161. Springer, Heidelberg (2008) 9. Ratnasamy, S., Handley, M., Karp, R.M., Shenker, S.: Topologically-aware overlay construction and server selection. In: INFOCOM (2002) 10. Kacsuk, P., Marosi, A.C., Kovacs, J., Balaton, Z., Gombs, G., Vida, G., Kornafeld, A.: Sztaki desktop grid: A hierarchical desktop grid system. In: Proceedings of the Cracow Grid Workshop 2006, Cracow, Poland (2006) 11. Gorlatch, S., Glinka, F., Ploss, A., M¨ uller-Iden, J., Prodan, R., Nae, V., Fahringer, T.: Enhancing grids for massively multiplayer online computer games. In: Luque, E., Margalef, T., Ben´ıtez, D. (eds.) Euro-Par 2008. LNCS, vol. 5168, pp. 466–477. Springer, Heidelberg (2008) 12. Catalog of BOINC Powered Projects - Unofficial BOINC Wiki, http://www.boinc-wiki.info/Catalog_of_BOINC_Powered_Projects 13. Anderson, D.P.: Talk at Condor Week, Madison, WI, http://boinc.berkeley.ed-u/talks/condor_boinc_06.ppt (2006) 14. MySQL: Developer Zone, http://dev.mysql.com/ 15. BOINC Statistics, http://boincstats.com/stats/

Scheduling Scientific Workflows to Meet Soft Deadlines in the Absence of Failure Models Kassian Plankensteiner, Radu Prodan, and Thomas Fahringer Institute of Computer Science, University of Innsbruck, Technikerstr. 21a, 6020 Innsbruck, Austria {kassian.plankensteiner,radu,tf}@dps.uibk.ac.at

Abstract. Highly distributed systems such as Clouds and Grids are used to execute complex scientific workflow applications by researchers from various areas of science. While scientists rightfully expect efficient and reliable execution of their applications, current systems often cannot deliver the required Quality of Service. We propose a dynamic execution and scheduling heuristic able to schedule workflow applications with a high degree of fault tolerance, while taking into account soft deadlines. Experimental results show that our method meets soft deadlines in volatile highly distributed systems in the absence of historic failure trace data or complex failure models of the target system.

1

Introduction

The workflow paradigm has emerged as one of the most successful paradigms for execution of scientific applications on highly distributed systems such as Clouds, computational Grids or meta-computers. While scientists rightfully expect efficient and reliable execution of their applications on these systems, current systems often cannot deliver the Quality of Service (QoS) needed for widespread acceptance as ubiquitous scientific tools [10]. The problem lies in the complex and highly dynamic nature of highly distributed systems, which exhibit high failure rates that the systems have to be able to cope with [6]. In real-life, users typically want to know an estimation of the execution time for their application before deciding to have it executed. In many cases, this estimation can be considered to be a soft deadline, defined as the requirement that the execution time satisfies a certain deadline with some probability α [16]. In other words, if a soft deadline is not met, the results are still useful, the only difference being in the QoS provided by the system. If a system regularly fails to meet soft deadlines by large amounts, users will be dissatisfied with the Quality of Service offered by the system. If a hard deadline is broken, there is no need for the results at all. Weather forecasting and medical simulations are two areas with typical hard deadlines.

This work is partially funded by the European Union, grant agreement number 261585 (SHIWA project).


368

K. Plankensteiner, R. Prodan, and T. Fahringer

In previous work, we proposed the Resubmission Impact (RI) heuristic [9] for scheduling workflows in faulty environments in the absence of failure models, and showed that it can provide a high degree of fault tolerance with a significantly reduced resource consumption compared to other common techniques. In this paper, we build on top of this heuristic a new method and algorithm to meet soft deadlines for workflows in highly distributed systems without failure models. The paper is organised as follows. Section 2 discusses related work. Section 3 summaries previous work on the Resubmission Impact heuristic, which forms the foundation for our work in this paper. In Section 4 we introduce a new dynamic scheduling heuristic for meeting soft deadlines without a failure model. We present experimental results in Section 5 and conclude in Section 6.

2

Related Work

Considering faults in the scheduling and execution of a workflow application, we can divide existing techniques into three broad categories. The first category of techniques [7,15] build on (often extensive and complex) failure models that aim to describe the failure probability of a task on a resource in a certain time interval. Building such a model can be a very difficult task that often requires years worth of traces of failure data about the specific target environment. On the other hand, commercial Cloud providers such as Amazon do not disclose any information regarding their infrastructure at all, and failure traces are often a closely guarded secret. The second category of techniques [2,5,13,14] tackle the problem with the help of Service Level Agreements (SLAs) or Advance Reservations (AR). Techniques in this category usually rely on agreements with the end user as well as the resource providers, describing the QoS requirements for the execution of the given workflow. The workflow execution is accepted and started only if binding agreements with the resource providers can be reached. Given the strict criteria of QoS in the SLAs, this shifts the responsibility of dealing with faults to the resource providers. The problem with this approach is the lack of support for negotiating the required fine-grained QoS terms in todays Cloud/Grid systems. The third category of techniques can often be found in current Grid workflow systems. As described in our earlier work on fault tolerance in [10], they often deal with the unreliability of the environments by resorting to fault tolerance mechanisms such as task replication and task resubmission without considering a failure model of the target system. This method has the advantage of applicability to new and unknown distributed environments, where no suitable failure model is available. Unfortunately, it often leads to unnecessarily large resource consumption and to large differences of the expected execution time (as promised to the end user) and the real execution time of a workflow and therefore degrades the Quality of Service. Our work in this paper belongs to this third category. It offers better QoS than many of the existing approaches in this category by foreseeing possible deadline violations at runtime and taking appropriate measures to positively influence the execution of the remainder of the workflow.

Soft Deadlines in the Absence of Failure Models

3

369

Resubmission Impact

Resubmission Impact (RI) [9] is a heuristic for scheduling workflow applications in the presence of faults. RI is based on the idea of finding a compromise solution that balances the tradeoffs between resource consumption when using replication and the increase in execution time when using resubmission, by establishing a metric that describes the impact of the resubmission of a task on the workflow makespan. Let T denote the set of tasks in a workflow and E the set of control flow and data flow dependencies between tasks. Let φi ∈ Φ denote the amount of work (i.e. instructions) that each task ti ∈ T requires to be completely processed. Additionally, we define Ψ to be a set of integer values defining the replication sizes ψi ∈ Ψ for each task ti ∈ T . Initially, we have ∀ψi ∈ Ψ : ψi = 0. A workflow Γ is defined as a 4-tuple: Γ = (T, E, Φ, Ψ ). We also define Θ to be the total set of distributed resources available, and repl max , res max as the maximum replication count and maximum resubmission count, respectively. Both repl max and res max have to be specified by the user as input. The RI heuristic can be seen in Algorithm 2 and consists of two phases. The first phase (lines 1 − 7) establishes the resubmission impact metric. First, we make a copy Φ of the set Φ defining the work amount of each task in the workflow. Then, we enlarge the amount of work φi in task ti by multiplying it with the resubmission count res max , and define a new workflow Γ containing the new work amount for each task. Afterwards, we compute the difference in expected execution time δi of Γ and Γ , by scheduling both of them using the HEFT [11] algorithm (line 5). We repeat these steps for every task ti ∈ T . Finally, we compute the resubmission impact λi for every task ti by normalising δi against the maximum of all δj , j = 0 . . . n (line 7). The resubmission impact λi reflects the impact of having to the resubmit task ti the given amount of times on the expected execution time of the whole workflow. In the second phase of the RI algorithm (lines 8 − 11), we create a copy of the replication count set Ψ and call it Ψ . For every ψj ∈ Ψ , we set ψj to be λj · repl max (line 9). This defines the replication count for every task tj ∈ T . Finally, we replace the set of replication counts Ψ ∈ Γ with the newly defined Ψ and send it to HEFT replication [9] for scheduling. HEFT replication (Algorithm 1) is an extension to the Heterogeneous Earliest Finish Time (HEFT) [11] algorithm. It works in two phases: in the first phase (line 1 − 3), it computes an upward rank for all the tasks based on the length of the critical path to the end of the workflow (also called the B-level), and sorts them based on the rank. In the second phase (line 4 − 14), it maps the tasks in descending order of their ranks to the resource giving the earliest expected completion time (ECT). After each task is mapped, its parent tasks are checked (line 7 − 12). If all child tasks have been mapped to a resource already, the previously set number of replicas of the parent are created and mapped to resources according to the earliest expected completion time.

370


Algorithm 1. The HEFT algorithm with task replication. 1: HEFT_replication(Workflow wf, ResList r) 2: List ranks = computeRank_u(wf); 3: TaskList tasks = sortTasks(wf, ranks); 4: for each task t in tasks 5: mapAccordingToECT(t, r); 6: TaskList parents = t.getAllParents(); 7: for each p in parents 8: if(p.allChildrenMapped()) 9: int rc = p.getReplicaCount(); 10: TaskList replica = p.clone(rc); 11: mapAllAccordingToECT(replica,r); 12: endfor 13: endfor 14: return wf;

Algorithm 2. The Resubmission Impact heuristic. Require: Γ = (T, E, Φ, Ψ ): scientific workflow, Θ: set of resources, repl max : maximum replication count, res max : maximum resubmission count 1: for all ti ∈ T do 2: Φ ← Φ 3: φi ← φi · res max 4: Γ ← (T, E, Φ , Ψ ) 5: δi ← HEFT(Γ , Θ)−HEFT(Γ, Θ) {compute the difference in predicted execution time} 6: end for 7: λi ← maxδi (δj ) , i = 0 . . . n j=0...n

8: 9: 10: 11:

4

Ψ ← Ψ ψj ← λj · repl max , j = 0 . . . n Γ ← (T, E, Φ, Ψ ) return HEFT replication(Γ, Θ)

Scheduling for Soft Deadlines

The RI heuristic is able to schedule workflows with a high degree of fault tolerance while consuming a reasonably low amount of resources. However, since RI schedules the full workflow before it is executed, it cannot react to unexpected environment behaviour like periods of high failure occurrences. This can lead to situations where the replication approach can fail, and many tasks have to fall back to the resubmission technique. Consequently, the application runtimes can exhibit large deviations from the value expected immediately after scheduling, degrading the QoS for the end user. To this end, we propose an extension to the RI heuristic that is able to provide an estimation of the execution time in the form of a soft deadline to the end


371

user. To achieve this, we apply changes to RI in the following areas: (1) execution control by introducing a monitoring step after the completion of each task and rescheduling the workflow remainder if the soft deadline is likely to be missed; (2) scheduling by introducing a rescheduling heuristic that is able to influence the parameters of the fault tolerance strategy of RI, therefore decreasing the likelihood of a violation of the soft deadline. 4.1

Execution Control

Execution Control systems relying on pure full-ahead schedulers such as the RI heuristic are unable to readjust their fault tolerance strategy at runtime if the high fault rate makes the initially chosen replication amount ineffective. Once this happens, the execution systems have to fall back to resubmission, which leads to spikes in the workflow makespans that exceed the expected execution times often by large amounts, deteriorating the QoS for the end-user. To solve this issue, we propose a dynamic execution control strategy as illustrated in Algorithm 3. We start by sending the workflow to the scheduler (line 1), which uses the RI heuristic to map tasks to computational resources and assigns to every task an expected start time and an expected finish time. Based on the schedule, we also obtain the expected execution time of the workflow and generate a soft-deadline for the execution (line 2). The workflow soft-deadline td is computed as follows: ζ max 1 td = ts · 1 + r · , (1) n n=1 where ts is the expected execution time as determined by the initial schedule of the workflow, r is the rescheduling threshold, and ζmax is the maximum (re)scheduling count. The rescheduling threshold r > 0 is a percentage given by the user that defines how much the real execution time is allowed to exceed the initial schedule before the first rescheduling cycle is started. The maximum (re)scheduling count ζmax > 1 defines the upper limit to the number of scheduling cycles to be performed for a given workflow during runtime, and is also given by the user. In line 3, we compute the initial value for the rescheduling threshold time tr as follows: ζ ri , (2) tr = ts · n n=1 where ζ is the current scheduling cycle count (i.e. ζ = 1 for the initial cycle). We enter the execution control loop in line 4 and start with the workflow execution by identifying the tasks that are ready to execute according to the current state of the workflow. Every ready task is sent to its corresponding resource based on the mapping done by the scheduler. Once the ready tasks are submitted, we wait for finished tasks to arrive back at the execution control component (line 8). As finished tasks return to the executor (line 9), we include a monitoring step that decides if we need a rescheduling operation. We compare the real finish time of the tasks to their planned finish time as calculated

372


Algorithm 3. Adapted workflow execution control. 1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: 15: 16:

send the workflow to the scheduler compute workflow deadline td, inform user compute rescheduling threshold time tr while (workflow not completed) do get list of ready tasks for each ready task T send T to resource wait for a task to complete receive completed task T1 if (max(0,T1.realEndTime-T1.expectedEndTime) > tr) && ( zeta < zeta_max ) then reschedule workflow remainder zeta = zeta + 1 compute new tr if (last task received successfully) then set workflow to be completed end while

by the initial scheduling step. If the time difference is larger than the current rescheduling threshold time tr , we invoke a rescheduling cycle for the remainder of the workflow (see Section 4.2), compute a new rescheduling threshold time by using Equation 2, and increment the scheduling cycle count ζ (line 11-13). Since every rescheduling cycle increases the amount of replication in the rest of the workflow, it is no longer invoked if the scheduling cycle count reaches ζmax to avoid excessive resource usage. 4.2

Rescheduling

As explained in the previous section, the new rescheduling heuristic is invoked if the execution control component discovers that the real execution of the workflow application is too far behind the scheduled workflow execution plan, which implies a high probability of missing the soft-deadline as given to the user. The job of the rescheduling heuristic is to decrease the probability of violating a deadline by adjusting the amount of replication, ψj , of each remaining task, tj , of a workflow as follows: 1 (3) ψj = λj ζ · repl max where λj and repl max are the resubmission impact of a task tj ∈ T and the maximum replication count, respectively. 1 Figure 1 shows the effect of using the term λj ζ in Equation 3. We can see that the higher the scheduling cycle ζ is, the higher the replication count becomes for tasks with a smaller resubmission impact. In other words, with every rescheduling cycle we start replicating tasks that have not yet been replicated and increase the replication count on tasks that are already subject to replication. We base


373

1

replication multiplier

0.8 0.6 0.4 =1 =2 =3 =4

0.2 0 0

0.2

0.4

0.6

0.8

1

resubmission impact

Fig. 1. Replication multiplier

Algorithm 4. The Resubmission Impact rescheduling heuristic. Require: Γ = (T, E, Φ, Ψ ): scientific workflow, Θ: set of resources, repl max : maximum replication count (integer), ζ: scheduling cycle (integer) 1: Ψ ← Ψ 1 2: ψj ← λj ζ · repl max , j = 0 . . . n 3: Γ ← (T, E, Φ, Ψ ) 4: return HEFT replication(Γ, Θ)

this on the resubmission impact λ of a task because it is a measure of the impact of rescheduling that task on the global execution time of a workflow. Algorithm 4 shows the proposed rescheduling heuristic, which in line 2 uses Equation 3 to adjust the replication count ψ of the tasks in the workflow. Finally, HEFT replication (see Algorithm 1) is used to schedule the remaining workflow tasks including the replicas.

5

Experiments

To evaluate our method we implemented a workflow management system based on the GridSim [3] Grid simulator, capable of scheduling and executing workflows according to Algorithms 3 and 4. 5.1

Resource Failure Model

In a recent study, Iosup et al. [6] show that the mean time between failure (MTBF) for clusters in existing large-scale Grid environments can be modeled using a Weibull distribution. We utilised their findings to create a resource failure model for our simulated environment which considers the MTBF, the size of a failure in terms of the number of processors affected, as well as the duration of a

374


failure (or mean time to recover (MTTR)). Thus, according to [6], we model the MTBF with a Weibull distribution having a shape parameter ranging between 11.5 and 12.5, the duration of a failure using a LogNormal distribution and the size of a failure using a Weibull distribution with a shape parameter between 1.5 and 2.4. 5.2

Simulation Setup

Our simulated heterogeneous resource environment comprises 5480 processors with different computational power distributed over twelve clusters. We generated failure models for each of the twelve clusters based on the Weibull distributions for MTBF and failure size, and based on the LogNormal distribution for the MTTR, as described in Section 5.1. To study the impact of resource reliability on our (re)scheduling algorithm, we varied the Weibull scale parameter to create three different failure models based on different MTBF values: five hours for shaky resources, three days for the normal (average case) resources, and one week for stable resources. We generated three types of workflows modeled after three real-world scientific workflow applications: Wien2K [1], a scientific workflow for performing electronic structure calculations of solids; MeteoAG, a workflow application designed for meteorological simulations based on the RAMS [4] numerical atmospheric model; POV-Ray [8], a movie-rendering workflow based on the popular open source Persistence of Vision raytracer. For each application type, we created 50 workflows based on randomly generated input problem sizes. Our experimental suite therefore consists of 150 different workflows with a wide range of computational tasks, task lengths, data transfer sizes and estimated execution times (from a few hours to longer than one month). The workflows are created based on historical performance data logged from real workflow runs executed in the Austrian Grid [12] during the past few years. We set r = 0.2, resmax = 3 and two different values for repmax , 2 and 4, to study the impact of the maximum replication count on our new technique. To compare the behaviour of our new scheduling method to our previous method of full-ahead scheduling with RI, we investigate a maximum scheduling cycle count ζmax of 5 (initial scheduling and up to 4 rescheduling cycles) and 1 (only initial scheduling - equivalent to RI). To summarise, we run the execution batch of 150 workflows utilising two different scheduling schemes (new rescheduling technique and the previous RI technique), three different resource failure characteristics (shaky, normal, stable) and two different maximum replication sizes (two and four), summing up to over 1800 workflow runs, accumulating a total of tens of millions of CPU hours. 5.3


While the pure full-ahead scheduling approach in our previous work on RI achieved a high level of fault tolerance for workflow executions, it proved to also generate large spikes in the execution time behavior of workflows in environments with a high fault rate. This is explained by the fact that the method is unable to readjust its fault tolerance strategy at runtime if the high fault rate


375

made the chosen replication amount ineffective. Once this happens, the method has to fall back to resubmission, which leads to spikes in the workflow makespans that exceed the expected execution times by large amounts, deteriorating the QoS for the end-user. Our new method foresees potential situations leading to longer than expected workflow makespans and actively steers against this by adjusting the replication size at runtime. The ultimate goals of our approach are twofold. Firstly, we want to mitigate spikes in the makespan of workflows, thereby increasing the QoS delivered to the end user. Secondly, we want to increase the percentage of workflows that finish within the given soft-deadline. Figure 2 and 3 show the cumulative distribution functions (CDF) of the successfully finished workflow executions with a maximum replication count repmax of 4 and 2, respectively, for all three fault characteristics: stable (a), normal (b) and shaky (c) and for both the new rescheduling technique and the previous RI approach. We define RES (shown on the horizontal axis) to be the ratio of the workflow makespan to the workflow soft-deadline td . RES values below 1 denote that a workflow has successfully completed before the given soft-deadline, while RES values larger than 1 denote that the given soft-deadline was overshot. A workflow with an RES of 1.0 has finished exactly at the given soft-deadline. The vertical axis shows the percentage of workflows that finished within a given RES. More concretely, assuming that the point [0.8, 60.0] belongs to the drawn function, then 60% of the workflows finished within an RES of 0.8, or in other words, 60% of the workflows finished at least 20% faster than planned by the soft-deadline. All the functions in Figures 2 and 3 exhibit a steep slope at an RES value of around 0.65. This is a result of workflows being executed entirely according to the initial full-ahead schedule, without experiencing faults that the initial replication setting could not compensate for. These workflows therefore did not have to resort to resubmission for their tasks, hence they were able to finish within the timespan initially expected by the first schedule. To evaluate the improvements towards our first goal, the mitigation of large spikes in the workflow makespans, we compare the maximum RES value of both approaches. A lower maximum RES value represents a better mitigation of the makespan spikes. In the case of shaky resources and repmax = 4 (Figure 2(c)), the new rescheduling approach can finish all of the successfully executed workflows within an RES of 1.06. In contrast, the previous approach, RI with full-ahead scheduling, only leads to an RES value of 1.81, which means that the workflow makespans overshot the soft-deadline by up to 81%. For normal resources and repmax = 4 (Figure 2(b)), our new technique shows a 12% improvement in the maximum RES value from 1.17 to 1.05, while in the stable resource case (Figure 2(a)) it exhibits a slightly higher improvement, from 1.17 to 1.04. Figure 3 shows that our technique also manages to maintain an advantage when using a maximum replication count of 2, although it is not able to mitigate the makespan spikes as well as in the case of repmax = 4. This is due to the limited fault tolerance potential when constrained to a maximum of two replicas for each task. The fact that the maximum RES value of our approach in the case

376

K. Plankensteiner, R. Prodan, and T. Fahringer 100 Percentage of Workflows

Percentage of Workflows

100

80

60

40

80

60

40

rescheduling RI only 20

0.5

0.75

1

1.25

1.5

rescheduling RI only 1.75

20

2

0.5

0.75

1

RES

(a) Stable environment.

Percentage of Workflows

60

40

60

40

0.5

0.75

1

1.25

1.5


20

2

0.5

0.75

1

RES

1.25

1.5

1.75

2

RES

(b) Normal environment.

(b) Normal environment. 100 Percentage of Workflows

100 Percentage of Workflows

2

80

rescheduling RI only

80

60

40

80

60

40

rescheduling RI only 20

1.75

100

80

20

1.5

(a) Stable environment.

100 Percentage of Workflows

1.25 RES

0.5

0.75

1

1.25

1.5


2

RES

20

0.5

0.75

1

1.25

1.5

1.75

2

RES

(c) Shaky environment.

(c) Shaky environment.

Fig. 2. CDF of workflow executions with repmax = 4 and different fault properties

Fig. 3. CDF of workflow executions with repmax = 2 and different fault properties

of stable and normal environments is smaller at repmax = 2 than at repmax = 4 is due to the lower execution success rate when using less replication. To quantify the improvements towards the second goal, a higher percentage of workflows finishing within the given soft-deadline, we compare the vertical distance between the function plots of the two methods at an RES of 1.0. A higher distance represents a larger improvement towards our goal. Figure 2(c) shows that out of all the successfully executed workflows on shaky resources and a maximum replication count of 4, our new technique finishes 95% within


377

the soft-deadline, while the RI technique only manages to finish 78% within the deadline. For the normal resources (Figure 2(b)), the new approach finishes 99% of the workflows within the deadline, while RI achieves 95.5%. On stable resources (Figure 2(a)), our proposed method still exhibits a slight improvement: 99.7%, as opposed to 96% for the RI approach. Similarly, in the case of repmax = 2 (Figure 3), our new technique consistently manages to finish a higher percentage of the workflows within the given deadline compared to the RI method. Despite the lower potential for dynamic adaptation with a maximum replication size of 2 as opposed to 4 in the previous cases, the new approach still manages to increase the percentage of workflows that meet the deadline from 76% to 87% on the shaky resources and from 99.3% to 100% on the normal and stable resources. In conclusion, our experiments show that our newly proposed technique drastically reduces spikes in the makespans of workflow executions in faulty environments. Additionally, the remaining makespan spikes are of considerably shorter durations, up to 75%, than the ones produced by the full-ahead scheduling RI approach. Finally, we demonstrated that our method is able to consistently meet soft-deadlines for a large percentage of the successfully executed workflows, even in unreliable environments.

6

Conclusions

We proposed in this paper a new method and algorithm to meet soft deadlines for workflow applications on highly distributed systems in the absence of failure models. To improve upon the problem of large makespan spikes and low Quality of Service in other approaches, our new method foresees possible deadline violations at runtime and takes appropriate measures to positively influence the execution of the remainder of the workflow. We achieve this by improving on our previous work, the RI heuristic, by introducing a new method for execution control as well as a rescheduling algorithm that is able to dynamically readjust the fault tolerance strategy at runtime. We evaluated our method through comprehensive simulation using traces of three real applications executed in the Austrian Grid environment. Our results show that our approach meets all of the key goals: it strongly reduces spikes in the workflow makespans (by up to 75%) and increases the percentage of workflows that finish within the soft-deadline (by up to 17%). Compared to related work, our approach does not rely on resource failure prediction which is difficult to achieve even with years of historic failure trace data of the target environment. Therefore, our approach has the advantage of immediate applicability in new, unknown, or even proprietary systems (i.e. commercial Cloud systems) where historical trace data is unavailable.

References 1. Blaha, P., Schwarz, K., Madsen, G., Kvasnicka, D., Luitz, J.: WIEN2k: An Augmented Plane Wave plus Local Orbitals Program for Calculating Crystal Properties, Institute of Physical and Theoretical Chemistry, TU Vienna (2001)

378


2. Brandic, I., Pllana, S., Benkner, S.: Specification, planning, and execution of qosaware grid workflows within the amadeus environment. Concurrency Computat.: Pract. Exper. (January 2008) 3. Buyya, R., Murshed, M.: GridSim: a toolkit for the modeling and simulation of distributed resource management and scheduling for Grid computing. Concurrency and Computation-Practice & Experience 14(13-15), 1175–1220 (2002) 4. Cotton, W.R., Pielke Sr., R.A., Walko, R.L., Liston, G.E., Tremback, C.J., Jiang, H., McAnelly, R.L., Harrington, J.Y., Nicholls, M.E., Carrio, G.G., McFadden, J.P.: RAMS 2001: Current status and future directions. Meteorology and Atmospheric Physics 82(1), 5–29 (2003) 5. Guo, L., McGough, A., Akram, A., Colling, D., Martyniak, J.: Qos for service based workflow on grid. In: Proceedings of UK e-Science 2007 All Hands Meeting (January 2007) 6. Iosup, A., Jan, M., Sonmez, O., Epema, D.: On the dynamic resource availability in grids. In: 2007 8th IEEE/ACM International Conference on Grid Computing, pp. 26–33 (2007) 7. Kandaswamy, G., Mandal, A., Reed, D.: Fault tolerance and recovery of scientific workflows on computational grids. In: 8th IEEE International Symposium on Cluster Computing and the Grid, CCGRID 2008, pp. 777–782 (2008) 8. Plachetka, T.: POVRAY – persistence of vision parallel raytracer. In: Computer Graphics International, pp. 123–129 (1998) 9. Plankensteiner, K., Prodan, R., Fahringer, T.: A new fault tolerance heuristic for scientific workflows in highly distributed environments based on resubmission impact. In: Fifth IEEE International Conference on e-Science, e-Science 2009, pp. 313–320 (December 2009) 10. Plankensteiner, K., Prodan, R., Fahringer, T., Kertesz, A., Kacsuk, P.: Faulttolerant behavior in state-of-the-art grid worklow management systems. Tech. Rep. TR-0091, Institute on Grid Information, Resource and Worklow Monitoring Services, CoreGRID - Network of Excellence (October 2007) 11. Topcuoglu, H., Hariri, S., Wu, M.: Performance-effective and low-complexity task scheduling for heterogeneous computing. IEEE Transactions on Parallel and Distributed Systems 13(3), 260–274 (2002) 12. Volkert, J.: Austrian grid: Overview on the project with focus on parallel applications. In: International Symposium on Parallel and Distributed Computing (2006) 13. Wieczorek, M., Siddiqui, M., Villazon, A., Prodan, R., Fahringer, T.: Applying advance reservation to increase predictability of workflow execution on the grid. In: Second IEEE International Conference on e-Science and Grid Computing (2006) 14. Yu, J., Buyya, R., Tham, C.: Qos-based scheduling of workflow applications on service grids. In: Proceedings of the 1st IEEE International . . . (January 2005) 15. Zhang, Y., Mandal, A., Koelbel, C., Cooper, K.: Combined fault tolerance and scheduling techniques for workflow applications on computational grids. In: 9th IEEE/ACM International Symposium on Cluster Computing and the Grid, CCGRID 2009, pp. 244–251 (2009) 16. Zheng, T., Woodside, M.: Heuristic optimization of scheduling and allocation for distributed systems with soft deadlines. In: Computer Performance, pp. 169–181 (2003)

A GPGPU Transparent Virtualization Component for High Performance Computing Clouds Giulio Giunta, Raffaele Montella, Giuseppe Agrillo, and Giuseppe Coviello University of Napoli Parthenope - Department of Applied Science {giulio.giunta,raffaele.montella}@uniparthenope.it, {giuseppe.agrillo,giuseppe.coviello}@uniparthenope.it

Abstract. The GPU Virtualization Service (gVirtuS) presented in this work tries to fill the gap between in-house hosted computing clusters, equipped with GPGPUs devices, and pay-for-use high performance virtual clusters deployed via public or private computing clouds. gVirtuS allows an instanced virtual machine to access GPGPUs in a transparent and hypervisor independent way, with an overhead slightly greater than a real machine/GPGPU setup. The performance of the components of gVirtuS is assessed through a suite of tests in different deployment scenarios, such as providing GPGPU power to cloud computing based HPC clusters and sharing remotely hosted GPGPUs among HPC nodes. Keywords: Grid, Cloud, High-performance computing, many core, Virtualization, Graphic processing units, Hypervisor, GPGPU.

1

Introduction

In a grid environment, computing power is offered on-demand to perform large numerical simulations on a network of machines, potentially extended all over the world [13]. Virtualization techniques are a promising effort to run generic complex high performance scientific software on a grid, inspiring a novel computing paradigm in which virtualized resources are spread in a cloud of real high performance hardware infrastructures. This model, well known as Cloud Computing [16], is characterized by resource allocation elasticity, high efficiency in using resources, and pay-per-use charge In cloud computing, hardware appliances and software applications are provisioned respectively by means of hardware virtualization and software-as-a-service solutions giving scientists the chance to deal with their specific research needs [8]. Especially in the field of parallel computing applications, virtual clusters instanced on cloud infrastructures suffers from the poorness of message passing performances between virtual machine instances running on the same real machine and also from the impossibility to access hardware specific accelerating devices as GPUs [11],[20]. Recently, scientific computing has experienced on general purpose graphics processing units to accelerate data parallel computing tasks. One the most successful GPU based P. D’Ambra, M. Guarracino, and D. Talia (Eds.): Euro-Par 2010, Part I, LNCS 6271, pp. 379–391, 2010. c Springer-Verlag Berlin Heidelberg 2010

380

G. Giunta et al.

accelerating system is provided by nVIDIA and relies on the CUDA programming paradigm supporting high level languages tools [3]. Presently, virtualization does not allow a transparent use of accelerators as CUDA based GPUs, as virtual /real machines and guest / host real machines communication issues rise serious limitations to the overall potential performance of a cloud computing infrastructure based on elastically allocated resources. In this paper we present the component gVirtuS (GPU Virtualization Service) [1] as results in GPGPUs transparent virtualization targeting mainly the use of nVIDIA CUDA based accelerator boards through virtual machines instanced to accelerate scientific computations. The paper is organized as follows: section 2 contains a brief description of the software architecture and the main design choices; section 3 presents our implementation of the GPU virtualization; section 4 describes the test suite, the performance tests and the obtained results; section 5 shows how gVirtuS can be used in a private high performance cloud computing environment in order to accelerate virtual machines; in section 6 we compare our approach to other existing solutions; finally, section 7 draws conclusions and discusses some future developments.

2

System Architecture and Design

In our prototype the hardware platform consists of two general purpose x86 quad core hyper-threaded processors and two specialized graphics accelerators based on nVIDIA GPUs Tesla 1060C plus a nVIDIA Quadro FX 5600 summing up 8 CPU cores and about 720 GPU cores. A hypervisor concurrently deploys the applications requiring access to the GPU accelerators as VM appliances. The device is under control of the hypervisor, whereas the interface between guest and host machine is performed by a front end/back end system. An access to the GPU is routed via the front end/back end layers under control of a management component, and data are moved from GPU to guest VM application, and vice versa. The front end and the back end layers implement the uncoupling between the hypervisor and the communication layer (Fig. 1a). A key property of the proposed system is its ability to execute CUDA kernels with an overall performance similar to that obtained by real machines with direct access to the accelerators. This has been achieved by developing a component that provides a high performance communication between virtual machines and their hosts. The GPU virtualization is independent of the hypervisor. The choice of the hypervisor deeply affects the efficiency of the communication between guest and host machines and then between the GPU virtualization front end and back end. Xen [7], [9] is a hypervisor that runs directly on the top of the hardware through a custom Linux kernel. Xen provides a communication library between guest and host machines, called XenLoop, which implements low latency and wide bandwidth TCP/IP and UDP connections. XenLoop is application transparent and offers an automatic discovery of the supported virtual machines [21]. VMware [6] is a commercial hypervisor running at the application level. VMware provides a datagram API to exchange small messages, a shared memory API to

A GPGPU Transparent Virtualization Component

381

Fig. 1. The gVirtuS architecture (a, left) and components (b, right)

share data, an access control API to control which resources a virtual machine can access and a discovery service for publishing and retrieving resources [5]. KVM (Kernel-based virtual machine) [2] is a Linux loadable kernel module now embedded as a standard component in most Linux distributions. KVM supplies a high performance guest/host communication component that exposes a set of fully emulated inter virtual machines and virtual/real machines serial ports and network sockets, based on a shared memory approach [4]. gVirtuS relies on a specially designed, fully pluggable communication component independent of the hypervisor and of the communication channel. In our system, the CUDA library, instead of dealing directly with the hardware accelerator, interacts with a GPU virtualization front end. This front end packs the library function invocation and sends it to the back end. The back end deals with the hardware using the CUDA driver; it unpacks the library function invocation and suitably maps memory pointers. Then it executes the CUDA operation, retrieves the results and sends them to the front end using the communicator. Finally, the front end interacts with the CUDA library by terminating the GPU operation and providing results to the calling program. This design is hypervisor independent, communicator independent and even accelerator independent, since the same approach could be followed to implement different kinds of virtualization.

3

gVirtuS Implementation

We have developed a wrapper stub CUDA library which exposes the same interface of the nVIDIA CUDA library. The wrapper library intercepts all the CUDA calls made by an application, collects arguments, packs them into a CUDA call packet, and sends the packet to the front end. The front end driver manages the connection between guest virtual machine and host machine. It uses the services offered by the communicator component to establish event channels through both sides, receiving call packets from the wrap CUDA library, sending these requests to the back end for execution and relaying responses back to the wrap

382

G. Giunta et al.

CUDA library (Fig. 1b). In order to test the front end/back end interaction, the wrap CUDA library behavior and the component interface, we developed a TCPIP based communicator. This component exhibits poor performances due to the network stack, but permits deployment of a complex infrastructure in which the multi-core computing elements and the GPU elements are hosted on diverse hardware systems. We focused on VMware and KVM hypervisors. The VMware Communicator Interface (VMCI) provides a high performance communication among virtual machine instances and the hosting machine. The communicator uses the VMCI SDK APIs to pass packed CUDA function invocations and return the results. VMSocket is the component we have designed to obtain a high performance communicator. vmSocket exposes Unix Sockets on virtual machine instances thanks to a QEMU device connected to the virtual PCI bus. On the host side, the back end is responsible for executing the CUDA calls received from the front end and for returning the computed results. Once a call has been executed, it notifies the guest and passes the results through the connector component. GPU based high performance computing applications usually require massive data transfer between host (CPU) memory and device (GPU) memory. In gVirtuS, the front end / back end interaction turns out to be effective and efficient because there is no mapping between guest memory and device memory and the memory device pointers are never de-referenced on the host side of the CUDA enabled software, since CUDA kernels are executed on our back end side, where the pointers are fully consistent.

4


In order to evaluate and assess the impact of GPU virtualization and its related overhead introduced by gVirtuS, we have carried out an extensive suite of performance tests. We used a workstation Genesis GE-i940 Tesla equipped with a i7- 940 2,93 133 GHz fsb, Quad Core hyper-threaded 8 Mb cache CPU and 12Gb RAM. The GPU subsystem is enforced by one nVIDIA Quadro FX5800 4Gb RAM video card and two nVIDIA Tesla C1060 4 Gb RAM summing up 720 CUDA cores. The testing system has been built on top of the Fedora 12 Linux operating system, the nVIDIA CUDA Driver, and the SDK/Toolkit version 2.3. We have also compared and contrasted the commercial VMware hypervisor and the open source KVM/QEMU. In fact, the aim of our tests is twofold: to stress the reliability of the CPU/GPU virtual system and to gather quantitative information on performance. The first typology of tests is targeted to verify whether our software stack works in a stable and fully transparent way respect to the software system provided by the nVIDIA CUDA SDK. Each program has been set up in order to execute the same algorithm on both CPU and GPU and check the differences between the results of the two executions. All tests have shown that the gVirtuS stack does not affect the accuracy of the numerical results. The performance tests have been developed in order to assess the efficiency of the virtualized systems under different gVirtuS stacks conditions. We compared the efficiency (elapsed time) of the execution of some selected algorithm by varying


383

Table 1. Averaged results summary of ScalarProd, MatrixMul and Histogram in our test cases (percent respect Host/cpu)

host/gpu kvm/cpu vmware/cpu host/tcp kvm/tcp vmware/tcp host/afunix kvm/vmsocket vmware/vmci

histogram matrixMul scalarProd 9.51 9.24 8.37 105.57 99.48 106.75 103.63 105.34 106.58 67.07 52.73 40.87 67.54 50.43 42.95 67.73 50.37 41.54 11.72 16.73 9.06 15.23 31.21 10.33 28.38 52.66 18.03

both the combinations of hypervisors (no hypervisor, VMware, KVM-QEMU) and communication channels (TCP/IP, VMCI and vmSocket), and the size of the input data. The benchmark software is the implementation of three basic numerical algorithms: ScalarProd, MatrixMul and Histogram (Table 1). ScalarProd computes k scalar products of two real vectors of length m. Notice that each product is executed by a CUDA thread on the GPU so no synchronization is required. MatrixMul computes a matrix multiplication. The matrices are m × n and n × p, respectively. It partitions the input matrices in blocks and associates a CUDA thread to each block. As in the previous case, there is no need of synchronization. Histogram returns the histogram of a set of m uniformly distributed real random numbers in 64 bins. The set is distributed among the CUDA threads each computing a local histogram. The final result is obtained through synchronization and reduction techniques. The benchmark results are syntetically shown in the Table 1 Fig. 2 shows the results of the ScalarProd program. In the test process, the input size of the problem increases by jointly varying k in the range [28 , 212 ] and m in [213 , 216 ]. We performed each experiment using both host/cpu and host/gpu setups as reference. The first experiment returns the execution time when only the CPU on the real machine is used; the second experiment returns the execution time when the GPU with the standard CUDA Tools/SDK is used. As the problem size grows exponentially in the SalarProd test, the CPU time increases following the same exponential trend. Notice that, as expected given the scale of the input size, when the GPU is used the execution time is reduced by three orders of magnitude. The first gVirtuS experiment is the host/afunix, i.e our GPU virtualization stack is used in an un-virtualized environment. Both the front end and the back end run on the same real machine and the communicator is implemented through Unix sockets. This test estimates the virtualization stack overhead without the virtualization itself, that is just considering the performance of the packed CUDA function calls. In the top chart as it is shown in Fig. 2, the difference between host/gpu and host/afunix is not noticeable

384

G. Giunta et al.

Fig. 2. Results of the ScalarProd performance test. Linear y-scale (top) and logarithmic y-scale (bottom).

on the linear scale (top), and the overhead is emphasized by the logarithmic scale (bottom). The problem size cut off value is near 29 × 21 3; for a larger size the overhead introduced by the virtualization stack, without considering the communication between virtual and real machines, can be neglected. The experiment host/tcp measures the impact of the standard tcp/ip stack communicator component. The communication between the front end and the back end exhibits low performance when compared to the results of the other communicators. Nevertheless, it is worth noting that the problem size cut occurs near 211 × 215 ; for a larger size the GPU execution time using gVirtuS appears to be lower than the CPU time. The VMware/cpu and kvm/cpu experiments involve the virtualization. As reported above, we have used two different hypervisors in order to compare and contrast commercial vs open source virtualization solutions. To summarize, the VMware/cpu and kvm/cpu experiments demonstrate that there is no appreciable loss of efficiency in using virtualization with both hypervisors, as long as the computing time is concerned. The VMware/tcp and the kvm/tcp experiments describe the performance results obtained with a tcp based communicator; notice that in both cases the communication is the bottleneck. When the problem size is greater than the cut off value (211 x215 , Fig. 2), a deployment scenario where CPU based computing elements (CE) are connected to GPU cores provided CEs seems to be feasible. The VMware/vmci experiment reports the ScalarProd performance when the GPU is used through gVirtuS


385

Fig. 3. Results of the MatrixMul performance test. Linear y-scale (top) and logarithmic y-scale (bottom)

with a communicator based on the VMCI using the VMware hypervisor. The experiment defined as kvm/vmsocket is referred to the performance evaluation of our vmSocket based communicator running a fully open source setup relying on the KVM/QEMU hypervisor. The difference in performance between the two setups can be assessed by looking at the results shown in the bottom side of Fig. 2. The best performance is achieved in the kvm/vmsocket experiment, that exhibits an efficiency close to that obtained by host/afunix. From the latter result we can reasonably conclude that, on this test problem, gVirtuS has an overhead near to the minimum and it shows the best virtual/real machine communication performance. Notice that the problem size cut off value is 211 x215 and when the size increases further both VMware/vmci and kvm/vmsocket perform in a very similar way. Fig. 3 shows the results of the MatrixMul program. The size of the test problems increases from 24 × 25 ∗ 25 × 26 (i.e. m = 24 , n = 25 , p = 26 ) to 28 × 29 ∗ 29 × 210 (i.e. m = 28 , n = 29 , p = 210 ). The host, VMware, kvm/cpu experiments have shown similar performance results. We note that the host, VMware, kvm/tpc experiments give evidence of a cut off size of 27 × 28 ∗ 28 × 29 . Because of the very high efficiency of the matrix multiplication algorithm on the GPU, the use of the logarithm scale is needed in order to emphasize the behavior of VMware/vmci vs kvm/vmsocket. The bottom side of Fig. 3 shows that the kvm/vmsocket executes faster than VMware/vmci, confirming the conclusion drawn in the previous ScalProd test. Finally, Fig. 4 shows the results of

386

G. Giunta et al.

Fig. 4. Results of the Histogram performance test. Linear y-scale (top) and logarithmic y-scale (bottom).

the Hystogram program. This algorithm differs from the previous ones because it needs a synchronization and a reduction step among the concurrent threads in order to compute the final result. Even this benchmark program confirms that gVirtuS-kvm/vmsocket gives the best efficiency with the less impact respect to the raw host/gpu setup.

5

High Performance Computing Cloud Deployment Performance Evaluation

We have designed and implemented the gVirtuS GPU virtualization system with the primary goal of using GPUs in a private computing cloud for high performance scientific computing. We have set up a department prototypal cloud computing system leveraging on the Eucalyptus open source software. Eucalyptus, developed at the University of Santa Barbara in California [18], implements an Infrastructure as a Service (IaaS), as it is commonly referred to. Eucalyptus enables users familiar with existing Grid and HPC systems to explore new cloud computing functionalities while maintaining access to existing, familiar application development software and Grid middleware[17]. In our test aimed at assessing the performance of gVirtuS in a high performance computing cloud system, we have used an Intel based 12 computing nodes cluster, where each


387

Fig. 5. The high performance computing cloud deployment experiment

node is equipped with a quad core 64 bit CPU and an nVIDIA GeForce GT 9400 video card with 16 CUDA cores and a memory of 1 Gbyte. The software stack as in the Tesla based workstation, i.e. Fedora 12 Linux operating system, nVIDIA CUDA drivers and Tools/SDK 2.3, and KVM-QEMU, was adopted. Moreover, we carried out an experiment focused on the gVirtuS behavior in a private cloud, where the GPUs are seen as virtual computing nodes building a virtual cluster dynamically deployed on a private cloud [12]. We developed an ad hoc benchmark software implementing a matrix multiplication algorithms. This software uses a classic memory distributed parallel approach. The first matrix is distributed by rows, the second one by columns, and each process has to perform a local matrix multiplication. MPICH 2[10] is message passing interface among processes, whereas each process uses the CUDA library to perform the local matrix multiplication. Fig. 5 shows the results of the performance test. We have used virtual clusters composed of 2,4 and 8 nodes computing nodes, respectively. The problem size increases in the range [2E 3 × 2E 3 ∗ 2E 3 × 2E 3 , 4E 3 × 4E 3 ∗ 4E 3 × 4E 3 ], i.e. involving square matrices whose order varies from 2 × 103 to 4 × 103 . The top side of Fig. 5 presents results obtained by running the MPI-based algorithm on the cluster. The bottom side of Fig. 5 gives a comprehensive summary of the tests of gVirtuS-based algorithm for the case CPU only and the case CUDA-GPU. These results show that the gVirtuS GPU virtualization and the related sharing system allows an effective exploitation of the computing power of the GPUs. We note that without such component the

388

G. Giunta et al.

Fig. 6. The high performance computing cloud deployment and GPU virtualization/sharing schema

GPUs could not be seen by the virtual machine and it would be imposssible to run this experiment on a public or private cloud hosting an dynamic (on demand) virtual cluster. The Fig. 6 shows a possible deployment schema in a high performance cloud computing scenario. Virtual machines on computing nodes can use virtualized CUDA acceleration without any regards respect the fact the real machine is equipped or not with a CUDA device. Less GPU performance demanding VMs can share devices available on machines connected with a high performance local network. In a similar manner a high performance cloud computing infrastructure could be build on classic HPC clusters with some nodes dedicated to massive GPU acceleration with many CUDA enabled devices.

6

Comparison to Related Works

The GViM (GPU-accelerated Virtual Machines) [14] is a XEN-based system that provides some features resembling those of gVirtuS. GViM allows virtual machines to access a GPU through a communicator channel between a front end executed on the VM and a back end on the XEN Domain 0. vCUDA [19] is another GPGPU computing solution for virtual machines. It allows an application which is executed within a virtual machine to be accelerated by CUDA cores provided hardware. gVirtuS, GViM, and vCUDA use a similar approach based on the API call interception and redirection through a front end/back end communication system. This is a common choice in virtualization. The main difference lies in the use of the hypervisor and the communication technique. Both GViM, and vCUDA make use of XEN as hypervisor that executes CUDA drivers on the Domain 0. GViM uses the XenStore component to provide communication between the two faces of the CUDA split driver. The performance exhibited by GViM is mainly affected by the XenStore behavior, and in particular by its memory copy operation, as shown in [14]. vCUDA uses a XML-RPC to envelope the CUDA API calls and sends it out the virtual machine through


389

the frontend/backend architecture. This is a standard and commonly used approach, but it suffers a serious drawback in performance as demonstrated by the tests presented in [19]. Our solution differs from the latter two solutions in the chosen open source hypervisor, i.e. KVM; KVM is not a modified kernel, but a Linux loadable kernel module included in mainline Linux. This ensures future compatibility between the hypervisor and the CUDA drivers. Moreover, gVirtuS is completely independent of the hypervisor as we have proved by comparing and contrasting gVirtuS in a VMware setup and in a KVM setup. We also note that in presence of official nVIDIA CUDA drivers for XEN, it could work even on this hypervisor. Finally, gVirtuS relies on an actual high performance communication channel between the virtual and the real machines, thanks to our mvSocket component. Last but not least, the communicator interface is open and new communicators could be developed and plugged in. A direct compare and contrast among gVirtuS and other related systems performances will done as well the other systems will be released for public use. We also emphasize that neither GViM nor vCUDA are able to use a TCP based communicator for the deployment on asymmetric computing clusters. VMGL [15] is a XEN-based technology, GPGPU independent, aimed at using OpenGL on virtual machine through a specific management and monitor system (VMM). VMGL is dedicated to high performance visual computing and not to general purpose high performance scientific computing on GPUs, so it cannot be used to run CUDA applications on virtual machines. Even though the current version of gVirtuS does not provide CUDA-OpenGL interoperability, the two projects could be joined in order to fix this embarrassing gap.

7

Conclusions and Future Development

In this paper we have proposed the gVirtuS system, a GPU virtualization and sharing service. The main goal was to enable a virtual machine instance running in a high performance computing cloud to properly exploit the computing power of the nVIDIA CUDA system. In discussing the architecture of our framework, the adopted design and implementation solutions, and the key communication component vmSocket, we stressed the main features of gVirtuS, i.e. hypervisor independence, fully transparent behavior and especially its performance. We have reported the results of an extensive test process to assess how gVirtuS performs in different and realistic setups. We have also shown that our system can effectively operate even in a high performance computing cloud on a standard cluster powered by nVIDIA GPUs. Finally, gVirtuS has been successfully compared to other existing virtualization systems for GPUs. gVirtuS leverages on our previous works on high performance computing grids, and our interest in the apparently poor performing TCP communicator is related to other more complex deployment schemes. For example a private cloud could be set up on a massive multi core cluster, where to host general purpose virtual machine instances and GPU powered computing elements for compute-intensive scientific applications. A short term enhancement can be the implementation of OpenGL interoperability to integrate gVirtuS and VMGL for 3D graphics virtualization. Since the

390

G. Giunta et al.

vmSocket component of gVirtuS is able to allow very general connections, we are also investigating the possibility to integrate MPICH2 with vmSocket in order to implement a high performance message passing standard interface. In our vision, a vmSocket enhanced MPI could communicate with other virtual machine instances running on the same host, and increase the communication speed between virtual machines on different clusters using a suitable compression. This could foster the use of cloud technologies in scientific computing and transform the concept of e(lectronic)-science into the more practical and effective tool of e(lastic)e-science.

References 1. 2. 3. 4. 5. 6. 7. 8.

9.

10.

11.

12. 13. 14.

15. 16.

Gvirtus website, http://osl.uniparthenope.it/projects/gvirtus Kvm website, http://www.linux-kvm.org Nvidia cuda website, http://www.nVIDIA.com/object/cuda_home_new.html Vmchannel website, http://www.linux-kvm.org/page/VMchannel_Requirements Vmci website, http://pubs.VMware.com/vmci-sdk/ Vmware website, http://www.VMware.com/ Xen website, http://www.xen.org/ Armbrust, M., Fox, A., Griffith, R., Joseph, A.D., Katz, R.H., Konwinski, A., Lee, G., Patterson, D.A., Rabkin, A., Stoica, I., Zaharia, M.: Above the clouds: A berkeley view of cloud computing. Technical Report UCB/EECS-2009-28, Electrical Engineering and Computer Sciences University of California at Berkeley (February 2009) Barham, P., Dragovic, B., Fraser, K., Hand, S., Harris, T., Ho, A., Neugebauer, R., Pratt, I., Warfield, A.: Xen and the art of virtualization. In: SOSP 2003: Proceedings of the Nineteenth ACM Symposium on Operating Systems Principles, pp. 164–177 (2003) Buntinas, D., Mercier, G., Gropp, W.: Design and evaluation of nemesis, a scalable, low-latency, message-passing communication subsystem. In: CCGRID 2006: Proceedings of the Sixth IEEE International Symposium on Cluster Computing and the Grid, Washington, DC, USA, pp. 521–530. IEEE Computer Society, Los Alamitos (2006) Che, S., Li, J., Sheaffer, J.W., Skadron, K., Lach, J.: Accelerating ComputeIntensive Applications with GPUs and FPGAs. In: Symposium on Application Specific Processors, SASP 2008, pp. 101–107 (2008) Fenn, M., Murphy, M.A., Goasguen, S.: A study of a kvm-based cluster for grid computing. In: ACM Southeast Regional Conference (2009) Foster, I., Zhao, Y., Raicu, I., Lu, S.: Cloud computing and grid computing 360degree compared Gupta, V., Gavrilovska, A., Schwan, K., Kharche, H., Tolia, N., Talwar, V., Ranganathan, P.: Gvim: Gpu-accelerated virtual machines. In: HPCVirt 2009: Proceedings of the 3rd ACM Workshop on System-level Virtualization for High Performance Computing, pp. 17–24. ACM, New York (2009) Andrs Lagar-cavilla, H., Satyanarayanan, M.: Vmm-independent graphics acceleration. In: Proceedings of VEE 2007. ACM Press, New York (2007) Tim Grance Mell, P.: The nist definition of cloud computing. Technical Report Version 15, National Institute of Standards and Technology (July 2009)


391

17. Nurmi, D., Wolski, R., Grzegorczyk, C., Obertelli, G., Soman, S., Youseff, L., Zagorodnov, D.: Eucalyptus: an open-source cloud computing infrastructure. Journal of Physics: Conference Series 180(1), 012051 (2009) 18. Nurmi, D., Wolski, R., Grzegorczyk, C., Obertelli, G., Soman, S., Youseff, L., Zagorodnov, D.: The eucalyptus open-source cloud-computing system. In: IEEE International Symposium on Cluster Computing and the Grid, pp. 124–131 (2009) 19. Shi, L., Chen, H., Sun, J.: vcuda: Gpu accelerated high performance computing in virtual machines. In: IPDPS 2009: Proceedings of the 2009 IEEE International Symposium on Parallel&Distributed Processing, Washington, DC, USA, pp. 1–11. IEEE Computer Society, Los Alamitos (2009) 20. Tarditi, D., Puri, S., Oglesby, J.: Accelerator: using data parallelism to program gpus for general-purpose uses. In: ASPLOS-XII: Proceedings of the 12th International Conference on Architectural Support for Programming Languages and Operating Systems, pp. 325–335. ACM, New York (2006) 21. Wang, J., Wright, K.l., Gopalan, K.: Xenloop: a transparent high performance inter-vm network loopback. In: Proceedings of the 17th International Symposium on High Performance Distributed Computing (HPDC 2008), pp. 109–118 (2008)

What Is the Price of Simplicity? A Cross-Platform Evaluation of the SAGA API Mathijs den Burger1, Ceriel Jacobs1 , Thilo Kielmann1 , Andre Merzky2 , Ole Weidner2 , and Hartmut Kaiser2 1

Vrije Universiteit Amsterdam {mathijs,ceriel,kielmann}@cs.vu.nl 2 Louisiana State University {andre,oweidner,hkaiser}@cct.lsu.edu

Abstract. The abundance of middleware to access grids and clouds and their often complex APIs hinders ease of programming and portability. The Open Grid Forum (OGF) has therefore initiated the development and standardization of SAGA: a Simple API for Grid Applications. SAGA provides a simple yet powerful API with high-level constructs that abstract from the details of the underlying infrastructure. In this paper we investigate the price that possibly comes with such an API. We discuss the effects on expressiveness and ease of programming, and analyze the performance overhead of three different SAGA implementations (written in Java, Python, and C++) on various middleware. We conclude that SAGA is a good pragmatic approach to make grids easily accessible. The API considerably improves usability and uniformity, but offers a compromise between expressiveness and runtime dependencies. The overall performance of the tested implementations is acceptable, but the strict API semantics require various runtime checks that occasionally cause significant overhead, depending on the underlying infrastructure.

1

Introduction

The amount of different middleware to harvest the computational power and storage capacity provided by grids and clouds has exploded into the face of their users. The multiplicity and complexity of all the available APIs hinders ease of programming and portability of applications. After all, average users of these systems are not programming experts with detailed technical knowledge, but scientists that simply want to run their applications faster and/or analyze larger problems. Learning a certain grid middleware API requires a considerable effort for them. Once mastered, the users are still tied to that specific middleware API, which may change completely with the next middleware release, their next cluster or when collaborating with other grids. The remedy for this situation lies in a simple yet powerful API for grids and clouds, with high-level constructs that shield programmers of scientific applications from the diversity and complexity of these environments. The Open Grid Forum (OGF) has initiated the development and standardization of such an API, called the Simple API for Grid Applications (SAGA). P. D’Ambra, M. Guarracino, and D. Talia (Eds.): Euro-Par 2010, Part I, LNCS 6271, pp. 392–404, 2010. c Springer-Verlag Berlin Heidelberg 2010

What Is the Price of Simplicity?

393

Version 1.0 of the SAGA specification is based on a set of use cases [12] and a requirements analysis [11], and was released in January 2008 [6]. Since then, various SAGA implementations have been developed in several programming languages on top of various middleware. The experience gained from these implementations is currently used to refine the SAGA API. The SAGA API is defined in a language-independent manner. Each implementation language requires a language binding that defines what SAGA looks like in that language. A SAGA implementation can be built on top of one specific middleware package, or on top of multiple middleware packages simultaneously. The SAGA implementations examined in this paper fall in the latter category, and allow an application to access many different backends using the same API. Like every high-level API, SAGA abstracts from lower-level complexity. Such an abstraction may come with a certain cost: possible loss of expressiveness, performance overhead, etc. The goal of this paper is to investigate at what price the SAGA API and its implementations offer a simpler programming interface for various middleware. The remainder of this paper is organized as follows. Section 2 starts with an overview of the SAGA API and then discusses its advantages and disadvantages. In Sect. 3, we describe three SAGA implementations, discuss their design, and evaluate their performance overhead. Finally, we draw our conclusions in Sect. 4.

2

The SAGA API

SAGA provides an extensible, object-oriented API framework. It consists of a look-and-feel part and an extensible set of functional packages. The look-and-feel consists of the following parts: Base object:

provides all SAGA objects with a unique identifier, and associates session and shallow-copy semantics. Session object: isolates independent sets of SAGA objects from each other. Context object: contains security information for middleware. A session can contain multiple contexts. URL object: provides uniform naming of (possibly remote) jobs, files, services etc. I/O buffer: provides unified access to data in memory, either managed by the application or by the SAGA implementation. Error handling: uses error codes or exceptions (whatever maps best to the implementation language) Monitoring: provides callback functions for events in certain SAGA objects (e.g., job state changes). Task model: allows both synchronous and asynchronous execution of methods and object creation. Permission model: lets an application allow or deny certain operations on SAGA objects.

394

M. den Burger et al.

Orthogonal to the look-and-feel are the functional packages, providing the actual functionality of the underlying distributed system. Currently, the set of standardized functional packages consists of: Job: Namespace: File: Replica: Streams: RPC:

runs and controls jobs. manipulates entries in an abstract hierarchical name space. provides file access. manages replicated files. provides network communication. allows inter-process communication.

In cloud terms, SAGA represents the IaaS model. The job, namespace, and file package support job submission and file access on reserved cloud resources (e.g. via Amazon EC2 or SSH). A package for the selection and reservation of resources is ongoing work. The complete, language-independent specification of the SAGA API can be found in [6]. 2.1

Discussion

We will first describe the advantages of the SAGA API, and then discuss its disadvantages. The three main advantages of the SAGA API are: Simplicity: the SAGA API is much simpler to use than most middleware APIs. It does not require detailed knowledge of the underlying protocols, but offers a clear set of objects and high-level operations. Programming a grid application is therefore much easier and also comprehensible for non-technical users. Uniformity: each functional package of the SAGA API is the same for all middleware. For example, file access via GridFTP [1] looks exactly the same as file access via SSH. The same holds for job submission, name space manipulations, etc. The learning curve to access new middleware is therefore removed, and switching middleware (e.g., to boost performance) is trivial. SAGA applications are also very easy to port. Moreover, the SAGA API is very similar across programming languages. The specification of the SAGA API is language independent, and each language binding offers a very similar set of concepts, objects, method names, etc. Once having learned how to use SAGA in programming language A, switching to language B is fairly easy. Expressiveness: a SAGA implementation often adds functionality that is not supported natively by the middleware or not working in practice. For example, SAGA offers callback methods that notify an application about the status of a submitted job. Various middleware does not provide such callbacks, or the mechanism does not work in practice because of firewalls. A SAGA implementation can then fall back to polling, which may be less efficient but at least works. Other examples are the caching of Globus connections to avoid repeated authentication [15], or the use of Condor’s glide-in mechanism for job submission [4]. In general, a SAGA implementation can add any workarounds or performance optimizations available for certain middleware.


395

We demonstrate the simplicity and uniformity of the SAGA API with example Java code to make a remote copy of a single file. Figure 1 shows the 33 lines of code needed to do this with Globus GridFTP. Figure 2 shows the over 31 lines of code that do the same using the SSH Trilead library [16]. Both examples are reasonably complex, and require detailed knowledge of the underlying middleware (e.g., whether to use active or passive mode in FTP, or that the stdout and stderr of a remotely executed command have to be read away to prevent deadlocks). Even worse, both examples are completely different. Using SSH instead of GridFTP to transfer files requires a user to learn a completely new library, while the essential operation remains exactly the same.

1 2 3 4 5 6

String host = "host.example.com"; int port = 2811; // default GridFTP port String path = "/home/john"; String src = "file.dat"; String dst = "newfile.dat"; GridFTPClient c1, c2;

7 8 9 10 11

try { GSSManager manager = ExtendedGSSManager.getInstance(); GSSCredential credential = manager.createCredential(GSSCredential.INITIATE_AND_ACCEPT);

12 13 14 15

GridFTPClient c1 = new GridFTPClient(host, port); c1.authenticate(credential); c1.setType(GridFTPSession.TYPE_IMAGE);

16 17 18 19

GridFTPClient c2 = createClient(host, port); c2.authenticate(credential); c2.setType(GridFTPSession.TYPE_IMAGE);

20 21 22

HostPort hp = c2.setPassive(); c1.setActive(hp);

23 24 25

c1.changeDir(path); c2.changeDir(path);

26 27 28 29 30 31 32 33

c1.transfer(src, c2, dst, true, null); } catch (Exception e) { e.printStackTrace(); } finally { if (c2 != null) c2.close(); if (c1 != null) c1.close(); }

Fig. 1. File copy example using the Globus GridFTP API

In contrast, Fig. 3 shows how a remote file can be copied via Globus GridFTP using the SAGA API. This example contains only 11 lines of code that just express the high-level file copy operation. To copy the file via SSH instead of

396

1 2 3 4 5


String host = "host.example.com"; int port = 22; // default SSH port String path = "/home/john"; String src = "file.dat"; String dst = "newfile.dat";

6 7 8 9 10

try { HostKeyVerifier v = new HostKeyVerifier(false, true, true); Connection c = Connecter.getConnection(host, port, v, true); Session s = connection.openSession();

11 12

s.execCommand("cp " + path + "/" + src + " " + path + "/" + dst);

13 14 15

InputStream stdout = new StreamGobbler(s.getStdout()); InputStream stderr = new StreamGobbler(s.getStderr());

16 17

String error = readStreams(stdout, stderr);

18 19 20 21 22 23

while (s.getExitStatus() == null) { Thread.sleep(500); } int exitValue = s.getExitStatus(); s.close();

24 25 26 27 28 29 30 31

if (exitValue != 0 || (error.length() != 0 && !error.startsWith("Warning:"))) { System.err.println("Copy failed: " + error); } } catch (Exception e) { e.printStackTrace(); }

Fig. 2. File copy example using the Trilead SSH library. The method readStreams() reads away stdout and stderr in two separate threads and returns the stderr output, but its implementation is omitted to make this example fit on a single page.

Globus, a user would only have to edit the dir variable at line 1 and change the URL scheme from globus:// to ssh://. However, we also experienced several disadvantages of the SAGA API: Runtime dependencies: many middleware systems do not offer all functionality exposed by the SAGA API. Sometimes, SAGA can add such missing functionality itself (one of its advantages), but such workarounds are often simply not possible. For example, if the method job.suspend() is invoked for a job submission system that cannot suspend jobs, a SAGA implementation can only throw a NotImplemented exception. Such a limitation will only be revealed at runtime. To avoid runtime dependencies entirely, the functionality of the SAGA API would have to be confined to the smallest common denominator of all middleware functionality. SAGA’s expressiveness would then be severely limited. Instead, the


1 2 3

397

URL dir = URLFactory.createURL("globus://host.example.com/home/john"); URL src = URLFactory.createURL("file.dat"); URL dst = URLFactory.createURL("newfile.dat");

4 5 6 7 8 9 10 11

try { NSDirectory d = NSFactory.createNSDirectory(dir); d.copy(src, dst); d.close(); } catch (SagaException e) { e.printStackTrace(); }

Fig. 3. File copy example using the SAGA API

API now contains various operations that can only be implemented on a limited number of middleware systems. Examples include extended I/O for files (only available in GridFTP), suspension and restarting of jobs (often not possible), and the creation of symbolic links to name space entries (not present in some backends and programming languages). Finding out which SAGA functionality does work on which middleware requires either detailed knowledge of the underlying middleware (something SAGA explicitly tries to avoid), detailed documentation read by users (which is possible but unlikely), or some trial and error and the interpretation of NotImplemented exceptions. Limited expressiveness: some middleware may offer functionality that cannot be expressed in the SAGA API. For example, SAGA currently does not support the reservations of resources to run jobs on (i.e., nodes or VM images). The SAGA community actively develops new functional packages for the most prominent omissions, but will always lag behind the latest greatest features of specific middleware. Strict semantics can limit performance: the strict semantics of the SAGA API sometimes hinder an efficient implementation. For example, SAGA currently only supports random file I/O, while streaming I/O can be much more efficient in some cases. Another example is the method file.copy(target, flags) that copies a file to another location. The SAGA specification [6] prescribes: – If the target is a directory, the source entry is copied into that directory. – A BadParameter exception is thrown if the source is a directory and the Recursive flag is not set, or the source is not a directory and that flag is set. – If the target lies in a non-existing part of the name space, a DoesNotExist exception is thrown, unless the CreateParents flag is given - then that part of the name space must be created. – If the target already exists, it will be overwritten if the Overwrite flag is set, otherwise an AlreadyExists exception is thrown.

398


– If a directory is to be copied recursively, but the target exists and is neither a directory nor a link to a directory, an AlreadyExists exception is thrown even if the Overwrite flag is set. These API semantics require a SAGA implementation to perform various checks for each invocation of the copy() method: does the target exist, is it a file or directory, does the parent directory exist? Depending on the middleware, the underlying infrastructure, and the number of method calls, such checks can be fairly expensive. Yet at application level such checks can be unnecessary, e.g., because the target directory was just created and is therefore known to exist and be empty.

3

SAGA Implementations

The SAGA API has been implemented in several programming languages by several independent institutes. An overview of the various implementations can be found at the SAGA home page [14]. In this paper we will examine three SAGA implementations in more detail: the reference implementation in C++ created at Louisiana State University [10] and the Java and Python implementations created at Vrije Universiteit Amsterdam. Both the C++ and the Java SAGA implementation have a plugin-based architecture that was pioneered in SAGA’s predecessor, the JavaGAT [13]. Figure 4 depicts this design. A small dynamic engine provides dynamic call switching of SAGA API calls to middleware bindings (adaptors) which are dynamically loaded on demand and bound at runtime. Each implementation includes several adaptors for various middleware packages. We demonstrate the behavior of an adaptor-based SAGA implementation via the creation of a SAGA file object. The constructor of a file gets a URL with the location of the file, and (optionally) some flags that indicate whether to create a new file, or to overwrite an existing one, etc. The SAGA engine then tries to load all available file adaptors. Many will fail, e.g., because they reject the URL scheme. For example, an SSH adaptor will reject a URL starting with ftp://. The engine remembers the adaptors that succeed, and forwards later method invocations to the first successful one. Later errors trigger the engine to try another adaptor. Only when all adaptors fail, an exception is thrown to the user. Unlike its C++ and Java counterparts, the Python SAGA implementation does not use Python-specific SAGA adaptors to implement the functionality of the various SAGA packages. Instead, it acts as a wrapper on top of a Java SAGA implementation. All SAGA functionality is therefore available via Pythonspecific constructs. Internally, all Python SAGA objects use the Java SAGA language bindings to implement all functionality. The Python SAGA implementation relies on Jython [9], a Python interpreter written in Java. Jython allows a Python application to use Java objects and methods, which makes it relatively easy to implement Python SAGA on top of Java SAGA.


399

Fig. 4. General architecture of the Java and C++ SAGA implementations: a lightweight engine dispatches SAGA calls to dynamically loaded middleware

3.1

Discussion

All three SAGA implementations use or rely on an adaptor-based implementation. The two main advantages of such a design are: Extensibility: to support new middleware, only new adaptors have to be written. These adaptors can then be easily included in a new SAGA release, or even be added dynamically to an existing SAGA installation. Since SAGA is becoming a standard, one could even envision that middleware developers create and maintain SAGA adaptors themselves. Flexibility: a single SAGA application can use multiple middleware simultaneously with almost no additional effort. To use different middleware, the application only has to provide another URL when creating a SAGA object. Some middleware also requires specific credentials via SAGA context objects. Fortunately, SAGA picks up the default credentials automatically (e.g., SSH keys, Globus certificates, etc.), so specific context objects are usually unnecessary. However, an adaptor-based SAGA implementation also has some disadvantages: Error handling is hard: the SAGA engine will try to instantiate all available adaptors for a SAGA object. When all adaptors fail, a compound exception is thrown that contains a list of all exceptions thrown by the individual adaptors. The compound top-level exception is a copy of the exception in the list that the engine deems the ’most relevant’ one. Interpreting such a compound exception is hard, both in code and for humans. Only examining the top-level exception will ignore the errors of the other adaptors, which may be relevant to understand what is going on. It also requires SAGA to correctly choose the most relevant exception. Examining the whole list instead is more secure but tedious, and requires a user to comprehend why every available adaptor failed. The more available adaptors, the harder is becomes to understand all the errors.

400


Performance overhead: SAGA adds an additional layer between the application and the middleware, which always introduces a performance penalty. Every additional adaptor increases the initialization time per SAGA object. 3.2

Performance

To quantify the performance overhead of SAGA, we have created four SAGA applications that benchmark the functional packages job, namespace, and file: – Benchmark 1 runs one long job (the UNIX command /bin/sleep 60 ) and waits until it has finished. – Benchmark 2 runs a sequence of many short jobs (sixty times /bin/sleep 1 ) and waits until they have finished. – Benchmark 3 executes a series of name space operations: 1. create 10 directories (dir000 to dir009 ) 2. in each directory, create 10 subdirectories (subdir000 to subdir009 ) 3. in each sub directory, create 10 empty files (file000 to file009 ) 4. recursively print the name, type, and size of all created entries 5. move all directories dir* to d* 6. copy all files file* to f* 7. recursively remove each directory d* – Benchmark 4 performs a sequence of file operations: 1. create a file foo of 100 MB using 3200 write operations of 32 KB 2. copy file foo to file bar 3. read file bar in 3200 blocks of 32 KB 4. remove foo and bar We have implemented these four benchmarks on top of the Java, Python, and C++ SAGA implementations and use them in two experiments. In the first experiment, we compare the overhead of each SAGA implementation using only the local machine as the back-end of each benchmark. We compare each benchmark against a native implementation that uses the default language constructs to run local jobs and access local files and directories (i.e., the JDK 1.6, the Python 2.x standard library, and standard C++ together with the Boost Filesystem library. The test machine is located at TU Delft, The Netherlands, and is one of the head nodes of the DAS3 grid system [2]. It features a dual-CPU, dual-core 2.4 GHz AMD Opteron DP 280, 4 GB of main memory and a fast RAID6 storage system. We run each benchmark program 10 times and calculate the average running time. Figure 5 shows the results. Running one long job using SAGA takes almost no overhead in all implementations. With many short jobs, the C++ implementation has a slightly larger overhead compared to the Java and Python implementation. The name space benchmark shows a much larger difference between the native and SAGA API, although the overall running time is still quite fast (i.e., less than 3 seconds). The high overhead is caused by all the strict semantics of the SAGA API that requires many additional checks per name space operation. File access speed is comparable in Java and C++, but much slower in Python due to the rather


Native SAGA

60 50 40 30 20 10 0

Java

Python

average running time (sec)


40 30 20 10 Java

3

2 1.5 1 0.5 Java

50

Python

C++

(c) Benchmark 3: name space

Python

C++

(b) Benchmark 2: many short jobs

Native SAGA

2.5

0

60

0

C++

(a) Benchmark 1: one long job 3

Native SAGA

70 average running time (sec)


70

401

Native SAGA

2.5 2 1.5 1 0.5 0

Java

Python

C++

(d) Benchmark 4: file

Fig. 5. Average running times of the four benchmarks using SAGA and the native local constructs in Java, Python, and C++

inefficient way Jython handles binary data. Overall, all SAGA implementations show good performance and only moderate overhead. Only with many small operations (e.g., running many jobs or performing many name space manipulations) the overhead starts to show. In the second experiment, we compare the running time of each SAGA implementation on top of various middleware systems. We also measure the running time of a native Java implementation on top of all middleware to show the overhead of SAGA. We compare local job execution to remote execution using Globus 4.0.3 [3], GridSAM 2.0.1 [7], and SSH. The name space and file benchmarks access the local filesystem and remote volumes via GridFTP [1], SSH and XtreemFS [8]. The local machine in this second experiment is the same as in the first one. All remote machines are located at Vrije Universiteit in Amsterdam, The Netherlands. These two sites are connected by a 1 Gbit link with an RTT of around 3 ms. Figure 6 shows the average running time of 10 repetitions of each benchmark. The overhead of running a single long job is again negligible in all SAGA implementations. Globus needs about 10 seconds to recognize a finished job, which explains its higher baseline in Fig. 6(a). To avoid very long running times, we omitted Globus from the second benchmark with many small jobs. In this case, the C++ SAGA implementation is a bit slower than the Java implementation for local execution and SSH, but faster when using GridSAM. The differences in speed are caused by the different middleware APIs used in the adaptors. For



120

Java Native Java SAGA Python SAGA C++ SAGA

100 80 60 40 20 0

Local

Globus

GridSAM

16 4

Local

60 40 20 Local

1024

64

1

80

GridFTP

SSH

XtreemFS

(c) Benchmark 3: name space

GridSAM

SSH

(b) Benchmark 2: many short jobs



256


100

0

SSH

(a) Benchmark 1: one long job 1024


120 average running time (sec)

402

256


64 16 4 1

Local

GridFTP

SSH

XtreemFS

(d) Benchmark 4: file

Fig. 6. Average running times of the four benchmarks using native Java and the three SAGA implementations on top of various middleware

example, the C++ GridSAM adaptor uses a simpler and faster authentication scheme than the Java GridSAM adaptor. The standard deviation of the running time of most job benchmarks is really small, in the order of 1 second. Only with many small GridSAM jobs the standard deviation gets quite large in Java and Python SAGA. However, this is caused by a memory leak in a security library that is triggered by the (default) authentication mechanism of GridSAM, which significantly slows down the GridSAM server after a while. The simpler authentication mechanism used in C++ SAGA does not trigger this bug, but does require a reconfiguration of the server. The major speed differences between the SAGA implementations in the name space and file benchmarks are caused by the different implementations of the adaptors. Figure 6(c)s and 6(d) use a logarithmic scale on the Y-axis to accommodate the wide variety in running times. Not surprisingly, all middleware systems add a considerable amount of overhead compared to the local benchmarks. Especially GridFTP adds a large amount of latency to each operation, which makes the name space benchmark quite slow in all implementations. In the file benchmark, the performance difference between the C++ implementation and the other ones is caused by the distinct Globus APIs in Java and C++, which come with different performance overheads. The Java GridFTP file


403

adaptor tunes the polling frequency of the GridFTP control channel to minimize the overhead per read and write operation. The C++ GridFTP adaptor relies on callbacks provided by Globus, which slow down each operation. Besides the random I/O SAGA file API, the Java SAGA language bindings also offer streaming I/O since almost all Java I/O is based on streams. We therefore performed the GridFTP file benchmark on top of streaming I/O as well. To our surprise, it then runs an order of magnitude faster than on top of random I/O. We suspect that streaming I/O in GridFTP prevents an explicit acknowledgment of each of the 3200 consecutive read and write operations, which saves an enormous amount of overhead. The speed differences with the SSH benchmarks are caused by the different middleware APIs used in the adaptors. The SSH adaptor in Java SAGA (and hence also Python SAGA on top of it) uses the Trilead SSH library, while the C++ SSH adaptor uses FUSE [5] to mount a remote filesystem locally and accesses it via the local file adaptor. The former approach is apparently faster for many small name space operations, while the latter is faster for file access. All SAGA implementations use FUSE to access XtreemFS volumes, and achieve comparable speed in both the name space and file benchmarks.

4

Conclusions

The SAGA API offers a simple programming interface for existing grid and cloud middleware. Each functional package of the SAGA API is uniform for all middleware, and very similar across programming languages. SAGA therefore greatly enhances portability and lowers the learning curve for actual users significantly. On the downside, the SAGA API introduces runtime dependencies in the form of NotImplemented exceptions. SAGA may also offer less features than the actual middleware itself, although it can add missing functionality as well (e.g., callbacks of job status via automatic polling). The strict semantics of the SAGA API can sometimes cause significant performance overhead, e.g., in case of many small name space operations or random I/O on top of Globus GridFTP. For end-user applications, the performance overhead of the tested SAGA implementations is certainly acceptable, and varies with the number of operations, the middleware used, and the underlying infrastructure. We conclude that SAGA’s major benefit, its simple and uniform API, largely outweighs the price users have to pay for in terms of runtime dependencies and performance overheads. As such, SAGA has become a viable generic API for grid and cloud environments.

Acknowledgements Part of this work was supported by the EU IST program as part of the XtreemOS project (contract FP6-033576). Work by Hartmut Kaiser, Andre Merzky and Ole Weidner has been supported by the UK EPSRC grant number GR/D0766171/1 (via OMII-UK) and HPCOPS NSF-OCI 0710874.

404


References 1. Allcock, W., Bresnahan, J., Kettimuthu, R., Link, M., Dumitrescu, C., Raicu, I., Foster, I.: The Globus Striped GridFTP Framework and Server. In: Proceedings of Supercomputing 2005 (SC 2005) (November 2005) 2. The Distributed ASCI Supercomputer 3 (2006), http://www.cs.vu.nl/das3/ 3. Foster, I.: Globus Toolkit Version 4: Software for Service-Oriented Systems. In: IFIP Int. Conf. on Network and Parallel Computing, pp. 2–13 (2006) 4. Frey, J., Tannenbaum, T., Foster, I., Livny, M., Tuecke, S.: Condor-G: A Computation Management Agent for Multi-Institutional Grids. In: Proceedings of the 10th IEEE Symposium on High Performance Distributed Computing, HPDC10 (August 2001) 5. Filesystem in Userspace (FUSE), http://fuse.sourceforge.net/ 6. Goodale, T., Jha, S., Kaiser, H., Kielmann, T., Kleijer, P., Merzky, A., Shalf, J., Smith, C.: A Simple API for Grid Applications (SAGA). Grid Forum Document GFD.90, Open Grid Forum (OGF) (January 2008) 7. GridSAM, http://www.omii.ac.uk/wiki/GridSAM 8. Hupfeld, F., Cortes, T., Kolbeck, B., Focht, E., Hess, M., Malo, J., Marti, J., Stender, J., Cesario, E.: XtreemFS - A Case for Object-based File Systems in Grids. Concurrency and Computation: Practice and Experience 20 (June 2008) 9. The Jython Project, http://www.jython.org 10. Kaiser, H., Merzky, A., Hirmer, S., Allen, G.: The SAGA C++ Reference Implementation. In: 2nd Int. Workshop on Library-Centric Software Design, LCSD 2006 (2006) 11. Merzky, A., Jha, S.: A Requirements Analysis for a Simple API for Grid Applications. Grid Forum Document GFD.71, Global Grid Forum (GGF) (May 2006) 12. Merzky, A., Jha, S.: Simple API for Grid Applications - Use Case Document. Grid Forum Document GFD.70, Global Grid Forum (GGF) (March 2006) 13. van Nieuwpoort, R.V., Kielmann, T., Bal, H.E.: User-Friendly and Reliable Grid Computing Based on Imperfect Middleware. In: Proceedings of the ACM/IEEE Conference on Supercomputing (SC 2007) (November 2007) 14. SAGA Home Page, http://saga.cct.lsu.edu/ 15. Thain, D., Moretti, C.: Efficient Access to Many Small Files in a Filesystem for Grid Computing. In: Proceedings of the 8th IEEE/ACM International Conference on Grid Computing, pp. 243–250 (September 2007) 16. Trillead SSH Library for Java and .NET, http://www.trilead.com/SSH_Library/

User-Centric, Heuristic Optimization of Service Composition in Clouds Kevin Kofler, Irfan ul Haq, and Erich Schikuta Department of Knowledge and Business Engineering University of Vienna, Austria [email protected], {irfan.ul.haq,erich.schikuta}@univie.ac.at

Abstract. With the advent of Cloud computing, there is a high potential for third-party solution providers such as composite service providers, aggregators or resellers to tie together services from different clouds to fulfill the pay-per-use demands of their customers. Customer satisfaction which is primarily based on the fulfillment of user-centric objectives is a crucial success factor to excel in such a service market. The clients’ requirements, if they change over time even after the desired solution composition, may result in a failure of this approach. On the other hand, business prospects expand with the possibility of reselling already designed solutions to different customers after the underlying services become available again. The service composition strategies must cope with the above-mentioned dynamic situations. In this paper we address these challenges in context with the customerdriven service selection. We present a formal approach to map customer requirements onto functional and non-functional attributes of the services. We define a happiness measure to guarantee user satisfaction and devise a parallelizable service composition algorithm to maximize this happiness measure. We devise a heuristic approach based on historical information of service composition to rapidly react to changes in client requirements at design time and indicate run-time remedies such as for service failures. The heuristic algorithm is also useful to recompose similar solutions for different clients with matching requirements. Our algorithms are evaluated by the results of a simulation developed on the workflow tool Kepler coupled with a C++ implementation of the optimization algorithms.

1

Introduction

In this paper, we pursue our vision of IT-based service markets, which opens up doors for totally new business processes for consumers and producers. So in the near future it will be a common practice to sell IT resources as services and not as goods. For example, “Writing a letter” can be as simple as using a telephone: Forget buying software and hardware! All we need is a simple interface to the services on the Internet, both the word processor functionality and the necessary physical resources (processor cycles and storage space); and everything is paid transparently via our telephone bill. P. D’Ambra, M. Guarracino, and D. Talia (Eds.): Euro-Par 2010, Part I, LNCS 6271, pp. 405–417, 2010. c Springer-Verlag Berlin Heidelberg 2010

406

K. Kofler, I. ul Haq, and E. Schikuta

Cloud Computing, building on the notions of Software as a Service (SaaS), Platform as a Service (PaaS) and Infrastructure as a Service (IaaS) strives for a similar vision to provide a platform for future free markets of IT resources. Marked as the most popular emerging technology of 2009 on the Gartner Hype Cycle [3], Cloud Computing attracts both service providers and consumers alike with its promise of cost reduction based on the pay-per-use model and the shift from the usual capital upfront investment model to an operational expense [2]. There is a high potential for third-party solution providers such as Composite Service Providers (CSPs) [7], aggregators or resellers to tie together services from different clouds [6] to fulfill the pay-per-use demands of their customers. Several CSPs, if joined together in a hierarchical fashion, will lead to service supply chains with a service composition process taking place at every level of the chain. Such compositions will result in new business processes underpinned by new business networks. The pay-per-use service models will not only help in the reduction of cost but various services, after being composed together with the help of orchestration technology, will become available again as more capable composite services with a guaranteed level of service. Customer satisfaction, which is primarily based on the fulfillment of usercentric objectives, is a crucial success factor to excel in such a service market[4]. Consumers will be able to access services from the Cloud under their desired level of service by mentioning the Quality of Service (QoS) requirements. Consumers and service providers will be bound to these requirements through Service Level Agreements (SLAs). The most convenient way for the end-user to specify its requirements is to first represent them in form of an abstract workflow, which allows a user to draw a sequence of activities representing his desired services along with the user’s functional and nonfunctional requirements. In the next step, the properties of the required services can be expressed in form of SLA templates in a registry similar to [5] where both user requirements and service offerings are represented as SLA templates so that both parties can discover each other. Suitable services satisfying user requirements are then searched from the registry, and mapped on the abstract workflow. The optimal service composition requires the best selection out of the available services. The selection of the services is based on user requirements. These requirements can be functional requirements such as order of the required services or non-functional such as total cost, total response time, availability, reliability and trust etc. Even after a service composition, there can be a need of recomposition, for reasons such as the user changing his requirements or service failures. This is a multi-fold problem and poses several challenges. Automatic adjustment and recovery in such situations demand a strong formal model to realize various interrelated entities contributing to service compositions, such as: the formal and non formal attributes of services, user requirements, and criteria for the service selection. The next challenge requires optimization algorithms to ensure the best design and performance of the composed services. There must be fault tolerance mechanisms that deal with the runtime failure of services. In this paper we present:

User-Centric, Heuristic Optimization of Service Composition in Clouds

407

– a formal model to automate the selection of optimal services fulfilling user requirements – a heuristic algorithm to cope with dynamic changes in user requirements by updating an existing solution – the design of our simulation environment, incorporating the Kepler workflow tool [18], and C++ based optimization components, and – the implementation details and a performance analysis of our model In the formal model, the user requirements are transformed to the QoS parameters of the services. We define a happiness measure based on these QoS parameters to grade the set of candidate services and as a result, the one with the highest happiness measure is selected. The heuristic algorithm keeps track of the best services and utilizes this historical information to remodel the service composition in response to the changing user requirements. In this paper, we have given a special focus to the cost and time requirements but our approach is equally valid for other types of functional and non functional attributes. This paper complements our previous work in this area, where we employ the data access mechanism of the Datagrid project of CERN to define a cost model for data replication [13], design a blackboard approach for dynamic workflow optimization [9] and devise a formal model and a parallelized algorithm for usercentric service compositions [8]. The typical user requirements in the Datagrid project include the bandwidth, disk space, compression algorithms, criteria of replication and the preferred sites for data storage. Our two-phase heuristic algorithm for workflow optimization has been designed to coordinate with our blackboard approach and the cost model. In section 2, we present our formal model for QoS optimization of service composition. Section 3 elaborates the two-phase heuristic algorithm for QoS optimization of service composition, whereas the sections 4 and 5 discuss the implementation details and the performance analysis respectively. Section 6 states the related work and section 7 winds up the paper with the conclusion and future work.

2 2.1

QoS Optimization of Workflows Motivational Scenario

We start with a scenario. Our user Alice wants to use the computing power of the Cloud for a graphical simulation. The simulation needs to be computed by a physics engine based on a mathematical model provided as input, rendered by a graphical engine using graphics and sound libraries, converted to a compressed video file by a codec – all examples of Software as a Service (SaaS) – and saved to a storage provider providing Infrastructure as a Service (IaaS). She formulates her requirements to a Platform as a Service (PaaS) provider in the Cloud through some interface which is out of the scope of this paper. The PaaS provider which Alice contacted and which appeared to her as a single provider is actually just an aggregator which delegates its activities to a physics engine service, a graphical

408


engine service (which may itself be composite, using graphics and sound library services), a codec service and a storage service in form of a workflow. Thus, it builds an abstract workflow of the above four activities and maps the requirements Alice has described onto those tasks. Some of those requirements must be fulfilled strictly, for instance she might require a maximum error of 0.1% in the physics, whereas others such as the desired compression rate of the output video should be satisfied as well as possible but are rather flexible. The aggregator then runs a branch and bound algorithm to search for the services that can map onto different tasks of the workflow, automatically selecting the best services fulfilling Alice’s requirements. A codec service out of many competing peers is selected on the basis of the lowest cost and high degree of compression. Later on, unsatisfied by the quality of output, Alice decides to change some requirements and increases the maximum allowed cost. The automated workflow tool uses a heuristic algorithm to cope with the new user preferences. A few hours later, a new user Bob also wishes to run a similar simulation using the same composite service provider. His requirements are very similar to Alice’s, so instead of running a branch and bound algorithm from scratch, the provider heuristically updates the solution computed for Alice to match Bob’s requirements. 2.2

Formal Model for QoS Optimization of Workflows

The given scenario poses a complex mathematical problem with interrelated issues. We formally define and summarize the relationships among these issues. A more detailed, fully formalized description of our model can be found in [8]. In our model, we denote the set of equivalent services, e.g. the video compression services, as a service class. We assume that each service belongs to exactly one service class. We model QoS parameters such as the compression rate as numeric service attributes, common to all services of a given class, but with different numeric values Q0 for the different offered services. For simplicity, we will assume values in real numbers, i.e. in the set R, this also covers booleans if we identify true as 1 and false as 0. We normalize those values to qualities Q in the range [0, 1], where 0 is the worst possible quality and 1 the best one. The map used for this normalization is the same for all services of a given class, making the normalized values comparable. It can be increasing (for attributes which directly indicate a quality) or decreasing (for attributes such as latencies where less is better). An abstract workflow W0 specifies the requirements the user has for the workflow. It is given by a directed graph whose nodes are the steps in the workflow, and for each node v ∈ V (the set of workflow nodes), the needed service class f0 (v), minimum (“must”) and desired (“should”) requirements for the attributes, which are respectively normalized to vectors Rm (v) ≤ Rd (v) (componentwise) of numbers in [0, 1] using the same map as for the attribute values, and weights w(v) given to the desired requirements, indicating how much value is given to the “should” request. We will not use the graph structure in our algorithm, thus we only consider a list of nodes with the above properties.


409

A concrete workflow W specifies a concrete workflow instance which should match the user’s requirements. We define a concrete workflow as a list of nodes v with a requested service f (v). We say W is sensible for W0 if they have the same number of nodes (or as graphs, the same graph structure) and the service for each node is actually of the requested class for that node. We say W is feasible for W0 if W is sensible for W0 and ∀v ∈ V : Qf (v) ≥ Rm (v) (componentwise), i.e. if all the minimum requirements are satisfied. We define a happiness measure which quantifies how happy the user is with a given workflow considering his/her desired requirements and weights. For all pairs (W0 , W ) where W is sensible for W0 , we define h(W0 , W ) as h(W0 , W ) = hvf0 (v) v∈V

where for each choice of service s for v ms hvs = wl (v)hvsl l=1

(ms being the number of attributes of the service s) and ⎧ ⎨ 0, (Qs )l < (Rm (v))l hvsl = 1, (Qs )l ≥ (Rd (v))l , ⎩ (Qs )l −(Rm (v))l , else (Rd (v))l −(Rm (v))l i.e. 0 for infeasible qualities, 1 for qualities at least as high as desired and linearly increasing between the minimum and the desired requirement. Note that this is a linear happiness measure. We assume it makes sense to define such a linear happiness, which is a requirement on the normalization maps. With the above definitions, we can state the problem in mathematical terms: Given an abstract workflow W0 , we want to find a concrete workflow W which optimizes: max h(W0 , W ) s.t. W is feasible for W0 ≤K v∈V Q0 f (v) 1 ≤T v∈V Q0 f (v) 2

where Q0 f (v) 1 is assumed to be the cost of the service f (v) and Q0 f (v) 2 its execution time, and thus K and T are upper bounds on the total cost and the total execution time, respectively. We will call those additional constraints aggregate constraints, because they are the constraints which aggregate the QoS parameters of the different services, whereas workflow feasibility considers each node individually. Our approach can also handle other similar constraints. It can be easily seen that, after filtering out the services which do not satisfy the minimum requirements, our problem becomes equivalent to a Multidimensional Multi-choice Knapsack Problem (MMKP) [1]: the utilities in the MMKP are our happiness values hvs .

410

3


A Heuristic Algorithm for Optimization of Service Composition

In this section, we will present an algorithm to realize our motivational scenario in two phases: a precomputation phase in which we aim at the QoS-aware optimization of service composition and an updating phase using heuristics to react to dynamic changes in user requirements or reuse solutions for users with similar requirements.

Fig. 1. The two phases of the optimization algorithm

3.1

Phase 1: Precomputation

For the precomputation phase, we use a branch and bound algorithm, one of the most successful, flexible and easy to parallelize algorithms for optimization problems. The input data for our algorithm is composed of a vector V = {v1 , . . . , vp } of nodes in the workflow graph and a vector S of vectors Si = {si1 , . . . , siki } of services (one vector Si for each service class). The entries in the vectors are simple structures: a node has an integer f0 , the service class required for the node in the abstract workflow and vectors Rm , Rd , w, the normalized requirements and weights for each attribute of the service class; a service has vectors Q, its normalized QoS parameters and Q0 , its actual QoS parameters (used for the aggregate constraints). Our solutions are represented as decision vectors d = {d1 , . . . , dp }, each entry dv of which corresponds to the choice of a service for the node v. The details of our branch and bound approach are out of the scope of this paper, they can be found in [8]. 3.2

Phase 2: Dynamic Heuristic Updates

The drawback of the branch and bound approach is that it is expensive. Its worstcase performance can be proven to be exponential. Its practical performance is highly dependent on input data, and in the case of a threaded or distributed implementation, also non-deterministic. (The algorithm is non-deterministic due to thread or process scheduling, in the distributed case also timing of network communication. Only the resulting happiness is deterministic, as the algorithm is guaranteed to find the exact optimum.) Unfortunately, this is not merely an issue with the implementation: the problem is NP-hard, due to the total cost


411

and time constraints, which make it equivalent to a two-dimensional knapsack problem. One of the ways to deal with this scalability problem is to parallelize the implementation. In [8], we have achieved promising results with a CORBAbased distributed implementation, showing nearly linear speedup on sufficiently large testcases, however that is out of the scope of this paper. We will instead focus on techniques fast enough to react to changes to the problem in real time. As explained above, in order to efficiently react to changed user requirements, a different approach is needed, based on heuristics. Our proposed solution is to reuse the existing solution to the original problem as a starting point and update it for the changed requirements. This can be done efficiently (in low-order polynomial time) and generally results in a near-optimal solution to the modified problem. However, this is only a heuristic: the optimality of the obtained solution cannot be guaranteed. Only recomputing everything, which is NP-hard (as discussed above), can guarantee that. Algorithm 1 describes the approach used for the total cost constraint. Exactly the same procedure is also used for the total time constraint, and in principle similar updates (whose details are beyond the scope of this paper) could also be done for other changes in user requirements or service offerings, e.g.: – changes in user’s quality requirements or weights – changes in service parameters – added / removed services. As this procedure is very efficient, it is also be possible to use it for runtime changes, e.g. service failure. In this case, we have to consider the structure of the workflow, as it does not make sense to change a service which already completed or with which a non-refundable SLA has already been agreed to. Thus, one would have to replace the failed service with another service, then make up for cost or time overruns, if any, by replacing the services which are not fixed yet by cheaper resp. faster services. The heuristic update can also be employed to reuse a solution designed for a given client for a new client with similar requirements. This case can be treated just as if the original client changed their requirements. Another possibility worth trying would be to avoid the branch and bound procedure entirely and rely only on the updating heuristics: 1. We compute a solution without the constraints on K and T . This can be done in polynomial time. We use this solution as our starting solution. 2. We update that solution heuristically to honor K and T . This procedure would be significantly faster than branch and bound, but the drawback is that it would no longer be guaranteed to find an optimal solution. Another issue is that the current heuristics may fail entirely, because adjusting for the new value of one constraint can violate the other. Thus, we implemented a failsafe version of algorithm 1 which considers only those alternative services which do not take more time when adjusting for cost and vice-versa. We try algorithm 1 first, then if it fails, the failsafe version. When adjusting for an

412 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35

K. Kofler, I. ul Haq, and E. Schikuta Compute cost of old solution; if K > old K then foreach node do Compute happiness for currently chosen service; foreach service satisfying minimum requirements, happiness hs > old hs and cost > old cost do hs−old hs Compute q = cost−old ; cost end Add best (largest) q to priority queue; end while queue not empty do Pick top queue entry; if new cost ≤ K then apply update; end Check solution; if infeasible then return FAILURE; Output updated solution; else if cost > K then foreach node do Compute happiness for currently chosen service; foreach service satisfying minimum requirements, happiness hs < old hs and cost < old cost do hs−old hs Compute q = cost−old ; cost end Add best (smallest) q to priority queue; end while queue not empty and cost > K do Pick top queue entry; Apply update; end Check solution; if infeasible then return FAILURE; Output updated solution; else Output old solution; end return SUCCESS;

Algorithm 1. Heuristic update for K

increased constraint, this will always lead to a feasible solution which is at least as good as the initial one. For a decreased constraint, it can still fail, in which case our implementation falls back to recomputing a new solution using branch and bound. A completely failsafe approach to maintaining feasibility is NP-hard, just like the original problem, because it amounts to minimizing one constraint while satisfying the other, which is equivalent to a knapsack problem.


413

Fig. 2. Structure of our workflow (screenshot from Kepler)

4

Implementation

To model the different components of our optimizer, we used the workflow tool Kepler [18]. However, there is no easy way to represent a parallel branch and bound algorithm in Kepler, as it can only spawn a fixed amount of threads or processes (one for each component in the Kepler workflow), whereas we would need a way to spawn them dynamically, one for each search graph node. Other workflow tools, such as Triana and Taverna, have this same limitation. Thus, we decided to use an external process for the optimizer core, cooperating with the Kepler workflow. The core is written in C++ using libQtCore [19]. Figure 2 shows the workflow used for our optimization method: there are components for the user interface, a service manager, the optimizer and a report module. In the simulation, we use simple dialogs or files for input. In the real world, we would have a user-friendly graphical user interface and the service manager would communicate with third-party services. Our design also allows for a runtime optimization module which can react to dynamic changes at runtime, but that component is out of the scope of this paper. The first implementation of the optimizer core was based on threads. As threads have high overhead and as they mean effectively leaving the search strategy to the operating system (because the thread scheduler decides which node in the search graph to process next), which knows nothing about the structure of our problem, we also implemented a deterministic sequential version using breadth-first search.

5

Performance Analysis

We compared our branch and bound implementation with the updating heuristics to see how much faster the heuristics are. 5.1

Test Method

As we did not have access to any real-world data, we generated some feasible, but otherwise random, synthetic testcases with a pseudorandom number generator.

414


We did tests with 10, 12, 15, 20, 25 and 30 workflow nodes, each of which was mapped to a different service class. The timings reported for the branch and bound algorithm are from a sequential breadth-first implementation, which proved the most efficient on a single CPU. For each testcase, we ran a series of tests with constant services and user requirements, changing only the values of K and T (one at a time) and we compared the results of the heuristics with repeated runs of branch and bound, looking at both the execution time of the optimization process and the quality of the solution (i.e. how close to the optimum it is). The tests were run on a single-core 2.6 GHz Pentium 4 with HyperThreading disabled. We used the following pairs (K, T ) for each testcase: 10 nodes: (350, 500), (400, 500), (400, 550), (400, 500), (380, 500), (350, 500), 12 nodes: (400, 600), (450, 600), (450, 650), (450, 600), (430, 600), (400, 600), 15 nodes: (550, 700), (600, 700), (600, 750), (600, 700), (580, 700), (550, 700), 20 nodes: (800, 850), (850, 850), (850, 900), (850, 850), (830, 850), (800, 850), 25 nodes: (900, 1200), (950, 1200), (950, 1250), (950, 1200), (930, 1200), (900, 1200), 30 nodes: (950, 1500), (1000, 1500), (980, 1500), (950, 1500).

5.2

Results

Figure 3 shows the results of the performance measurements. Almost invisible bars in the figures mean the value is very small or zero. The results show that the branch and bound algorithm scales up to problem sizes in the order of 30 workflow nodes with acceptable performance, but that the heuristic update is several orders of magnitude faster. Note that the first testcase of each set is the initial solution, which is always computed using branch and bound. We also found that in the testcases with 10 and 12 workflow nodes, the heuristic updates always found the optimum solution. This is not always guaranteed, because the update is only a heuristic. Indeed, for the testcases with 15 or more workflow nodes, the solutions found by the heuristic approach were not always optimal, but they came very close (within 98% of the happiness) to the optimum. Figure 4 shows the ratios between the happiness values for the solutions found by the heuristics and the optimum happiness values as found by branch and bound. We also tried using only the heuristics instead of the branch and bound process, using the solution without constraints for K and T as the starting solution. This turned out to be much faster than branch and bound, which matches the expectations, as the heuristics are polynomial, whereas the branch and bound is exponential. The initial heuristic updates are as fast as the subsequent ones. Unfortunately, this is only preliminary due to the problems described in the previous section: the heuristic update can fail if the update to reduce the value of one constraint makes it exceed the bound for the other one, the “failsafe” version can fail if a more expensive service needs to be picked to reduce the computation time or vice-versa, and there is no guarantee of optimality. However, these problems are inherent to heuristics.

10 workflow nodes

12 workflow nodes

15 workflow nodes

no heuristics

no heuristics

no heuristics

3 2.5 2 1.5 1 0.5 0

80 70 60 50 40 30 20 10 0

testcase

heuristics

700 600 500 400 300 200 100 0

testcase

testcase

20 workflow nodes

25 workflow nodes

30 workflow nodes

no heuristics

no heuristics

no heuristics

computation time me (s)

heuristics

15 1.5 1 0.5 0

heuristics

90 80 70 60 50 40 30 20 10 0


2

heuristics

120 100 80 60 40 20 0

testcase

testcase

testcase

* . . . failsafe heuristics

Fig. 3. Performance comparison branch and bound vs. heuristic update

99.8%

99.2%

testcase

30 workflow nodes 100.0%

99.8%

99.5%

)

98.5%

*

)

50 ,1

50 (9

50

0)

00

,1 80

(9

0,

15

50 ,1 50 (9

* . . . failsafe heuristics

0)

98.0%

0)

,1

20

0)

,1 30

(9

00

20

0)

,1

20

* 0)

25

50

testcase

(9

(9

(9

,1

20 ,1

,1

20

50

(9

00

0)

99.0%

00

99.2%

99.0%

(1

99.4%

0)

happiness ratio

99.6%

50

)

testcase


,8 00

)

50 ,8

(8

)*

50

30

00

,8

,9

50

(8

(8

)*

50

(8

(8

00

,8

50

)

99.0%

)

)

00 ,7

)

00 ,7

50

80

00

,7

(5

(5

)

)*

50

,7

(6

(6

00

00

00

00 ,7

00

50

(6

(5

,7

)

99.0%

99.4%

(8

99.2%

99.6%

,8

99.4%

50

99.6%

50

happiness ratio

99.8%

0)

happiness ratio


50

happiness ratio


(9


heuristics computation mputation time (ms)

heuristics computation mputation time (ms)

computation mputation time (ms)


testcase

Fig. 4. Happiness ratio between heuristic solution and optimum

415

416


We have presented a solution to the well-known problem of user-centric optimization of service composition and our approach shows its qualities in the second phase by demonstrating very promising results for real-time response to changing user requirements. Even more, in practical use the two-phase algorithm as a whole showed an acceptable runtime behavior, justifying it to be a working solution to the workflow optimization problem.

6

State of the Art

The expert group report EC Information Society and Media [2], published in 2010, has highlighted the role of resellers and aggregators in the Cloud Computing, who will aggregate resources according to the customer requirements. Buyya et al. [6] have also described the third-party resellers who can tie together services from different Clouds to meet the customer-defined QoS constraints. QoS constraints are an essential part of the workflow design and play an important role in service selection and scheduling [10] [14]. Binder et al. [15] extract the resource requirements of the services from the OWL-S [17] descriptions that allow defining nonfunctional properties of the components. A mathematical model then computes the execution cost of the workflow and a genetic algorithm is used afterwards to optimize the workflow execution. This approach very successfully maps the resources on workflow tasks but does not discuss dynamically changing conditions. Huang et al. [16] present a very good approach to workflow optimization by dynamic web service selection. An optimal service is selected based on the history data and real-time data. Their approach does not discuss the case of adapting to user-defined QoS constraints. Jia Yu et al. [10] propose a QoS-based workflow management system and scheduling algorithm for the service Grid that minimizes the execution cost and yet meets the time constraints imposed by the user. The QoS-level constraints can be defined at task level as well as at workflow level. In [11] and [9], we developed a blackboard [12] approach coupled with an A∗ algorithm to automatically construct and optimize the Grid-based workflows. We used CERN’s Datagrid project as a use case for our approach. The blackboard knowledge sources assess the characteristic parameter values for specific services such as cost, execution speed, bandwidth etc. needed in the construction process. This paper complements our previous work and aims at the challenge of workflow optimization at the next level.

7

Conclusion and Future Work

We have presented a two-phase heuristic algorithm to optimize service composition in Clouds that is based on a well-defined formal model. We have presented the details of its implementation using the workflow tool Kepler coupled with our optimization algorithm written in C++. The computation-intensive initial optimization can be done in parallel, a CORBA-based version of our algorithm demonstrating its scalable behavior in heterogeneous distributed environments is presented in [8]. We have also elaborated our heuristic optimization strategy for efficiently reacting to changing user requirements or reusing solutions


417

for users with similar requirements and briefly introduced its extension to runtime changes in service availability. We are working on the runtime optimization module that will be integrated with our system.

References 1. Yu, T., Lin, K.-J.: Services Selection Algorithms for Composing Complex Services with Multiple QoS Constraints. In: Benatallah, B., Casati, F., Traverso, P. (eds.) ICSOC 2005. LNCS, vol. 3826, pp. 130–143. Springer, Heidelberg (2005) 2. Jeffery, K., et al.: The Future of Cloud Computing: Opportunities for European Cloud Computing beyond 2010, EC, Information Society and Media (2010) 3. Gartner Hype Cycle for Emerging Technologies (2009), http://www.gartner.com/it/page.jsp?id=1124212 (last access: February 2010) 4. Van Looy, B., Gemmel, P., Van Dierdonck, R. (eds.): Services Management: An Integrated Approach, in: Financial Times. Prentice Hall, Harlow (2003) 5. Brandic, I., Music, D., Leitner, P., Dustdar, S.: VieSLAF Framework: Enabling Adaptive and Versatile SLA-Management. In: Altmann, J., Buyya, R., Rana, O.F. (eds.) GECON 2009. LNCS, vol. 5745, pp. 60–73. Springer, Heidelberg (2009) 6. Buyya, R., et al.: Cloud computing and emerging IT platforms: Vision, hype, and reality for delivering computing as the 5th utility. In: Future Generation Computer Systems, vol. 25, pp. 599–616. Elsevier, Amsterdam (2009) 7. Alonso, G., Casati, F., Kuno, H., Machiraju, V.: Web services: concepts, architectures and applications. Springer, New York (2004) 8. Kofler, K., Haq, I.U., Schikuta, E.: A Parallel Branch and Bound Algorithm for Workflow QoS Optimization. In: Proc. ICPP 2009, Vienna, Austria (September 2009) 9. Schikuta, E., Wanek, H., Haq, I.U.: Grid Workflow Optimization regarding Dynamically Changing Resources and Conditions. Journal of CCPE (2008) 10. Yu, J., Buyya, R., Tham, C.-K.: QoS-based Scheduling of Workflow Applications on Service Grids, in Technical Report, GRIDS-TR-2005-8, Grid Computing and Distributed Systems Laboratory, University of Melbourne, Australia, June 9 (2005) 11. Wanek, H., Schikuta, E.: Using Blackboards to Optimize Grid Workflows with Respect to Quality Constraints. In: GCCW, Huhan, China (October 2006) 12. Corkill, D.D.: Blackboard systems. AI Expert 6(9), 40–47 (1991) 13. Stockinger, H., Stockinger, K., Schikuta, E., Willers, I.: Towards a Cost Model for Distributed and Replicated Data Stores. In: Proceedings of 9th Euromicro Workshop on Parallel and Distributed Processing (PDP 2001). IEEE CS Press, Los Alamitos (2001) 14. Chen, J., Yang, Y.: Adaptive Selection of Necessary and Sufficient Checkpoints for Dynamic Verification of Temporal Constraints in Grid Workflow Systems. ACM Transactions on Autonomous and Adaptive Systems, Article 6 2(2) (June 2007) 15. Binder, W., Constantinescu, I., Faltings, B., Heterd, N.: Optimal Workflow Execution in Grid Environments. In: NODe/GSEM 2005, pp. 276–295 (2005) 16. Huang, L., Walker, D.W., Huang, Y., Rana, O.F.: Dynamic Web Service Selection for Workflow Optimisation. In: Proceedings of 4th UK e-Science Programme All Hands Meeting (AHM), Nottingham, UK (September 2005) 17. Martin, D., et al.: Bringing Semantics to Web Services: The OWL-S Approach. In: Cardoso, J., Sheth, A.P. (eds.) SWSWPC 2004. LNCS, vol. 3387, pp. 26–42. Springer, Heidelberg (2005) 18. The Kepler workflow tool, https://kepler-project.org/ (last access June 2010) 19. Qt – A cross-platform application and UI framework, http://qt.nokia.com/ (last access June 2010)

A Distributed Market Framework for Large-Scale Resource Sharing Marian Mihailescu and Yong Meng Teo Department of Computer Science, National University of Singapore Computing 1, 13 Computing Drive, Singapore 117417 {marianmi,teoym}@comp.nus.edu.sg

Abstract. Current distributed computing infrastructures, such as peerto-peer networks, grids, and more recently clouds, make sharing and trading resources ubiquitous. In these large distributed systems, rational users are both providers and consumers of resources. Currently, there is growing interest in exploiting economic models for the allocation of shared computing resources that incentivize rational users. However, when the number of resource types and users increases, computational complexity of the allocation algorithms grows rapidly and efficiency deteriorates. In this paper, we propose a scalable distributed market framework for the allocation of shared resources in large distributed systems. We use mechanism design to create a pricing scheme that allocates a request for multiple resource types, by trading economic efficiency for computational efficiency, strategy-proof and budget-balance. To address scalability, our proposed framework leverages on a peer-to-peer overlay for resource discovery and management. We prototype our framework using FreePastry, a popular overlay network based on the Pastry protocol. We show that our scheme is efficient and scalable using both simulation experiments and results from the deployment on PlantLab.

1

Introduction

Current distributed systems are slowly converging towards federated sharing of computing resources [7,11]. In peer-to-peer networks, grid computing and more recently cloud computing, users share and trade different types of resources over the network [22,23,24]. Recent results show that users that share resources are rational, i.e. they create strategies and manipulate the system in order to achieve their own objectives and maximize their own benefit [8,15]. For example, performance in file-sharing peer-to-peer networks is affected by free-riders, users that consume more than their fair share [19]; in a computational grid, users compete for the same resources, which results in increased waiting times [22]; some users of SETI@home, a popular distributed computing project, modified the software client to report false-negatives in order to achieve higher rankings [19]. Market-oriented economies have been successful in managing resource allocation in social systems with rational users. Consequently, there is growing interest in adopting market-based approaches for the allocation of shared resources in P. D’Ambra, M. Guarracino, and D. Talia (Eds.): Euro-Par 2010, Part I, LNCS 6271, pp. 418–430, 2010. c Springer-Verlag Berlin Heidelberg 2010

A Distributed Market Framework for Large-Scale Resource Sharing

419

computational systems. In addition to rational users, an economic-inspired resource allocation system can also address the dynamic nature of demand and supply. This is useful in systems where resource demand and supply can vary significantly over time, such as flash crowds or distributed systems with unreliable hosts that are geographically distributed over different administrative domains. In this paper we study the scalability of dynamic market-based sharing systems when the number of market participants and the number of resource types in a request increases. Specifically, pricing mechanisms proposed for the allocation of shared resources are either computationally hard, such as combinatorial auctions [5], or inherently centralized, such as double auctions [6]. Thus, increasing the number of peers or the number of resource types in a request leads to large waiting times for users, due to the computational complexity and the communication complexity, respectively. Consequently, we propose a distributed market framework where rational users are both buyers and sellers of resources. Our approach is to leverage existing work in peer-to-peer systems and use a distributed hash table (DHT) to manage resource information. Specifically, we map a user buy request on the DHT lookup operation, and a user sell offer on the DHT store operation. For pricing, our framework makes use of a reverse auction-based mechanism that is able to allocate a request containing more than one resource types [21]. In addition to multiple resource type allocations, other properties achieved by the pricing mechanism are incentive compatibility and budget balance, at the expense of economic efficiency. However, according to the Myerson-Satterthwite impossibility theorem [14], no pricing mechanism can be efficient, strategy-proof, and budget-balanced at the same time. The paper is organized as follows. In Sec. 2 we discuss related work and the main issues in distributed resource pricing: economic properties, such as strategyproof and Pareto efficiency; computational complexity; and scalability. Next, in Sec. 3, we propose a distributed market framework built on top of a peer-to-peer overlay, where all peers participate in allocating resources, as buyers and sellers, or determine an allocation, as request brokers or resource brokers. To determine the allocation time we perform theoretical and experimental analysis in Sec. 4. Lastly, Sec. 5 presents our conclusions and insights for future work.

2

Related Work

Market-based solutions for the allocation of shared resources have been previously proposed, but need to improve either the economic or the computational efficiency in order to become practical. Bellagio and Mirage [2] are two marketbased allocation systems that focus on maximizing economic efficiency. Accordingly, both systems use a repeated, sealed-bid combinatorial auction [5], a pricing mechanism that is Pareto-efficient and strategy-proof. Moreover, in a combinatorial auction the users can bid on bundles of resources, containing more than one resource types, as opposed to individual resources. However, determining which users get allocated using a combinatorial auction is a NP-hard problem [17]. Thus, both Bellagio and Mirage use a greedy algorithm to determine the

420

M. Mihailescu and Y.M. Teo

allocation winners, such that the allocation time does not depend on the number of resource types in the buyer request [4]. There are two drawbacks in this approach. Firstly, using a greedy algorithm leads to an outcome that is not Pareto-efficient. Secondly, the allocation mechanism is no longer strategy-proof. Thus, economic efficiency is further reduced when users are untruthful [4]. The distributed market framework we propose in this paper uses a strategy-proof pricing mechanism [21]. Although Pareto efficiency is not achieved, economic efficiency is not affected by the users’ degree of untruthfulness. Several market-based allocation systems for grids, such as Sorma [16] and Nimrod/G [3], use bargaining or negotiation to determine the resource price. The advantage of this approach is that sellers and buyers communicate directly, without a third party mediating an allocation. However, bargaining results in high communication costs. In a large dynamic market, each buyer has to negotiate with all sellers of a resource type in order to maximize his utility. The communication costs further increase when a buyer requires more than one resource types. Thus, scalability is an issue both when increasing the number of users and the number of resource types in a request. We propose to manage resource information using a peer-to-peer overlay network, where each resource type lookup can be processed in parallel by different peers. Cloud computing uses fixed pricing to provide computational resources and services on-demand over the Internet. Although simple to implement, fixed pricing is not suitable for a system with multiple providers, or where users are both sellers and buyers of resources [13]. Federated clouds, a topic of recent interest, aims to integrate cloud resources from different providers, to increase scalability and reliability [4]. With a large number of providers (sellers) and users (buyers), fixed pricing cannot adapt to the changes in demand and supply. More suitable for federated clouds, dynamic pricing mechanisms such as the one used in the proposed framework sets resource payments according to demand and supply. In PeerMart [10], the authors propose a distributed market for peer-to-peer services built on top of a peer-to-peer overlay. Although resource location in PeerMart and the proposed framework is similar, our framework provides several key advantages. Firstly, PeerMart does not support multiple resource type allocations. When a user requires several resource types, it has to aggregate resources manually, which is not efficient. Secondly, PeerMart is not strategyproof. Pricing takes place using a simple double-auction mechanism, where the payment is the mean price between seller price and buyer price. Thus, users are encouraged to submit untruthful prices to increase their utility.

3

Proposed Distributed Market Architecture

We have identified three major components that constitute the market architecture: resource location, the pricing mechanism, and allocation administration. A fundamental problem in dealing with large collections of shared resources is resource location. As shown in Fig. 2, the resource market receives resource information from sellers in publish messages, and query requests from buyers in


RESOURCE LOCATION

Buyer Request

Resource Lookup Query Request(s)

Available Resources

Resource Market Resource Information Seller Request

PRICING MECHANISM

Winner Determination Available Resources

Winners

ALLOCATION ADMINISTRATION

Payment Computation Reserved Price

421

Allocation

Final Payments

Market Maker

Market Maker

Market Maker

SCF

Payment func.

Inform winners

Reserved Price Payments Transfer

Publish Resource

Resource Use

Fig. 1. Market Architecture

lookup messages. The time required to perform resource lookup is a significant part in the total allocation time. Searching for available resources in a lookup takes longer time when the number of resources is large. Moreover, resource information and availability can vary over time, as resources join, leave, or fail in the system. To perform efficiently in these conditions, the resource location service requires scalability and support for dynamic resources. The pricing mechanism is key part in the resource allocation framework. Based on the social choice function (SCF) and available resources, the pricing mechanism determines which users are allocated (Winner Determination). Next, using the published prices and the selected payment functions, the pricing mechanism computes the user payments, both for sellers and buyers (Payment Computation). The performance of the pricing mechanism performance takes into consideration several features: strategy-proof, budget-balance, Pareto-efficiency, and multiple type allocations; the computational efficiency; and scalability. Additionally, the performance of the pricing mechanism is measured by the percent of successful requests and resources allocated. In our previous work [21], we describe the resource market using a mechanism design problem and propose a reverse auction-based pricing mechanism with Vickrey-Clarke-Groves [9] seller payments and the sum of seller payments for the buyer. The proposed pricing mechanism allocates resources for single or multiple resource types buyer requests, while achieving incentive-compatibility and budget-balance. For simplicity, we have considered a centralized marketmaker that manages the available resources and requests. However, our results have shown that scalability becomes an issue when the number of users or the number of resource types is increased [13]. In this paper, we focus on resource location and propose a scalable, distributed market framework. We refer to vertical scalability when increasing the number of resource types, and horizontal scalability when increasing the number of users. In the last step of resource allocation, the winning sellers and buyers are informed of the allocation, and the buyer can start using the resources. In addition, payments take place. Additional features such as management or monitoring, can

422

M. Mihailescu and Y.M. Teo Hash(A)

Resource Broker Resource Type A

Hash(A)


Lookup Message Resource Type A Request Broker

Publish Message Resource Type A Seller Resource Type A Resource Type B

Publish Message Resource Type B

Request Message Resource Type A Resource Type B

Hash(B) Resource Broker Resource Type B

Buyer Resource Type A Resource Type B

(a) Seller and Resource Brokers

Lookup Message Resource Type B

Hash(B) Resource Broker Resource Type B

(b) Buyer and Request Broker

Fig. 2. Distributed Resource Market

be added to allocation administration. The performance of allocation administration depends on the payment system used by the framework, which must be scalable and secure. 3.1

Distributed Auction Mechanism

In order to maintain the economic properties of the proposed pricing mechanism, the distributed payment computation requires complete information about resources in the system in order to determine payments. In our distributed model, resource information such as owner address, resource type, number of items, and the cost for each item, is divided between different hosts to provide scalability. Similarly, a buyer request is also divided into several lookups that are sent to the relevant hosts for the price to be computed. Accordingly, our approach is to use distributed hash tables (DHT), as the infrastructure to maintain resource information and to build a scalable resource location service for buyers, where resource information is distributed according to the resource type. Peers in a DHT are organized as a structured overlay network, and data is mapped to nodes in the overlay network based on their identifier. A DHT provides an interface lookup(key) for retrieving a key-value pair. The key is an identifier assigned to a resource, while the value is an object from the DHT. To facilitate object location, a DHT overlay network provides a routing operation, based on the destination node key. Thus, the distributed hash table is a natural data structure to support the proposed distributed resource market model, where the resource type is the key used to store the list of sellers for the respective resource type. Figure 2 shows the basic architecture of our system. A peer represents a user that joins the distributed market overlay. All peers can perform any of the following roles: Seller: A peer becomes a seller after publishing a resource. The seller sends a different publish message for each resource type, containing the number of available items and the cost for each item. The publish operation is performed using the DHT store interface, using the hash of the resource type as the key.


423

Thus, the published resource information is stored by the resource broker, the peer with the identifier closest to the hash of the resource type. Buyer: A peer becomes a buyer when it sends a request message. The request message contains one or more resource types, the number of items for each resource type, and the total price the buyer is willing to pay. The key for the request message is a random identifier. Accordingly, the request message is routed by the overlay to the peer with the identifier closest to the random key, which becomes a request broker. Resource Broker: A resource broker is the peer in the overlay network with the identifier closest to the hash of a resource type published by sellers. The relation “closest” is defined by the specific overlay implementation, e.g. numerically closest forPastry [18], the first node in clockwise direction for Chord [20], etc. Resource brokers keep a list of published resources for each resource type they are responsible for. After receiving a lookup request, the resource broker determines the winners and computes the payments for the respective resource type. If the allocation is successful, it updates the resource list and informs the winning sellers about the allocation and payments. Request Broker: A peer in the overlay becomes a request broker when its node identifier is closest to a buyer request identifier. When receiving a buyer request, the broker sends a lookup for each resource type in the request, using the hash of the resource type as the key. Thus, each lookup is routed by the overlay to the resource broker responsible for the respective resource type. After receiving seller payments for each resource type, the request broker computes the buyer payment. If the allocation is possible, i.e. the buyer payment is less or equal to the buyer price, the request broker sends a commit message directly to the resource brokers. In addition, the request broker informs the buyer about the payments and allocation. 3.2

Deadlock-Free Auctions Protocol

The distributed auction mechanism outlined above is, in fact, a two-phase commit protocol. Firstly, a buyer request is routed by the overlay to a request broker. Then, for each resource type, the request broker sends a lookup message using the hash of the resource type as the key. This corresponds to the commit-request phase. The lookup messages are routed by the overlay network to the resource brokers responsible for the respective resource types. Next, the resource brokers compute payments and reply to the request brokers, similar to an agreement message. Lastly, the request brokers send the commit message. Using a blocking protocol such as two-phase commit has several implications. Firstly, lookup requests are serialized by resource brokers. Thus, between sending the payment and receiving the commit message, resource brokers are not able to compute payments for other requests. However, this strategy helps maintaining the strategy-proof property of the pricing mechanism we employ. In [21], we show that buyer strategy-proof is achieved by selecting requests independent of the buyer valuation, such as the first-come-first-serve strategy. Accordingly, a blocking protocol implements the same requirement for a distributed market.

424


One of the drawbacks of using a two-phase commit is that concurrent requests for more than one resource types may lead to deadlocks, when lookup messages for different requests arrive at multiple resource brokers in different order. In order to prevent deadlock, we propose an algorithm which employs a threephase commit protocol in conjunction with a synchronization mechanism inspired from Lamport’s logical clocks [12]. Figure 3 contains a diagram of the deadlock-free auction protocol. Initially, the buyer request is routed by the overlay network to the request broker. We use dotted arrows for routed messages and solid arrows for direct peer-to-peer messages. In the first phase of the proposed protocol (I), the request broker sends a lookup message for each resource type in the buyer request. Each resource broker responds with a ticket message, which contains the number of the next request in the resource broker lookup queue. The ticket numbers are used for synchronization in the second phase similar to the Lamport logical timestamps. Buyer Resource Type A Resource Type B

Request Broker

1. Request Message Resource Type A Resource Type B

I


Resource Broker Resource Type B

2A. Lookup Message Resource Type A 2B. Lookup Message Resource Type B 3A. Ticket Message

3B. Ticket Message

4. Position Messages

II

5B. Payment Message 5A. Payment Message

III

6. Commit Messages

Fig. 3. Proposed Auction Protocol

The request broker waits for all ticket numbers corresponding to each resource type in the buyer request. In the second phase (II), after all ticket numbers are received, the request broker sends to all resource brokers a position message containing the maximum ticket number. All resource brokers re-order their lookup queues by moving the respective lookup from the ticket to the position location in the queue. Next, the resource brokers compute payments for the lookup message in the head of the queue.


425

In the last phase (III), after the request broker receives all payments, the commit message is sent with the result of the allocation. If the allocation is successful, the buyer and sellers are sent allocation and payment information. The advantage of the proposed distributed auction mechanism is twofold. Firstly, buyer requests for different resource types are processed in parallel. In contrast to the centralized model, where all requests are serviced by the marketmaker sequentially, having a distributed market allows different resource brokers to make concurrent allocations. Secondly, the payment computation for the same request is also parallelized. Thus, the request broker distributes a request using several lookup messages, one for each resource type. Each lookup is processed by different resource brokers, and the computation time for the allocation is reduced to the computation time for one resource type. Using the peer-to-peer model, where any peer can be a seller, buyer, resource broker and request broker, adds scalability to the our market-based resource allocation framework.

4

Analysis

We analyze the vertical and horizontal scalability of the proposed pricing mechanism both theoretical and experimental. We developed a prototype implementation of the proposed distributed market using FreePastry [1] as the underlying overlay network. FreePastry allows us to measure the performance of our pricing and allocation scheme in two different environments: the FreePastry simulator, where we can simulate a large distributed market, and PlanetLab, where we deployed our framework and use it to validate the simulator results on a smaller number of distributed nodes. In the theoretical analysis, we identify the factors that affect the request allocation time. We verify the result of the theoretical analysis against experimental results obtained using simulations. 4.1

Vertical Scalability

To study vertical scalability, with respect to the number of resource types in a buyer request, we compare the average allocation time in the distributed market implementation with a centralized implementation. In the latter, we delegate one peer as the market-maker, while the other peers are either buyers and sellers. As shown in Fig. 4(a), we use 5, 10, 20 and 40 resource types in a buyer request for the centralized and distributed implementation, both on PlanetLab and the simulator. We create an overlay network containing 50 peers, where each peer generates 100 events. The total interarrival rate of the events is exponentially distributed with the mean of one second. We consider a balanced market, where demand equals supply. Accordingly, each peer event has an equal probability for a publish or a request message. Additionally, the number of items for each resource type is generated from an exponential distribution with mean 10, and the price for a resource item is uniformly distributed between 80 and 120. Our results show that in the distributed market, the auction protocol imposes a greater overhead and, when having a small number of resource types in a

426


request, the average allocation time is higher than the centralized implementation. However, as the number of resource types increases, the distributed scheme proves to be scalable and maintains a consistent allocation time of less than one second. The average allocation time obtained in simulations is higher than the measured time on PlanetLab because the simulations used a fixed network delay between peers of 100 ms, much higher than the delay between PlanetLab nodes. 4.2

Horizontal Scalability

To study horizontal scalability, we use the FreePastry simulator, as PlanetLab is not able to provide the necessary number of nodes for a scalability study. In our simulations, we were able to create an overlay network of approx. 33,000 nodes before the average peer join time increase exponentially. Consequently, our results summarized in Fig. 4(b) are for network sizes of 1,000, 5,000, 10,000, 20,000 and 30,000 peers. We have used similar settings as in the previous study: one second event interarrival rate, balanced market, number of items from an exponential distribution with mean 10, price uniformly distributed between 80 and 120, and 100 ms fixed network delay. The number of resource types is sampled from an uniform distribution between 1 and 10. We run the simulations for 600,000 events, or approx. 7 simulation days. Our results show that the proposed distributed market framework is horizontally scalable as the average allocation time increases logarithmic with the number of users. Additionally, we have measured the average number of hops for all messages required by the proposed protocol and found that both curves have the same gradient. Thus, our scheme adopts the horizontal scalability of the underlying overlay network. 4.3

Average Allocation Time

We consider the average allocation time (Talloc ) as the total time taken since a buyer sends the request until it receives the allocation results, averaged for all

4 3 2 1 0 5

10

20 30 Number of resource types

40

(a) Varying the Number of Resource Types

12

Average allocation time Average number of hops

1.15

11.5

1.1

11

1.05

10.5

1

10

0.95

9.5

0.9

9

0.85 0.8

8.5 1

5 10 Number of peers (x1,000)

20

30

8

(b) Varying the Number of Peers

Fig. 4. Horizontal and Vertical Scalability

Average number of hops

1.2

Centralized (PL) Distributed (PL) Distributed (SIM)

Average allocation time (s)


5


427

successful buyer requests. We identify three components that determine the total allocation time: i) communication time (Tn ), which represents the time taken to transmit messages in the network; ii) queue time (Tq ), which is the time spent by a lookup message in the resource broker queue since the position message is received until payments are computed; and iii) computational time (Tc ), which is the time taken to run the pricing algorithm: Talloc = Tn + Tq + Tc Communication Time. The communication cost incurred by the allocation of a buyer request is given by the messages exchanged between the buyer, request broker, and resource broker: request message, lookup message, ticket message, position message, payments message, commit message. We consider a stable overlay network, where routing takes at most log N steps, where N is total number of peers in the overlay. Thus, the request message is routed from the buyer to the request broker in at most log N hops. Similarly, the lookup message is routed from the request broker to the resource broker in at most log N steps. After the resource broker receives the lookup message, it can look at the sender address and reply with the ticket number in one hop. Similarly, the position, payments and allocation messages are forwarded in one hop. Considering an average network delay time, d, the total communication time is: Tn = d(log N + log N + 1 + 1 + 1 + 1) = 2d(log N + 2)

(1)

Service Time. To determine the queue time and the computational time, we assume Poisson arrivals for the buyer requests and we use queuing theory, considering the resource broker a M/M/1 system. The service time is computed as: 1 μ−λ where μ is the average service rate of the resource broker, and λ is the average lookup arrival rate (for a resource type). Since the resource broker serializes the requests, the service rate is given by the time since the computation starts, until the commit message is received, when a new lookup can be processed. Accordingly, the service time includes the computation time (Tc ), sending the payment message (d), and receiving the commit message (d). Thus, the service rate is: 1 μ= Tc + 2d Accordingly, the total service time is: Ts = Tq + Tc =

Ts =

Tc + 2d 1 − (Tc + 2d)λ

For simplicity, we consider that the computation time Tc negligible, compared to the average network delay time d, which in the Internet is ranging from several milliseconds to several hundreds of milliseconds. Thus, ignoring the computation time, we can express the total allocation time for a buyer request as:

428



20

Theoretical Simulator

15

10

5

0 0.1

0.2

0.3

0.4 0.5 0.6 0.7 Network delay (s)

0.8

0.9

Fig. 5. Varying the Network Delay

Talloc = 2d(log N + 2) +

1 2d − 2d = 2d(log N + 1 + ) 1 − 2dλ 1 − 2dλ

(2)

In summary, the factors which affect the scalability of the proposed framework are: i) N , the size of the overlay network; ii) d, the network delay; and iii) λ, the arrival rate of requests for a resource type. The previous results (Section 4.2) show that the average allocation time increases logarithmically with N , the number of peers in the network. 4.4

Network Delay

Our theoretical analysis show that network delay is one of the factors that influences the scalability of the proposed distributed market framework. To evaluate the impact of network delay, we have simulated an overlay network of 10,000 peers, with an event interarrival rate of one second in a balanced market. We simulated 600,000 events for different fixed values of network delay, ranging from 100 ms to 900 ms. For comparison, we plotted average allocation time for the simulations together with the theoretical worst-case scenario in Fig. 5. Although the network delay from the PlanetLab nodes was much lower (∼ 40 ms) than the values used in our simulation, we have found that a large network delay can result in increased user waiting times.

5


Recent work in distributed systems employs economic models to incentivize selfish users to share resources and to behave according to the rules defined by the system. In this paper, we have considered a market-based approach to resource allocation, where financial incentives are used together with user payments to manage rational users. By having a finite amount of currency to spend on resources, rational users are motivated to spend it based on their expected utility.


429

In large resource markets, scalability is an issue when information is centralized. We have addressed this issue with a distributed auction scheme, in which resource information is divided according to resource type. Our distributed market leverages on a peer-to-peer overlay to create a resource location service suitable for large markets. Each peer in the overlay network can be a seller, buyer, resource broker or request broker. Resource brokers maintain the resource information and compute payments for each resource type. Request brokers ensure buyer incentive compatibility by implementing a first-come-first-serve strategy for requests having common resource types, while different resource types can be allocated in parallel. Request brokers use the DHT lookup interface to locate the resource brokers for each resource type and decide if allocation is possible. Our results show that for more than 20 resource types, the average allocation time obtained using distributed auctions is around one second, much lower than using a centralized pricing scheme. When increasing the number of market participants, the average allocation time is increased logarithmic. Our experiments have been performed using a simulator using FreePastry as the underlying overlay network, which was validated by a prototype implementation on PlanetLab. In addition, we performed a theoretical analysis to reveal the remaining factors that influence the scalability of our scheme: large network delays and high request arrival rate. To further increase scalability, the strategic conditions imposed by the pricing scheme can be relaxed. Specifically, in order to achieve incentive compatibility, the current pricing scheme require complete information about resources of the same type. In our future work we intend to investigate resource allocation with incomplete information and the impact of maintaining incentive compatibility has on the overall allocation performance.

Acknowledgments This work is supported by the National University of Singapore under grant number R-252-000-339-112.

References 1. FreePastry - A scalable, decentralized, self-organizing and fault-tolerant substrate for peer-to-peer applications (2009), http://freepastry.org 2. Auyoung, A., Chun, B., Snoeren, A., Vahdat, A.: Resource Allocation in Federated Distributed Computing Infrastructures. In: Proc. of the Workshop on Operating System and Arch. Support for the On-demand IT Infr., Boston, USA (2004) 3. Buyya, R., Abramson, D., Giddy, J.: Nimrod/G: An Architecture of a Resource Management and Scheduling System in a Global Computational Grid. In: Proc. of the 4th Intl. Conf. on High Performance Computing in Asia-Pacific Region, Beijing, China, pp. 283–289 (2000) 4. Buyya, R., Bubendorfer, K. (eds.): Market Oriented Grid and Utility Computing. Wiley Press, Chichester (2009) 5. Cramton, P., Shoham, Y., Steinberg, R. (eds.): Combinatorial Auctions. MIT Press, Cambridge (2006)

430


6. Eymann, T., Reinicke, M., Ardaiz, O., Artigas, P., de Cerio, L.D., Freitag, F., Messeguer, R., Navarro, L., Royo, D., Sanjeevan, K.: Decentralized vs. centralized economic coordination of resource allocation in grids. In: European Across Grids Conf., Santiago de Compostela, Spain, pp. 9–16 (2003) 7. Feigenbaum, J., Papadimitriou, C.H., Shenker, S.: Sharing the Cost of Multicast Transmissions. Journal of Computer and System Sciences 63, 21–41 (2001) 8. Gon Chun, B., Fonseca, R., Stoica, I., Kubiatowicz, J.: Characterizing Selfishly Constructed Overlay Routing Networks. In: Proc. of INFOCOM 2004, Hong Kong, China, pp. 1329–1339 (2004) 9. Groves, T.: Incentives in Teams. Econometrica 41(4), 617–631 (1973) 10. Hausheer, D.: PeerMart: The Technology for a Distributed Auction-based Market for Peer-to-Peer Services. In: Proc. of the 40th IEEE Intl. Conf. on Communications, Seoul, Korea (2005) 11. Krauter, K., Buyya, R., Maheswaran, M.: A Taxonomy and Survey of Grid Resource Management Systems for Distributed Computing. Software Practice and Experience 32, 135–164 (2002) 12. Lamport, L.: Time, clocks, and the ordering of events in a distributed system. Communications of the ACM 21(7), 558–565 (1978) 13. Mihailescu, M., Teo, Y.M.: Strategic-Proof Dynamic Resource Pricing of Multiple Resource Types on Federated Clouds. In: Proc. of the 10th Intl. Conf. on Algorithms and Architectures for Parallel Processing, Busan, Korea, pp. 337–350 (2010) 14. Myerson, R.B., Satterthwaite, M.A.: Efficient Mechanisms for Bilateral Trading. Journal of Economic Theory 29(2), 265–281 (1983) 15. Nielson, S.J., Crosby, S.A.: A Taxonomy of Rational Attacks. In: Proc. of the 4th Intl. Workshop on Peer-to-Peer Systems, Ithaca, USA, pp. 36–46 (2005) 16. Nimis, J., Anandasivam, A., Borissov, N., Smith, G., Neumann, D., Wirstr¨ om, N., Rosenberg, E., Villa, M.: SORMA - Business Cases for an Open Grid Market. In: Grid Economics and Business Models, Berlin, Germany, pp. 173–184 (2008) 17. Nisan, N.: Bidding and Allocation in Combinatorial Auctions. In: Proc. of the 2nd ACM Conf. on Electronic Commerce, Minneapolis, USA, pp. 1–12 (2000) 18. Rowstron, A., Druschel, P.: Pastry: Scalable, Decentralized Object Address, and Routing for Large-Scale Peer-to-Peer Systems. In: Proc. of the Intl. Conf. on Distributed Systems Platforms, Heidelberg, Germany, pp. 329–350 (2001) 19. Shneidman, J., Parkes, D.C.: Rationality and Self-Interest in Peer to Peer Networks. In: Proc. of the 2nd Intl. Workshop on Peer-to-Peer Systems, Berkely, USA, pp. 139–148 (2003) 20. Stoica, I., Morris, R., Liben-Nowell, D., Karger, D., Kaashoek, M.F., Dabek, F., Balakrishnan, H.: Chord: A Scalable Peer-to-peer Lookup Service for Internet Applications. Networking, IEEE/ACM Transactions 11(1), 17–32 (2003) 21. Teo, Y.M., Mihailescu, M.: A Strategic-proof Pricing Scheme for Multiple Resource Type Allocations. In: Proc. of 38th Intl. Conf. on Parallel Processing, Vienna, Austria, pp. 172–179 (2009) 22. Wolski, R., Plank, J.S., Brevik, J., Bryan, T.: G-commerce: Market Formulations Controlling Resource Allocation on the Computational Grid. In: Proc. of the Intl. Parallel and Distributed Processing Symp., San Francisco, USA, pp. 46–54 (2001) 23. Wu, C., Li, B., Li, Z.: Dynamic Bandwidth Auctions in Multioverlay P2P Streaming with Network Coding. IEEE Transactions on Parallel Distributed Systems 19, 806– 820 (2008) 24. Yeo, C.S., Buyya, R.: A Taxonomy of Market-based Resource Management Systems for Utility-driven Cluster Computing. Software: Practice and Experience 36, 1381– 1419 (2006)

Using Network Information to Perform Meta-scheduling in Advance in Grids Luis Tomás1 , Agust´ın Caminero2 , Blanca Caminero1 , and Carmen Carrión1 1

Dept. of Computing Systems, The University of Castilla-La Mancha, Spain {luistb,blanca,carmen}@dsi.uclm.es 2 Dept. of Communication and Control Systems, The National University of Distance Education, Spain [email protected]

Abstract. In extremely heterogeneous and distributed systems, like Grid environments, it is quite difficult to provide quality of service (QoS). In addition, the dynamic behaviour of the resources makes the time needed to complete the execution of a job highly variable. So, fulfilling the user QoS requirements in a Grid is still an open issue. The main aim of this work is to provide QoS in Grid environments through network-aware job scheduling in advance. This paper presents a technique to manage idle/busy periods of resources using red-black trees which considers the network as a first level resource. Besides, no a priori knowledge on the duration of jobs is required, as opposed to other works. A performance evaluation using a real testbed is presented which illustrates the efficiency of this approach to meet the QoS requirements of users, and highlights the importance of taking the network into account when predicting the duration of jobs. Keywords: Grid meta-scheduling, network, QoS, red-black trees.

1 Introduction In a Grid environment the resources are in different domains under different access policies. This fact makes their search and use a hard task for users. Also, manually accomplishing this process is not feasible in a large-scale Grid environments with many potentially available resources. Hence, the Grid infrastructure must provide the needed services for automatic resource brokerage which take care of the resource selection and negotiation process [1]. This infrastructure is named “meta-scheduler” [2]. The user’s experience of the Grid is determined by the functionality and performance of this meta-scheduler system. But the heterogeneous and distributed nature of the Grid along with the different characteristics of applications complicate the brokering problem. To further complicate matters, the broker typically lacks total control and even complete knowledge of the status of the resources [3]. One key idea to solve this problem is to ensure that a specific resource is available when the job requires it. So, it becomes necessary to reserve or schedule the use of resources in-advance [4]. Reservation in advance can be defined as a restrictive or limited delegation of a particular resource capacity for a defined time interval [5]. The objective P. D’Ambra, M. Guarracino, and D. Talia (Eds.): Euro-Par 2010, Part I, LNCS 6271, pp. 431–443, 2010. c Springer-Verlag Berlin Heidelberg 2010

432

L. Tomás et al.

of such reservation in advance is to provide quality of service (QoS) by ensuring that a certain job meets its QoS requirements. The main challenge of reservations in advance is that it is difficult to decide whether a job can be executed fulfilling its QoS requirements without knowing the exact status of the resources in the future [6]. However, reservation in advance mechanisms enable QoS agreements with users and increase the predictability of a Grid system [7], at the expense of creating resource fragmentation [8]. As next sections depict, our work is different from the ones mentioned above because it is based on meta-scheduling in advance in Grids rather than reservations in advance, since reservation may not always be possible. This paper proposes a new network-aware algorithm to tackle the scheduling in advance problem. This algorithm is concerned with the dynamic behaviour of the Grid resources, their usage, and the characteristics of the jobs. This research focuses on low-cost computational heuristics that consider the network as a first level resource. This is needed because the network has a very important impact on the performance of jobs, as studied in [9,10,11,12], among others. The usage of resources is managed by means of red-black trees. This idea has already been tried in [6,8], where authors assume that users have a priori knowledge on the job duration – which may not be true most times. So estimations on the completion time of jobs need to be calculated. To this end, it becomes necessary to predict the dynamic behaviour of the resources in the future. In the present work, estimations for job duration are calculated in two different ways: (1) using the Total Completion Time (TCT), and (2) Execution and Transfer Times Separately (ETTS), thus estimations on network transfer times have to be calculated. In (1), estimations on job durations use information of previous executions, and does not consider the network transfers (only completion times, that include transfer and execution times). In (2), estimations on the execution and transfer times of jobs are calculated independently. For execution time, an estimation is calculated similarly to the completion time in (1), and transfer times are calculated using bandwidth predictions through log data along with the number of bytes to transfer. Thus, both techniques pay attention to the heterogeneity of Grid resources and do not assume users have a priori knowledge on the duration of jobs, as assumed in [6,8]. These ways of estimating the job completion times are presented and evaluated in this paper. The paper is organized as follows. Related work is presented in Section 2. In Section 3 a brief overview of the general network-aware meta-scheduling in advance framework is presented. Section 4 explains the extensions implemented to handle scheduling in advance. Section 5 presents the experiments carried out for evaluating the proposal. Finally, the conclusions obtained and the suggested guidelines for future work are outlined in Section 6.

2 Related Work A Grid application may need multiple heterogeneous resources which may span over administrative boundaries, thus making the management of resources a challenging task [13]. Software infrastructures required for resource management and other tasks such as security, information dissemination and remote access are provided through Grid toolkits such as Globus [14] and Legion [15].

Using Network Information to Perform Meta-scheduling in Advance in Grids

433

Regarding the aforementioned advanced reservations of resources, Globus Architecture for Reservation and Allocation (GARA) [16] was introduced for application-level dynamic scheduling of collection of resources, co-allocation and advanced reservations. GARA is one of the seminal works on advanced reservation and defines a basic architecture for the manipulation of advanced reservation of different resources. Since then, advanced reservations have been studied in numerous contexts, such as clusters (Maui Scheduler [17]). Among the systems that allow resource reservation in a Grid we can find Grid Capacity Planning [7], that provides users with reservations of Grid resources through negotiations, co-allocations and pricing. Another important system is VIOLA [18], which includes a meta-scheduling framework that provides co-allocation support for both computing and network resources. It allows the network to be treated as a resource within a meta-scheduling application. Despite support for reservation in the underlying infrastructure is currently limited, a reservation in advance feature is required to meet QoS guarantees in Grid environments, as several contributions conclude [7,16]. Qu [19] describes a method to overcome this shortcoming by adding a Grid advanced reservation manager on top of the local scheduler(s). The performance penalty imposed by the usage of advanced reservations (typically decreased resource utilization) has been studied in [20]. Furthermore, advanced reservations have been shown to increase the predictability of the system while maximizing its flexibility and its adaptability to cope with the dynamic behaviour of Grid environments [21]. On the other hand, this work aims at performing scheduling in advance. This way of scheduling needs to perform predictions about the future network status and about job duration into resources. Currently, a broad range of estimation techniques are built around historical observations. In [22], it is shown that although load exhibits complex properties, it is still consistently predictable from past behaviour. In [23], an evaluation of various linear time series models for prediction of CPU loads in the future is presented.

3 Network-Aware Meta-scheduling in Advance A Grid is an environment in which resources vary dynamically – they may fail, join or leave the Grid system at any time. Also, such dynamicity comes from the fact that every Grid resource needs to execute local tasks as well as tasks from Grid applications. From the viewpoint of a Grid application, all the tasks from both local users and Grid users are loads on the resource. So, everything in the system is evaluated by its influence on the application execution. Support for reservations in advance of resources plays a key role in Grid resource management as it allows the system to meet user expectations with regard to time requirements and temporal dependence of applications, and increases the predictability of the system [6]. A Grid reservation in advance process can be divided into two steps [5]: 1. Meta-scheduling in advance: Selection of the resources to execute the job, and the time period when the execution will be performed. 2. Negotiation for resource reservation: Consists on the physical reservation of the resources needed for the job, which may not always be possible.

434

L. Tomás et al.

In a real Grid environment, reservations may not be always feasible, since not all the Local Resource Management Systems (LRMS) permit them. Apart from that, there are other types of resources such as bandwidth (e.g. the Internet), which lack any management entity, and makes impossible their reservation. This is the reason why our work is aimed at performing meta-scheduling in advance rather than reservations in advance in order to provide QoS in Grids. That is, the system keeps track of the meta-scheduling decisions already made in order to make future decisions. So, if only Grid load exist, this would be enough to provide QoS since the meta-scheduler would not overlap jobs on resources. The algorithms for meta-scheduling in advance need to be efficient so they can adapt themselves to dynamic changes in resource availability and user demand without affecting system and user performance. Moreover, they must take into account resource heterogeneity since Grid environments are typically highly heterogeneous. For this reason, it could be useful to employ techniques from computational geometry to develop an efficient heterogeneity-aware scheduling algorithm [6]. In this work jobs do not have workflow dependencies and users provide both the input files and the application itself. Taking into account these assumptions an scheduling in advance process follows the next steps: 1. First, a “user request” is received. Every request must provide a tuple with information on the application and the input QoS parameters: (in f ile, app, t s, d). in f ile stands for the input files required to execute the application, app. In this approach the input QoS parameters are just specified by the start time, t s (earliest time jobs can start to be executed), and the deadline, d (time by which jobs must have been executed) [8]. 2. The meta-scheduler executes a gap search algorithm. This algorithm obtains both the resource and the time interval to be assigned for the execution of the job. 3. If it is not possible to fulfill the user’s QoS requirements using the resources of its own domain, communication with meta-schedulers from other domains starts. 4. If it is still not possible to fulfill the QoS requirements, a negotiation process with the user is started in order to define new QoS requirements. In this process, the goodness of scheduling depends heavily on the quality of available information regarding the resources, but independence and autonomy of domains is another obstacle. This is because domains may not want to share information on the load of their resources. Moreover, in a Grid environment, resource contention causes host load and availability to vary over time, and makes the execution time predictions quite difficult [24]. The prediction information can be derived in two ways [24]: application oriented and resource oriented. For application-oriented approaches, the running time of Grid tasks is directly predicted by using information about the application, such as the running time of previous similar tasks. For resource-oriented approaches, the future performance of a resource such as the CPU load and availability is predicted by using the available information about the resource, and then such predictions are used to predict the running time of a task, given the information on the task’s resource requirement.


435

Fig. 1. The Scheduler in Advance Layer (SA-layer)

In our case we use a mixture between these two approaches. We use applicationoriented approaches to sort out the execution time of the application and resourceoriented approaches to calculate the time needed to perform the network transfers.

4 A Framework for Network-Aware Meta-scheduling in Advance In this section, the implementation carried out allowing network-aware meta-scheduling in advance is outlined. First, the structure of the framework is presented, followed by the policies for allocating jobs into gaps in resources. Next, the data structures used for managing this information are shown. Finally the prediction needs are discussed. Our proposal is implemented as an extension to the GridWay meta-scheduler [2]. It is an intermediate layer, called Scheduler in Advance Layer (SA-layer), between the users and the on-demand Grid meta-scheduler, as Figure 1 depicts. The SA-layer is a modular component that uses functions provided by GridWay in terms of resource discovery and monitoring, job submission and execution monitoring, etc., and allows GridWay to perform network-aware meta-scheduling in advance. The SA-layer stores information concerning previous application executions (called DB Executions), and the status of resources and network over time (called DB Resources). Moreover, a new parameter has been added to GridWay, named JOB INFORMATION. In this new parameter the user may indicate some information about the job. First, if the user knows the input and output size, he sets this information. After that, the user may set other characteristics related to the jobs, such as job arguments, which enable a more accurate estimation for the job execution time. On that purpose, the execution time of jobs in a given resource is estimated by using prediction. This prediction takes into account the characteristics of the jobs, the power of the CPU of the resources and the network future status. By processing these information about applications and resources, a more accurate estimation of the job completion time in the different computational resources can be performed. Besides, the memory overhead is negligible (about several Mbits). 4.1 Gap Management In this implementation, resource usage is divided into time slots. Then, we have to schedule the future usage of resources by allocating the jobs into the resources at one

436

L. Tomás et al.

Fig. 2. Idle periods regions [6,8]

specific time (taking one or more time slots). For this reason, allocation policies (carried out by Gap Management module in Figure 1) to find the best slots for each job, data structures (represented by Data Structure in Figure 1) to keep a trace of slots usage and algorithms for estimations on job durations (implemented by Predictor in Figure 1) are needed, which are explained the next. The job allocation influences how many jobs can be scheduled due to the generated fragmentation. Different ways of searching and allocating jobs into resources can be developed considering both the already scheduled jobs and the generated fragmentation. In this work, fragmentation refers to the free time slots in between two consecutive allocations. In our first approach, a First Fit policy has been considered. This technique selects the first free gap found that fits the new job. It can create big fragmentation, as a result of which many jobs may be rejected. There also exist other techniques like Best Fit. This policy selects the free gap which leaves less free time slots after allocating the job. The created fragments are smaller, but it is harder to use those free slots to allocate new jobs. 4.2 Data Structure The data structure used to keep track of the free time slots is a key aspect. A suitable data structure yields better execution times and reduces the complexity of algorithms. Furthermore, the data structure will also influence on the scalability of the algorithm. That is the reason why the structure used in this work is red black trees [6,8]. The objective of using these type of trees is to develop techniques to efficiently identify feasible idle periods for each arriving job request, without having to examine all idle periods. This data structure is managed by the Gap Management module (see Figure 1). This module represent the information of tree data structure in a geometrical way. So, each job is represented by a single point in the plane as Figure 2 [6,8] depicts. The job coordinates are starting time and ending time. Labeled points represent the idle periods (gaps) with its start and finish time. P represents the earliest start and end times, whilst P’ represent the latests, for the current job. Thus, the line between P and P’ represents the periods when this new job can be scheduled. All the points above and to the right of this line represent possible gaps to allocate the job.


437

As Castillo explains in [8], the trees can be divided into two regions, named R1 and R2, as Figure 2 [6,8] depicts. R1 region represents the gaps which start at or before the job’s ready time. Therefore, any idle period in this region can accommodate the new job without delaying its execution. R2 region represents the gaps which start later than the job’s ready time. A job scheduled in an idle period will create at most two new idle periods: one between the begining of the gap and the start of the job (the leading idle period), and one after the end of the job and the end of the original idle period (the trailing idle period). The leading idle period will have zero length at any point in the region R2, since the start time of this gap is later than the job start time. So, this region is searched first. Work on studying other ways of searching the regions is among the future work. 4.3 Predictor Predictions of job execution time are quite difficult to obtain since there are performance differences between Grid resources and their performance characteristics may vary for different applications. Techniques for such predictions include applying statistical models to previous executions [22] and heuristics based on job and resource characteristics [11,23]. Based on this, the algorithm proposed by Castillo [6,8] is extended to take into account the heterogeneity of Grid resources. In the present work, estimations for the duration of jobs are calculated based on (1) using the Total Completion Time of jobs (TCT), and (2) Execution and Transfer Times Separately (ETTS). In (1), the mean completion time of previous executions of similar applications on the selected host is used to manage idle/busy periods on the resource. This may lead to poor resource usage since a resource may be considered as busy when in fact the job is being transferred – the job execution has not started yet. Besides, in (2), separate execution and transfer times are used, which improves the resource usage. For the execution time, an estimation is calculated similarly to the completion time in (1) – this includes execution and queueing times. Regarding transfer times, the mean bandwidth of the day before for the time interval in which the job will be allocated is calculated. Using this information, along with the total number of bytes to transfer, the time needed to complete the transfers is estimated. Estimating execution and transfer times separately yields more accurate predictions, which in turn lead to better utilization of resources and better QoS delivered to users. This is because the meta-scheduler knows for each time-slot if a job is actually being executed at a resource or being transferred to it, which allows the meta-scheduler to manage idle/busy periods of computing resources more efficiently. Thus, both techniques pay attention to the heterogeneity of Grid resources and do not assume that users have a priori knowledge on the duration of jobs, as assumed in [6,8]. These two ways of estimating the completion time of jobs are presented and evaluated in this paper. Also, in both cases, predictions are only calculated when a suitable gap has been found in the host, so that there is no need to calculate the completion times for all the hosts in the system – which would be quite inefficient. Please note that two applications are considered to belong to the same application type when they have the same input and output parameters – in terms of number, type and size.

438

L. Tomás et al.

5 Experiments and Results This section describes the experiments conducted to test the usefulness of this work, along with the results obtained. 5.1 Testbed Description The evaluation of the scheduling in advance implementation has been carried out in a local real Grid environment (depicted in Figure 3). The testbed is made of resources located in two different buildings belonging to The University of Castilla La-Mancha (UCLM). In one building there are, on the one hand, one machine which carries out the scheduler tasks, and on the other hand, several computational resources. In the second building, there is a cluster machine with 88 cores. All these machines belong to the same administrative domain (UCLM) but they are located in different subnets. Notice that these machines belong to other users, so they have their own local background load. 5.2 Workload Used One of the GRASP [25] benchmarks, named 3node, has been run to evaluate the implementation. The 3node test consists of sending a file from a “source node” to a “computation node”, which performs a search pattern, generating an output file with the number of successes. The output file is sent to the “result node”. This test is meant to mimic a pipelined application that obtains data at one site, computes a result on that data at another, and analyses the result on a third site. Furthermore, this test has parameterizable options to make it more computing intensive (compute scale parameter) and / or more network demanding (output scale parameter). There are important parameters that have to be considered in the workload used for measuring performance, as can be seen in Figure 4. “T max reservation” represents the advance with which we can make an scheduling in advance; “T Execi ” is the time needed to execute the job i; “Scheduling Window” shows the time interval in which the job has to be scheduled; “Arrival Ratio” depicts the average time between two jobs

Fig. 3. Grid testbed topology


439

Fig. 4. Workload characteristic

sent; and “Laxity” represents how strict the user is when scheduling a job, which is the difference between the “Scheduling Window” and the “T Exec” for a job. For this evaluation, both the compute scale and the output scale take values between 0 and 20, being the average 10. The T max reservation is up to 1 hour, with an average of 30 minutes. The Laxity is set between 0 and 10 minutes, being the average 5 minutes. The submission rate is from 1 to 4 jobs per minute. Finally, time slots last 1 minute. The results shown are the average of 5 executions for each case. 5.3 Performance Evaluation In this section, a comparison between the SA-layer with both techniques of calculating the job execution time and a straight-forward implementation of the algorithm proposed by Castillo et al. [6,8] is outlined. Our proposal has already been compared with the GridWay original framework (without support for scheduling in advance), resulting in a performance gain [26]. Now, this paper compares our framework with the original framework developed by Castillo et al. [6,8], and shows (1) there is no need to have an a priori knowledge on the execution time of jobs – as long as predictions can be made; and (2) the importance of performing estimations on transfer and execution times separately, rather than taking both parameters altogether. To evaluate the importance of using network information in the meta-scheduling process, several statistics can be used. % of Scheduled job rate is the fraction of accepted jobs, i.e., those whose deadline can be met [8]. % of QoS not fulfilled means the number of jobs rejected, plus the number of jobs that were initially accepted but their executions were eventually delayed. Thus, their QoS agreements were not fulfilled (the deadline was not met). These are measures of the QoS perceived by the user. From the point of view of the meta-scheduling system, there is another statistic, namely waste. It records the number of minutes inside reservations (not physical reservation) that were not used since the meta-scheduler thought that jobs would need more time to complete their executions. This statistic is related to the accuracy of predictions. Results from the user and system point of view are depicted in Figures 5 and 6, respectively. In these plots, estimations on the Total Completion Time are labelled as TCT, while estimations with Executions and Transfer Times Separately are labelled as ETTS. The results obtained when using the algorithm proposed by Castillo et al. [6,8] are labelled as Castillo. First, Figure 5 (a) represents the percentage of scheduled jobs (the meta-scheduler has enough free slots to allocate them, meeting the QoS requirements). The more jobs

440

L. Tomás et al.

80 60 40 20 0

Castillo TCT ETTS

30 % of QoS Not Fulfilled

% of Scheduled Job

35

Castillo TCT ETTS

100

25 20 15 10 5

1

2

3

0

4

1

Number of Submitted Jobs per minute

2

3

4


(a) Jobs Scheduled

(b) QoS not Fulfilled

Fig. 5. Estimating total completion time or separate estimations for execution and transfer times

Mean Waste per Job (in sec.)

160

Castillo TCT ETTS

140 120 100 80 60 40 20 0

1

2

3

4


Fig. 6. Estimations waste time

there are in the system, the more lost jobs there are. All the algorithms have a similar behaviour at low loads. The differences appear when the system load becomes close to saturation (3 jobs/min.). Notice that when using ETTS, lost jobs appear only when the load goes over 3 jobs/min. Moreover, the loss rate in this case is very small compared with the other two techniques. At this load, Castillo lost around 31 % of the scheduled jobs but ETTS only around 7.5 %. So, using ETTS to estimate execution times yields better results, since it can accept more jobs because the estimation is more accurate. Figure 5 (b) shows the percentage of jobs that were not executed with the QoS requested, and includes lost jobs and jobs completed beyond the deadline. Again, the more jobs there are in the system, the more jobs not executed with the requested QoS there are. For lower submission rates (1 and 2 jobs/min.), it is not essential to make separate estimations for executions and transfer times since there are enough free slots. Thus, reserving the number of slots in such a tight way does not show any enhancement. However, for higher submission rates (3 and 4 jobs/min.) it becomes very important to make this difference (ETTS line) since a noticeable reduction in the number of lost jobs is achieved. Figure 6 depicts the mean waste times in calculating the job completion time estimations. This graphic highlights that even having a greater number of running jobs, the waste is lowest when using ETTS than when using TCT, that is, without estimating


441

execution and network times separately in the gap reservation. With lower waste time, more jobs can be accepted since each accepted job requires fewer reserved slots. This explains the results showed in Figure 5 (a). Also, resource utilization is better since there is less wasted time in between executions of jobs. So, the resource will be idle for less time.

6 Conclusions and Future Work Several research works aim at providing QoS in Grids by means of advanced reservations. However, making reservations of resources is not always possible. Sometimes not all the LRMS permit reservations, while in other cases not all the resources belong to the same administrative domain. There are even other types of resources which may belong to several domains at the same time, such as network bandwidth. So, we proposed scheduling in advance (first step of advanced reservation) as a possible solution to provide QoS to Grid users. This type of scheduling allows to estimate whether a given application can be executed before the deadline specified by the user. But this requires to tackle many challenges, such as developing efficient scheduling algorithms that scale well or how to predict the jobs execution time. For this reason, the prediction of the Grid resources status is essential. In this work, a comparison between using estimations on the Total Completion Time (TCT) and Execution and Transfer Times Separately (ETTS) is presented. Also, both techniques are compared with an implementation of the scheduling in advance algorithm proposed by Castillo et al. [6,8]. This comparison highlights the importance of calculating network estimations independently because it improves the resource usage, thus allowing more jobs to be scheduled. Finally, recall that both the meta-scheduling in advance and advanced reservation in Grid environments are open fields that still need research since there are no definitive solutions (in terms of scalability and / or efficiency). Besides, many of the ideas developed to provide QoS in Grids have been evaluated in simulated environments, but our work is being carried out under a real environment. The development and implementation of new efficient and scalable algorithms is one of the challenges of this research. So, among the future work we are planning to include new parameters such as trust of resources. This parameter could be measured as the historical percentage of jobs assigned to a computational resource that do not get their QoS requirements. Another challenge is work on jobs rescheduling to provide the specified QoS in Grid. When the scheduler fails to allocate a job, it is possible to allocate new incoming jobs by rescheduling existing already scheduled jobs whenever possible without affecting their QoS (Replanning Capacity [6]).

Acknowledgments This work was supported by the Spanish MEC and MICINN, as well as European Commission FEDER funds, under Grants “CSD2006-00046” and “TIN2009-14475-C04”. It was also partly supported by JCCM under Grants “PBI08-0055-2800” and “PII1C090101-9476”.

442

L. Tomás et al.

References 1. Yahyapour, R.: Considerations for resource brokerage and scheduling in Grids. In: Proc. of Parallel Computing: Software Technology, Algorithms, Architectures and Applications (PARCO), Dresden, Germany (2003) 2. Huedo, E., Montero, R.S., Llorente, I.M.: A modular meta-scheduling architecture for interfacing with pre-WS and WS Grid resource management services. Future Generation Computing Systems 23(2), 252–261 (2007) 3. Elmroth, E., Tordsson, J.: An interoperable, standards-based grid resource broker and job submission service. In: Proc. of the 1st Intl. Conference on e-Science and Grid Computing (e-Science), Washington, DC, USA (2005) 4. Sulistio, A.: Advance Reservation and Revenue-based Resource Management for Grid Systems. PhD thesis, Department of Computer Science and Software Engineering, The University of Melbourne, Australia (2008) 5. GWD-I, Global Grid Forum (GGF): Advance reservations: State of the art. J. MacLaren (2003), http://www.ggf.org 6. Castillo, C., Rouskas, G.N., Harfoush, K.: Efficient resource management using advance reservations for heterogeneous grids. In: Proc. of the Intl. Parallel and Distributed Processing Symposium (IPDPS), Miami, USA (2008) 7. Siddiqui, M., Villazón, A., Fahringer, T.: Grid capacity planning with negotiation-based advance reservation for optimized QoS. In: Proc. of the 2006 Conference on Supercomputing (SC 2006), Tampa, USA (2006) 8. Castillo, C., Rouskas, G.N., Harfoush, K.: On the design of online scheduling algorithms for advance reservations and QoS in grids. In: Proc. of the Intl. Parallel and Distributed Processing Symposium (IPDPS), Los Alamitos, USA (2007) 9. Tomás, L., Caminero, A., Caminero, B., Carrión, C.: Studying the Influence of NetworkAware Grid Scheduling on the Performance Received by Users. In: Proc. of the Grid computing, high-PerformAnce and Distributed Applications (GADA), Monterrey, Mexico (2008) 10. Tomás, L., Caminero, A., Caminero, B., Carrión, C.: Improving GridWay with Network Information: Tuning the Monitoring Tool. In: Proc. of the High Performance Grid Computing Workshop (HPGC), Hold Jointly with the Intl. Parallel & Distributed Processing Symposium (IPDPS), Roma, Italy (2009) 11. Caminero, A., Rana, O., Caminero, B., Carrión, C.: Performance evaluation of an autonomic network-aware metascheduler for Grids. Concurrency and Computation: Practice and Experience 21(13), 1692–1708 (2009) 12. Tanwir, S., Battestilli, L., Perros, H.G., Karmous-Edwards, G.: Dynamic scheduling of network resources with advance reservations in optical grids. Int. Journal of Network Management 18(2), 79–105 (2008) 13. Farooq, U., Majumdar, S., Parsons, E.W.: Efficiently scheduling advance reservations in grids. Technical report, Carleton University, Department of Systems and Computer Engineering (2005) 14. The Globus Alliance (2009), http://www.globus.org 15. Legion Project (2009), http://legion.virginia.edu/ 16. Roy, A., Sander, V.: GARA: A Uniform Quality of Service Architecture. In: Grid Resource Management, pp. 377–394. Kluwer Academic Publishers, Dordrecht (2003) 17. Maui Cluster Scheduler (2009), http://www.clusterresources.com/products/maui/ 18. Waldrich, O., Wieder, P., Ziegler, W.: A meta-scheduling service for co-allocating arbitrary types of resources. In: Proc. of the 6th Intl. Conference on Parallel Processing and Applied Mathematics (PPAM), Poznan, Poland (2005)


443

19. Qu, C.: A grid advance reservation framework for co-allocation and co-reservation across heterogeneous local resource management systems. In: Proc. of 7th Intl. Conference on Parallel Processing and Applied Mathematics (PPAM), Gdansk, Poland (2007) 20. Smith, W., Foster, I., Taylor, V.: Scheduling with advanced reservations. In: Proc. of the 14th Intl. Parallel and Distributed Processing Symposium (IPDPS), Washington, DC, USA (2000) 21. Wieczorek, M., Siddiqui, M., Villazon, A., Prodan, R., Fahringer, T.: Applying Advance Reservation to Increase Predictability of Workflow Execution on the Grid. In: Proc. of the 2nd Intl. Conference on e-Science and Grid Computing (e-Science), Washington, DC, USA (2006) 22. Dinda, P.A.: The statistical properties of host load. Scientific Programming 7(3-4), 211–229 (1999) 23. Jin, H., Shi, X., Qiang, W., Zou, D.: An adaptive meta-scheduler for data-intensive applications. Intl. Journal of Grid and Utility Computing 1(1), 32–37 (2005) 24. Zhang, Y., Sun, W., Inoguchi, Y.: Predict task running time in grid environments based on CPU load predictions. Future Generation Computing Systems 24(6), 489–497 (2008) 25. Chun, G., Dail, H., Casanova, H., Snavely, A.: Benchmark probes for grid assessment. In: Proc. of 18th Intl. Parallel and Distributed Processing Symposium (IPDPS), Santa Fe, New Mexico (2004) 26. Tomás, L., Caminero, A., Carrión, C., Caminero, B.: Meta-Scheduling in Advance using Red-Black Trees in Heterogeneous Grids. In: Proc. of the High Performance Grid Computing Workshop (HPGC), Hold Jointly with the Intl. Parallel & Distributed Processing Symposium (IPDPS), Atlanta, USA (2010)

Peer-to-Peer Computing Adriana Iamnitchi1 , Paolo Trunfio1 , Jonathan Ledlie2 , and Florian Schintke2 1


2

After several years of intensive investigation, peer-to-peer computing has established itself as an accepted research topic in the general area of distributed systems. Going beyond the initial file sharing applications that spurred productive research and developement, peer-to-peer computing is associated with inherently decentralized, self-organizing, and self-coordinating large-scale systems. Performance requirements include adaptivity to churn, high resilience to failures, tolerance to network performance variations, and scalability to huge numbers of peers (tens of thousands to millions), but also stronger consistency and security. This year, fourteen papers were submitted to the Peer-to-Peer Computing track. Each paper was reviewed by at least three reviewers and, finally, we were able to select five regular papers. Paper “Dynamic publish/subscribe to meet subscriber-defined delay and bandwidth constraints” by M. A. Tariq, G. G. Koch, B. Koldehofe, I. Khan and K. Rothermel, proposes a protocol for publish/subscribe systems in which subscribers are served according to their specified message delay tolerance. In such system, subscribers with tight delay bounds are served first and then forward messages to those with lesser requirements. In “Combining Hilbert SFC and Bruijn Graphs for Searching Computing Markets in a P2P System” by D. Castellà, H. Blanco, F. Giné and F. Solsona, a resource discovery system for multiple-attribute, range queries in a P2P system is proposed. The solution combines Bruijn graphs with N-trees and Hilbert multi-dimensional space-filling curve functions. In “Overlay Management for Fully Distributed User-based Collaborative Filtering” by R. Orm´ andi, I. Hegedus and M. Jelasity, the problem of constructing an overlay network for performing collaborative filtering is addressed. The proposed system builds a network in which similar peers are connected via a directed edge based on a similarity metric. Paper “A Formal Credit-Based Incentive Model for Sharing Computer Resources” by J. Rius, I. Barri, F. Cores and F. Solsona, proposes an incentive mechanism for increasing peer participation. The mechanism includes a reinvestment policy that distributes credits non-uniformly to participating peers based on their contribution to the network. Finally, in “Sampling Bias in BitTorrent Measurements” by B. Zhang, A. Iosup, J. A. Pouwelse, D. Epema and H. Sips, a large study on different BitTorrent traces gathered over the last six years is presented. The paper analyzes different strategies of obtaining the traces and whether the methods will lead to a bias in the gathered data. P. D’Ambra, M. Guarracino, and D. Talia (Eds.): Euro-Par 2010, Part I, LNCS 6271, pp. 444–445, 2010. c Springer-Verlag Berlin Heidelberg 2010

Peer-to-Peer Computing

445

We would like to take the opportunity of thanking the authors who submitted a contribution, as well as the Euro-Par Organizing Committee, and the Referees with their highly useful comments, whose efforts have made this conference and this topic possible.

Overlay Management for Fully Distributed User-Based Collaborative Filtering Róbert Ormándi1, István Heged˝us1 , and Márk Jelasity2

2

1 University of Szeged, Hungary {ormandi,ihegedus}@inf.u-szeged.hu University of Szeged and Hungarian Academy of Sciences, Hungary [email protected]

Abstract. Offering personalized recommendation as a service in fully distributed applications such as file-sharing, distributed search, social networking, P2P television, etc, is an increasingly important problem. In such networked environments recommender algorithms should meet the same performance and reliability requirements as in centralized services. To achieve this is a challenge because a large amount of distributed data needs to be managed, and at the same time additional constraints need to be taken into account such as balancing resource usage over the network. In this paper we focus on a common component of many fully distributed recommender systems, namely the overlay network. We point out that the overlay topologies that are typically defined by node similarity have highly unbalanced degree distributions in a wide range of available benchmark datasets: a fact that has important—but so far largely overlooked—consequences on the load balancing of overlay protocols. We propose algorithms with a favorable convergence speed and prediction accuracy that also take load balancing into account. We perform extensive simulation experiments with the proposed algorithms, and compare them with known algorithms from related work on wellknown benchmark datasets.

1 Introduction Offering useful recommendations to users of fully distributed systems is clearly a desirable function in many application domains. Some examples for larger efforts towards this goal are the Tribler platform [1] and more recently the Gossple project [2]. A fully distributed approach is also more preferable relative to centralized solutions, due to the increasing concerns over privacy. However, the problem is also extremely challenging. Apart from the fact that centralized recommender systems—although working reasonably sometimes—are still far from perfect, offering good recommendations in fully distributed systems involves a number of special problems like efficiency, security and reliability, to name just a few. In this work we focus on a class of recommender systems, the so called user-based collaborative filtering algorithms that are fairly simple, yet provide a reasonable performance [3]. The key concept is a similarity metric over the users, and recommendations are made on the basis of information about similar users.

M. Jelasity was supported by the Bolyai Scholarship of the Hungarian Academy of Sciences. This work was partially supported by the Future and Emerging Technologies programme FP7COSI-ICT of the European Commission through project QLectives (grant no.: 231200).


Overlay Management for Fully Distributed User-Based Collaborative Filtering

447

This idea also naturally lends itself to a distributed implementation, as it can be easily supported by similarity-based overlay networks as a simple service, that also have applications in other domains such as search. Indeed, many distributed protocols from related work follow this path in some way or another. In this work we would like to shed light on the effects of the basic design choices in this domain with respect to recommendation performance, convergence time, and the balancing of the network load that the system generates during its operation. Our contribution is threefold. First, we draw attention to the potential load balancing problem in distributed systems that manage similarity-based overlays for any purpose including recommendation or search. Second, we propose novel algorithms for similarity-based overlay construction. Third, we perform extensive simulation experiments on large benchmark datasets and compare our set of algorithms with each other and with a number of baselines. We measure prediction performance, examine its convergence and dynamics, and we measure load balancing as well.

2 Related Work First we overview relevant ideas in recommender systems in general, and subsequently we discuss related work in the fully distributed implementations of these ideas, as well as additional related work that are based on similar abstractions. A recommender system can be viewed as a service which supports e-commerce activities by providing items of interest for the users [4]. These algorithms are often centralized and Web-based operating on huge amounts of data—mainly on the previous ratings of the users. The algorithms which are based on the previous ratings of other similar users follow the so-called collaborative filtering (CF) approach. They are based on the simple heuristic that people who agreed (or disagreed) in the past will probably agree (or disagree) again. Thus, the predicted rate of an unseen item for a given user can be estimated on the basis of the rates of other users with similar tastes. In the field of CF algorithms there exist numerous approaches. User-based approaches try to model the rating of a given item for a user by an aggregation of ratings of other users on the same item [3]. Although these approaches are very simple and intuitive, they provide a relatively good performance [5]. User-based CF algorithms are modular, hence they can be used with different aggregation methods and similarity metrics. One widely-used aggregation method is su,v (rv,i − r¯v ) u + r¯u (1) rû,i = v∈N v∈Nu |su,v | defined in [6], where ru,i and rû,i denote the known and the predicted rate of item i by user u, r¯u and Nu denote the average rate and the neighbor set of user u, and su,v measures the similarity between user u and v (e.g. Cosine similarity [3] or Pearson similarity [3] can be employed). Our preliminary experiments showed that (among several variants) the aggregation method in (1) combined with the Cosine user similarity gives the best performance on our particular benchmarks. Since the focus of the present work is not recommendation performance per se, but the analysis of several distributed implementations of the basic idea of user-based CF, we fixed these methods in our experiments.

448

R. Ormándi, I. Heged˝us, and M. Jelasity

We should mention that there are numerous other approaches for recommendation such as the ones based on machine learning [7,8], matrix factorization [9], generative models [10], clustering [11,8], and dimension-reduction [7,12]. Moving on to distributed methods, we emphasize that we focus on P2P recommendation, and not on parallel implementations of centralized recommender techniques (such as matrix factorization, etc.). We consider only works that go beyond a simple idea and present at least some evaluations on benchmarks. The largest group of methods define an overlay network based on some sort of similarity, and define a recommender algorithm on this network. For example, [4] and [13] follow this approach, although the overlay construction itself is not discussed or it is assumed to be done offline. The recommender algorithms then perform a search in this overlay up to a certain depth or up to a certain level of similarity, and aggregate the matching users with a standard method. A slightly weaker approach is described in [14], where only a random network is assumed and the recommendation problem is treated as a search problem where a node needs to find similar users using a flooding based unstructured search. A somewhat surprising result is described by Bakker at al [15], where they argue that in fact it is enough to take a random sample of the network and use the closest elements of that sample to make recommendations. Our results are consistent with this observation, although we describe better and equally cheap alternatives. A more sophisticated approach is described by Bickson et al [16]. They define recommendation as a smoothing operation over a social network, which is expressed as a minimization problem using an objective function that expresses the requirements for the recommendation. The problem is solved by using an iterative method. Unfortunately no results are given on recommender system benchmarks due to the slightly different formulation of the basic problem. It is of course possible to apply distributed hash tables [17]. Here, users are stored in a hash table and they are indexed by (item, rate) pairs as keys. Using this data structure, the users for a given item and rate are available from the distributed hash table (DHT) on demand. This method is not scalable if there are many recommendations to be made in the system, since the necessary information is not always available locally. One of the most detailed studies on distributed recommender systems with performance evaluation can be found in [18]. The proposed models were implemented on the basis of the B UDDY C AST [19] overlay management service, which is the main overlay management method of the Tribler file sharing protocol [1]. We used our own implementation of this model as a baseline method, since the original study [18] did not carry out load balancing measurements. Finally, although not directly related to the recommender systems, the area of exploiting semantic proximity for search also involves building overlay networks based on node similarity and therefore our algorithms and observations are relevant in this area as well. Examples of research in this area are described in [2,20,21,22].

3 Interesting Properties of CF Datasets In our simulations we applied three different benchmark datasets, namely the MovieLens [5] dataset, the Jester [12] dataset and the BookCrossing [23] dataset. In this section we introduce these benchmarks and show some of their properties that raise interesting—and so far largely overlooked—problems in distributed environments.


449

Table 1. Basic statistics of datasets MovieLens Jester BookCrossing # users 71,567 73,421 77,806 # items 10,681 100 185,974 size of train 9,301,274 3,695,834 397,011 sparsity 1.2168% 50.3376% 0.0027% size of eval 698,780 440,526 36,660 eval/train 7.5127% 11.9195% 9.2340% # items ≥ 20 15 1 rate set 1, . . . , 5 −10, . . . , 10 1, . . . , 10 MAE(med) 0.93948 4.52645 2.43277

Table 1 summarizes some basic statistics of our datasets. In the case of MovieLens we used the official ra partition so that its evaluation set contained 10 ratings per user. For Jester and BookCrossing we produced the evaluation set as proposed in [15]: we withheld 6 ratings from the training set where possible (if the user under consideration had at least 6 rated items). In this table ’# items ≥’ means the minimal number of items rated by some user. Sparsity denotes the ratio of existing and possible rates in the training sets. The value MAE(med) is a trivial baseline for prediction performance; it is defined as the mean absolute error (MAE) computed on the evaluation set using the median-rate of training set as a prediction value. Clearly, a very significant difference can be found in properties related to sparsity. This will have significant implications on the performance of our algorithms, as we show later. As mentioned before, in distributed settings one suitable and popular approach is to build and manage an overlay that connects similar users. This overlay can be viewed as a graph where each node corresponds to a user and there is a directed edge between user A and B if and only if user B belongs to the most similar users of A. This overlay plays an important role in a P2P recommender system. First, the performance of the recommendation depends on the structure of the overlay. Second, the costs and load balancing of the overlay management protocol depend on the topology of this similarity network. To the best of our knowledge, the second role of the similarity overlay has not been addressed so far in the literature. Nevertheless it is an important issue, since the load generated by the overlay management process might correlate with the number of nodes that link to a given node as one of its most similar nodes. More precisely, the load of a node might correlate with its in-degree in the overlay network. Thus, if the in-degree distribution of the overlay network is extremely unbalanced (e.g. if it has a power-law distribution), some of the nodes can experience a load that is orders of magnitudes higher than the average. Thus, it is very important to consider the in-degree distribution of the overlay when planning a P2P recommender system, and examine the incurred loads on the individual nodes as a function of this distribution. Figure 1 shows the in-degree distributions of the k nearest neighbor (kNN) overlay of each benchmark dataset. In this overlay each node has k directed outgoing edges to the k most similar nodes. As can be seen from the plots, the BookCrossing dataset has an almost power-law in-degree distribution, with many nodes having incoming links from almost every other node (note that the size of this dataset is around 77,806 users).

450


MovieLens 1000

Jester 1000

k=100 k=200

BookCrossing 10000

k=100 k=200

k=100 k=200

10

100

frequency

frequency

frequency

1000 100

100

10 10

1

1 1

10

100 1000 in-degree

10000

100000

1 1

10

100 1000 in-degree

10000

100000

1

10

100 1000 in-degree

10000

100000

Fig. 1. In-Degree Distribution of Benchmark Datasets

To see whether this might be a general property of high dimensional datasets, we need to consider some basic properties of high dimensional metric spaces. If we generate high dimensional uniform random datasets from the unit cube and construct their kNN graphs, we will find that most of the points lie on the convex hull of the dataset. These points are mostly situated at the same distance from each other. The nodes corresponding to these points have a mostly uniform and relatively small in-degree in the kNN graph. The very few points inside the convex hull are close to a huge number of points on the convex hull, and so have high in-degree. These observations indicate that we have to explicitly take into account load balancing when building a recommender system in a fully distributed manner.

4 Algorithms The algorithms we examine all rely on building and managing a user-similarity overlay. In the top level of the protocol hierarchy, they apply the same user-based CF algorithm for making recommendations, strictly using locally available information (that is, information about the neighbors in the overlay). Since we focus on overlay management, we fix the recommender algorithm and not discuss it any further. As it was mentioned in the previous sections, for this we need an aggregation method and a user similarity metric. We selected the aggregation shown in (1), proposed in [6]. Our similarity metric is Cosine similarity, which achieved the best performance on our benchmarks. Note that the selected user similarity is of course known to the overlay management algorithm and is used to direct the overlay construction. We also assume that the local views of the nodes contain not only the addresses of the neighbors, but also a descriptor for each neighbor, that contains ratings made by the corresponding user. This implies that computing recommendation scores do not load the network since all the necessary information is available locally. However, there is a drawback; namely the stored information is not up-to-date. As we will show later, this is not a serious problem since on the one hand, recommendation datasets are not extremely dynamic and, on the other hand, the descriptors are in fact refreshed rather frequently due to the management algorithms. In sum, the task of overlay management is to build and maintain the best possible overlay for computing recommendation scores, by taking into account bandwidth usage


451

Algorithm 1. Random Nodes based Overlay Management Parameters: k: the size of view; r: the number of randomly generated nodes 1. while true do 2. samples ← getRandomPeers(r) 3. for i = 1 to r do 4. peer ← get(samples, i) 5. peerDescriptor ← descriptor(peer) 6. insert(view, peerDescriptor)

at the nodes. We expect a minimal, uniform load from overlay management even when the in-degree distribution of the expected overlay graph is unbalanced. 4.1

B UDDY C AST Based Recommendation

As we mentioned earlier we applied the BUDDY C AST overlay management protocol as a baseline method. Now we give a very brief overview of this algorithm and its numerous parameters; for details please see [19]. The algorithm maintains a number of lists containing node descriptors. The taste buddy list contains the most similar users (peers), all those who communicated with the node before. The recommendation for a peer is calculated based on this list. The B UDDY C AST algorithm contains a mechanism for load balancing: a block list. Communication with a peer on the block list is not allowed. If a node communicates with another peer, it is put on the block list for four hours. Finally, a node also maintains a candidate list, which contains close peers for potential communication, as well as a random list that contains random samples from the network. For overlay maintenance, each node periodically (in every 15 seconds by default) connects to the best node from the candidate list with probability α, and to a random list with probability 1 − α, and exchanges its buddy list with the selected peer. 4.2 kNN Graph from Random Samples We assume that a node has a local view of size k that contains node descriptors. These will be used by the recommender algorithm. In Algorithm 1 each node is initialized with k random samples from the network, and they iteratively approximate the kNN graph. The convergence is based on a random sampling process which generates r random nodes from the whole network in each iteration. These nodes are inserted into the view which is implemented as a bounded priority queue. The size of this queue is k and the priority is based on the similarity function provided by the recommender module. Applying a priority queue here on the basis of similarities means that nodes remember the most similar nodes from the past iterations. This means that since random samples are taken from the entire network, each node will converge to its kNN view with positive probability. Method GET R ANDOM P EERS can be implemented, for example, using the N EWS C AST [24] protocol. This algorithm does converge, as argued above, albeit very slowly. However, it is guaranteed to generate an almost completely uniform load since the only communication that takes place is performed by the underlying peer sampling implementation (N EWS C AST), which has this property.

452


4.3 kNN Graph by T-M AN We can manage the overlay with the T-M AN algorithm as well [25]. This algorithm manages a view of size k, as in the random algorithm above. T-M AN periodically updates this view by first selecting a peer node to communicate with, then exchanging its view with the peer, and finally merging the two views and keeping the closest k descriptors. This is very similar to Algorithm 1, but instead of r random samples the update is performed using the k elements of the view of the selected peer. In this paper we examine the following methods for T-M AN which are employed as peer selection methods: Global: This approach selects the node for communication from the whole network randomly. This can be done by using a N EWS C AST layer as it was described in the previous section. We expect this approach to distribute the load in the network uniformly since with this selection the incoming communication requests do not depend on the in-degree of the kNN graph at all. View: In this approach the node for communication is selected from the view of the current node uniformly at random. The mechanism of this selection strategy is similar to the previous one, but the spectrum of the random selection is smaller since it is restricted to the view instead of the whole network. Proportional: This approach also selects a node for view exchange from the view of the current node, but here we define a different probability distribution. This distribution is different for each node and it is reversely proportional to the value of a selection counter, which measures the load of the node in the previous time interval. The exact definition of the selection probability for a neighbor j of node i is pi,j =

1 selj +1

1 k∈V iewi selk +1

,

(2)

where selk is the value of the selection counter of the kth neighbor. This information is stored in the node descriptors. The motivation for this selection method is to reduce the load on the nodes that have a high in-degree in the kNN graph, while maintaining the favorable convergence speed of the T-M AN algorithm. Best: The strategy that selects the most similar node for communication without any restriction. We expect that this strategy converges the most aggressively to the perfect kNN graph, but at the same time it results in the most unbalanced load. 4.4 Randomness Is Sometimes Better Our experimental results (to be presented in Section 6) indicated that in certain cases it is actually not optimal to use the kNN view for recommendation. It appears to be the case that a more relaxed view can give better recommendation performance. To test this hypothesis, we designed a randomization technique that is compatible with any of the algorithms above. The basic idea is that we introduce an additional parameter, n ≤ k. The nodes still have a view of size k, and we still use the same recommender algorithm based on these k neighbors. However, we apply any of the algorithms above to construct a (k-n)NN overlay graph (not a kNN graph), and we fill the remaining n elements in the following way: we take r ≥ n random samples (not necessarily independent in each cycle) and we take the closest n nodes from this list. With n = k we get the algorithms proposed in [15], and with n = 0 this modification has no effect, so we get the original algorithm for constructing the kNN graph.


453

5 System Model We consider a set of nodes connected through a routed network. Each node has an address that is necessary and sufficient for sending a message to it. To actually communicate, a node has to know the address of the other node. This is achieved by maintaining a partial view (view for short) at each node that contains a set of node descriptors. Views can be interpreted as sets of edges between nodes, naturally defining a directed graph over the nodes that determines the topology of an overlay network. Although the class of algorithms we discuss has been shown to tolerate unpredictable message delays and node failures well [25,24], in this work we focus on load balancing and prediction performance, so we assume that messages are delivered reliably and without delay, and we assume that the nodes are stable. Finally, we assume that all nodes have access to the peer sampling service [24] that returns random samples from the set of nodes in question. We will assume that these samples are indeed random. The results presented in [24] indicate that the peer sampling service has realistic implementations that provide high quality samples at a low cost.

6 Empirical Results We implemented our protocols and performed our experiments in PeerSim [26,27]. We performed a set of simulations of our algorithms with the following parameter value combinations: view update is random or T-M AN; peer selection for T-M AN is GLOBAL , VIEW, BEST or PROPORTIONAL ; and the number of random samples is 20, 50, or 100 for random, and 0 or 100 for T-M AN. The B UDDY C AST algorithm was implemented and executed with the following parameters: the size of the buddy list and the candidate list was 100, the size of the random list was 10, and α was 0.5. The size of the block list had to be restricted to be 100 as well, in order to be able to run our large scale simulations. The view size for the rest of the protocols was fixed at k = 100 in all experiments for practical reasons: this represents a tradeoff between a reasonably large k and the feasibility of large scale simulation. In these simulations we observe the prediction performance in terms of the MAE measure and the distribution of the number of incoming messages per cycle at a node. Note that the number of outgoing messages is exactly one in each case. Let us first discuss the effect of parameter r. This is a crucial parameter for random view update, while in the case of T-M AN the role of random samples is merely to help MovieLens, random view update (k=100) 0.84 0.83

0.82

0.81

0.81

0.8

0.8

0.79

0.79

0.78

0.78

0.77

best, r=100 best, r=0 view, r=100 view, r=0

0.83

0.82

MAE

MAE

MovieLens, T-Man (k=100) 0.84

r=20 r=50 r=100

0.77

0.76

0.76 1

10

100

1

10

cycles

Fig. 2. Effect of parameter r in a few settings

100 cycles

454


T-Man best T-Man proportional T-Man view T-Man global random kNN overlay BuddyCast

0.83 0.82

MAE

0.81 0.8 0.79 0.78

MovieLens (k=100, r=100)

maximal load (contacts)

MovieLens (k=100, r=100) 0.84

0.77

T-Man best

T-Man proportional 100

T-Man view BuddyCast

T-Man global, random

10

0.76 10

1

10

100 cycles

Jester (k=100, r=100)

Jester (k=100, r=100)

3.54

T-Man best T-Man proportional T-Man view T-Man global random kNN overlay BuddyCast

3.52 3.5 3.48

MAE

100 cycles

3.46 3.44 3.42


1

100

T-Man best T-Man proportional T-Man view

BuddyCast T-Man global, random

10

3.4 3.38 1

10

100

1

10

cycles BookCrossing (k=100, r=100) T-Man best T-Man proportional T-Man view T-Man global random kNN overlay BuddyCast

2.5 2.45

MAE

2.4 2.35 2.3 2.25 2.2

BookCrossing (k=100, r=100)

BuddyCast


2.55

100 cycles

1000

T-Man best T-Man view

100 T-Man proportional T-Man global, random

10

2.15 1

10

100 cycles

1

10

100 cycles

Fig. 3. Experimental results. The scale of the plots on the right is logarithmic.

the algorithm to avoid local optima, and to guarantee convergence. Figure 2 shows the effect of r in the case of the MovieLens database. The effect of r on the other databases and for other settings is similar. We can observe that in the case of a random view update, r simply is a multiplicative factor that determines the speed of convergence: twice as many samples per cycle result in a halving of the necessary cycles to achieve the same value. In the case of T-M AN, the version with random samples converges faster, while the generated load remains the same (not shown). Accordingly, in the following we discuss T-M AN algorithms only with r = 100, and random view update algorithms only with r = 100. In Figure 3 we show the results of the experiments, where the MAE and the maximal load is illustrated. The maximal load is defined as the maximal number of incoming messages any node receives during the given cycle. The first interesting observation is that the load balancing property of the different algorithms shows a similar pattern over the three datasets, however, the convergence of the MAE is rather different (see also Table 1). In particular, in the case of the MovieLens and BookCrossing benchmarks the MAE reaches a minimum, after which it approaches the top-k based prediction from below, whereas we do not see this behavior in the much denser Jester database.


455

MovieLens 0.8 0.79 0.78

MAE

0.77 0.76 0.75 0.74 0.73 0.72 100

k=100, k=200, k=400, k=800,

n=50 n=100 n=200 n=400 1000 r

Fig. 4. Effect of adding randomness to the view. Thin horizontal lines show the n = 0 case

Indeed, the reason for this behavior lies in the fact that for the sparse datasets a larger k is a better choice, and our setting (k = 100) is actually far from optimal. In the initial cycles the view approximates a random sample from a larger k parameter. To verify this, we calculated the MAE of the predictions based on the algorithm described in Section 4.4. The results are shown in Figure 4 later on. It is clear that for a small k it is actually better not to use the top k from the entire network; rather it is better to fill some of the views with the closest peers in a relatively small random sample from the network. Especially for the smallest k we examined (k = 100) this technique results in a significant improvement in the MAE compared to the recommendation based on the closest k peers in all datasets. This algorithm can easily be implemented, since we simply have to combine any of the convergent algorithms with an appropriate setting for k (such as k = 50) and use a peer sampling service to add to this list the best peers in a random sample of a given size. As a closely related note, the random view update algorithms can be “frozen” in the state of minimal MAE easily, without any extra communication, provided we know in advance the location (that is, the cycle number) of the minimum. Let us assume it is in cycle c. Then we can use, for a prediction at any point in time, the best k peers out of the union of c · r random samples collected in the previous c cycles, which is very similar to the approach taken in [15]. Clearly, the fastest convergence is shown by the T-M AN variants, but these result in unbalanced load at the same time. The PROPORTIONAL variant discussed in Section 4.3 reduces the maximal load, however, only when the topology has already converged. During the convergence phase, PROPORTIONAL behaves exactly like the variant VIEW. Quite surprisingly, the best compromise between speed and load balancing seems to be GLOBAL, where the peer is selected completely at random by T-M AN. In many topologies, such as a 2-dimensional grid, a random peer possesses no useful information for another node that is far from it in the topology, so we can in fact expect to do worse than the random view update algorithm. However, in target graphs such as kNN graphs based on similarity metrics, a large proportion of the network shares useful information, namely the addresses of the nodes that are more central. On such unbalanced graphs T-M AN GLOBAL is favorable, because it offers a faster convergence than a pure random search (in fact, it converges almost as fast as the more aggressive T-M AN variants), however, the load it generates over the network is completely identical to that of random search, and therefore the maximal load is very small:

456


the maximum of N samples from a Poisson distribution with a mean of 1 (where N is the network size). In addition, the node with the maximal load is different in each cycle. Finally, we can observe that on the BookCrossing database some algorithms, especially BuddyCast and T-M AN with BEST peer selection, result in an extremely unbalanced degree distribution (note the logarithmic scale of the plot). This correlates with the fact that the BookCrossing database has most unbalanced degree distribution (see Figure 1). Even though we have not optimized the parameters of BuddyCast, this result underlines our point that one has to pay attention to the in-degree distribution of the underlying kNN graph.

7 Conclusions In this paper we tackled the problem of the construction of similarity-based overlay networks with user-based collaborative filtering as an application. We pointed out that similarity-based overlays can have a very unbalanced degree distribution, and this fact might have a severe impact on the load balancing of some overlay management protocols. The main conclusion that we can draw is that in highly unbalanced overlays (that are rather frequent among similarity-based networks) the overlay construction converges reasonably fast even in the case of random updates; or, with T-M AN, uniform random peer selection from the network. At the same time, the traditional, aggressive peer selection strategies that have been proposed by other authors should be avoided because they result in a highly unbalanced load experienced by the nodes. In sum, in this domain T-M AN with global selection is a good choice, because it has a fully uniform load distribution combined with an acceptable convergence speed, which is better than that of the random view update. However, care should be taken because this conclusion holds only in these unbalanced domains, and in fact this algorithm is guaranteed to perform extremely badly in large-diameter topologies.

References 1. Garbacki, P., Iosup, A., Doumen, J., Roozenburg, J., Yuan, Y., Brinke, T.M., Musat, L., Zindel, F., van der Werf, F., Meulpolder, M., et al.: Tribler protocol specification 2. Kermarrec, A.M.: Challenges in personalizing and decentralizing the web: An overview of GOSSPLE. In: Guerraoui, R., Petit, F. (eds.) SSS 2009. LNCS, vol. 5873, pp. 1–16. Springer, Heidelberg (2009) 3. Adomavicius, G., Tuzhilin, E.: Toward the next generation of recommender systems: A survey of the state-of-the-art and possible extensions. IEEE Trans. on Knowledge and Data Engineering 17, 734–749 (2005) 4. Pitsilis, G., Marshall, L.: A trust-enabled P2P recommender system. In: Proc. 15th IEEE Intl. Workshops on Enabling Technologies: Infrastructure for Collaborative Enterprises (WETICE 2006), pp. 59–64 (2006) 5. Herlocker, J.L., Konstan, J.A., Borchers, A., Riedl, J.: An algorithmic framework for performing collaborative filtering. In: Proc. 22nd annual Intl. ACM SIGIR Conf. on Research and development in information retrieval (SIGIR 1999), pp. 230–237. ACM, New York (1999) 6. Resnick, P., Iacovou, N., Suchak, M., Bergstrom, P., Riedl, J.: Grouplens: an open architecture for collaborative filtering of netnews. In: Proc. 1994 ACM Conf. on Computer supported cooperative work (CSCW 1994), pp. 175–186. ACM, New York (1994) 7. Billsus, D., Pazzani, M.J.: Learning collaborative information filters. In: Proc. 15th Intl. Conf. on Machine Learning (ICML 1998), pp. 46–54. Morgan Kaufmann, San Francisco (1998)


457

8. Park, Y.-J., Tuzhilin, A.: The long tail of recommender systems and how to leverage it. In: Proc. 2008 ACM Conf. on Recommender systems (RecSys 2008), pp. 11–18. ACM, New York (2008) 9. Takács, G., Pilászy, I., Németh, B., Tikk, D.: Scalable collaborative filtering approaches for large recommender systems. Journal of Machine Learning Research 10, 623–656 (2009) 10. Lawrence, N.D., Urtasun, R.: Non-linear matrix factorization with gaussian processes. In: Proc. 26th Annual Intl. Conf. on Machine Learning (ICML 2009), pp. 601–608. ACM, New York (2009) 11. O‘Connor, M., Herlocker, J.: Clustering items for collaborative filtering. In: Workshop on Recommender Systems at 22nd ACM SIGIR (1999) 12. Goldberg, K., Roeder, T., Gupta, D., Perkins, C.: Eigentaste: A constant time collaborative filtering algorithm. Information Retrieval 4(2), 133–151 (2001) 13. Castagnos, S., Boyer, A.: Modeling preferences in a distributed recommender system. In: Conati, C., McCoy, K., Paliouras, G. (eds.) UM 2007. LNCS (LNAI), vol. 4511, pp. 400– 404. Springer, Heidelberg (2007) 14. Tveit, A.: Peer-to-peer based recommendations for mobile commerce. In: Proc. 1st Intl. workshop on Mobile commerce (WMC 2001), pp. 26–29. ACM, New York (2001) 15. Bakker, A., Ogston, E., van Steen, M.: Collaborative filtering using random neighbours in peer-to-peer networks. In: Proc. 1st ACM Intl. workshop on Complex networks meet information & knowledge management (CNIKM 2009), pp. 67–75. ACM, New York (2009) 16. Bickson, D., Malkhi, D., Zhou, L.: Peer-to-Peer rating. In: Proc. 7th IEEE Intl. Conf. on Peerto-Peer Computing, 2007 (P2P 2007), pp. 211–218. IEEE Computer Society, Los Alamitos (2007) 17. Han, P., Xie, B., Yang, F., Shen, R.: A scalable P2P recommender system based on distributed collaborative filtering. Expert Systems with Applications 27(2), 203–210 (2004) 18. Wang, J., de Vries, A.P., Reinders, M.J.T.: Unified relevance models for rating prediction in collaborative filtering. ACM Trans. on Information Systems (TOIS) 26(3), 1–42 (2008) 19. Pouwelse, J., Yang, J., Meulpolder, M., Epema, D., Sips, H.: Buddycast: an operational peerto-peer epidemic protocol stack. In: Proc. 14th Annual Conf. of the Advanced School for Computing and Imaging, ASCI, pp. 200–205 (2008) 20. Voulgaris, S., van Steen, M.: Epidemic-style management of semantic overlays for contentbased searching. In: Cunha, J.C., Medeiros, P.D. (eds.) Euro-Par 2005. LNCS, vol. 3648, pp. 1143–1152. Springer, Heidelberg (2005) 21. Garbacki, P., Epema, D.H.J., van Steen, M.: A two-level semantic caching scheme for superpeer networks. In: Proc. 10th Intl. Workshop on Web Content Caching and Distribution (WCW 2005), pp. 47–55. IEEE Computer Society, Los Alamitos (2005) 22. Akavipat, R., Wu, L.S., Menczer, F., Maguitman, A.: Emerging semantic communities in peer web search. In: Proc. Intl. workshop on Information retrieval in peer-to-peer networks (P2PIR 2006), pp. 1–8. ACM, New York (2006) 23. Ziegler, C.N., McNee, S.M., Konstan, J.A., Lausen, G.: Improving recommendation lists through topic diversification. In: Proc. 14th Intl. Conf. on WWW, pp. 22–32. ACM, New York (2005) 24. Jelasity, M., Voulgaris, S., Guerraoui, R., Kermarrec, A.M., van Steen, M.: Gossip-based peer sampling. ACM Trans. on Computer Systems 25(3), 8 (2007) 25. Jelasity, M., Montresor, A., Babaoglu, O.: T-Man: Gossip-based fast overlay topology construction. Computer Networks 53(13), 2321–2339 (2009) 26. Montresor, A., Jelasity, M.: Peersim: A scalable P2P simulator. In: Proc. Ninth IEEE Intl. Conf. on Peer-to-Peer Computing (P2P 2009), pp. 99–100. IEEE, Los Alamitos (2009) (extended abstract) 27. Jelasity, M., Montresor, A., Jesi, G.P., Voulgaris, S.: The Peersim simulator, http://peersim.sf.net

Dynamic Publish/Subscribe to Meet Subscriber-Defined Delay and Bandwidth Constraints Muhammad Adnan Tariq, Gerald G. Koch, Boris Koldehofe, Imran Khan, and Kurt Rothermel IPVS - Distributed Systems, University of Stuttgart {firstname.lastname}@ipvs.uni-stuttgart.de

Abstract. Current distributed publish/subscribe systems assume that all participants have similar QoS requirements and equally contribute to the system’s resources. However, in many real-world applications, the message delay tolerance of individual peers may differ widely. Disseminating messages according to individual delay requirements not only allows for the satisfaction of user-specific needs but also significantly improves the utilization of the resources in a publish/subscribe system. In this paper, we propose a peer-to-peer-based approach to satisfy the individual delay requirements of subscribers in the presence of bandwidth constraints. Our approach allows subscribers to dynamically adjust the granularity of their subscriptions according to their bandwidth constraints and delay requirements. Subscribers maintain the publish/subscribe overlay in a decentralized manner by establishing connections to peers that provide messages meeting exactly their subscription granularity and complying to their delay requirements. Evaluations show that for practical workloads, the proposed system scales up to a large number of subscribers and performs robustly in a very dynamic setting.

1

Introduction

Publish/subscribe is an important many-to-many communication paradigm for applications with loosely coupled entities where providers of information publish events while recipients subscribe to them. The advantage of this paradigm is the decoupling of publishers and subscribers: Events can be published by providers without knowledge on the set of relevant recipients, while recipients express their interest in certain information without the need to know the actual set of its providers. The evolution of publish/subscribe has followed two main objectives, namely an increased decentralization and an increased orientation on the participants’ specific needs. Former static broker-based architectures were overcome by decentralized systems where publishers and subscribers contribute as peers to the

This work was partially funded by the SpoVNet project of Baden-Wurttemberg Stiftung gGmbH.


Dynamic Publish/Subscribe to Meet Subscriber-Defined Constraints

459

dynamic maintenance of the publish/subscribe system and where they perform the dissemination of events collectively. Specific needs of subscribers were met by the transition from topic-based and channel-based publish/subscribe to contentbased publish/subscribe. Its expressive way to subscribe allows the definition of subscriber-specific restrictions on the event message content. There is still potential for the adaptation of publish/subscribe to peer-specific needs. For instance, many current systems assume that all subscribers expect the same quality of service (QoS) for their requested events. In fact, for many realworld settings, events are of different importance to individual subscribers which can therefore subscribe with different QoS requirements. Consider, for example, meteorological sensor information such as temperature and wind fields. The data itself is relevant for a large number of application entities such as news agencies, traffic monitoring, energy management, and rescue services. However, while local rescue services need to react fast and cannot tolerate large transmission delays, other recipients like a weather forecast service which has a large prediction window do not have that strict delay requirements. Accounting for individual QoS requirements is promising to better utilize the system’s resources. Again, resources such as bandwidth should be considered peer-specific constraints for the maintenance of the system rather than system constants. Considering peer-specific contributions, needs and constraints in publish/subscribe systems is severely complicated by its inherent decoupling. Therefore, in literature, only few approaches have addressed QoS for publish/subscribe. Solutions supporting message delay bounds either assume static topologies [20] or rely on complex management protocols such as advertisement and subscription forwarding to manage end-to-end state information with respect to each publisher [6,17] and therefore constrain the system’s scalability. Peer-specific resource contribution and its inter-dependencies to user-specific delay requirements have not been discussed yet in literature. In this paper, we present a broker-less content-based publish/subscribe system which satisfies the peers’ individual message delay requirements and supports system stability by accounting for resources contributed by individual peers. Subscribers arrange in an overlay so that subscribers with tight delay requirements are served first and then forward messages to peers with lesser requirements. Peers are motivated to contribute some of their bandwidth on receiving and forwarding events which do not meet their own subscriptions (false positives) in exchange for an increased opportunity to satisfy their individual delay requirements. Therefore, peers with tight delay requirements also significantly contribute to the stability of the publish/subscribe system, while they are still in control of their individual permissible ratio of false positives and thus can consider their bandwidth constraints. The evaluations demonstrate the viability of the proposed system under practical workloads and dynamic settings.

2

System Model and Problem Formulation

We consider a broker-less content-based publish/subscribe system consisting of an unbounded set of peers. Peers leave and join the system at arbitrary time, and

460

M.A. Tariq et al.

they can fail temporarily or permanently. The peers act as publishers and/or subscribers which connect in an overlay and forward events to relevant subscribers. The set of overlay connections of a peer s can be classified into incoming connections Fin (s) and outgoing connections Fout (s). We support event forwarding using an out-degree constraint m. It obliges peer s to be ready to forward received messages up to m times (Fout (s) ≤ m). The rate R(s) of events received over connections in Fin (s) is therefore constrained: it must not consume more B(s) than a fraction m+1 of the overall bandwidth B(s) provided by the access link that connects s with the physical network. The basis for all events and subscriptions is the event space denoted by Ω. It is composed of a global ordered set of d distinct attributes (Ai ): Ω = {A1 , A2 , . . . , Ad }. Each attribute Ai is characterized by a unique name, its data type and its domain. The data type can be any ordered type such as integer, floating point and character strings. The domain describes the range [Li , Ui ] of possible attribute values. The relations between events, subscriptions and advertisements can be demonstrated by modelling Ω geometrically as a d-dimensional space so that each dimension represents an attribute. A publisher’s advertisement is a sub-space of that space, and a published event is a single point ω in the space. A subscription is a hyper-rectangle in Ω. An event is matched by a subscription, iff the point ω defined by the event is located within the hyper-rectangle defined by the subscription. A subscription sub1 is covered by a subscription sub2 , iff the hyper-rectangle of sub1 is enclosed in the hyper-rectangle of sub2 . Apart from that, we allow a subscriber s to specify the delay Δ(s) that it is willing to tolerate when receiving events from any of its relevant publishers. In the publish/subscribe system described above, a peer clearly has two concerns. The first is to receive all relevant messages in compliance with its delay requirements. The second is, for the sake of saving bandwidth, to receive and forward only messages that exactly match the peer’s subscription. More precisely, let S be a set of subscribers and PS the set of publishers that publish events matching the subscriptions of S. E denotes the set of all overlay links and path(p, s) = {(p, i1 ), (i1 , i2 ), ..., (im , s)} ⊆ E defines the set of overlay links on the path from a publisher p ∈ PS over intermediate nodes ij to a subscriber s ∈ S. The delay on this path is defined as D(p, s) = e∈E:e∈path(p,s) d(e) where d(e) denotes the link delay on a link e ∈ E. The objective is to maintain the publish/subscribe overlay network in the presence of dynamic sets of publishers PS and subscribers S, so that 1. the delay constraints of a large number of subscribers are satisfied w.r.t. the sets of their relevant publishers (ideally, in the presence of sufficient resources, ∀s ∈ S, ∀p ∈ PS : D(p, s) ≤ Δ(s)), and 2. each subscriber can dynamically adjust the rate of false positives it receives B(s) so that its bandwidth constraints are not violated, i.e. m+1 ≥ R(s). Our approach can work with any monotonically increasing delay metric. However, for simplicity, in our algorithm description we use the hop count as delay metric, i.e. D(p, s) = |{e ∈ E|e ∈ path(p, s)}|.


3

461

Approach Overview

Meeting the objectives presented in Section 2 amounts to finding a good tradeoff between two contradicting goals: to minimise resource usage by avoiding false positives (i.e., a subscriber s receives and therefore forwards only messages that match its own subscription), and to ensure scalability by balancing the contribution of the peers according to their available resources. Fulfilling the first goal affects the scalability of the overall system especially in the presence of out-degree constraints. In the content-based model, subscriptions often intersect with each other rather than being in a containment relationship. Hence, the complete removal of false positives may require subscribers to maintain large number of incoming connections in order to cover their subscriptions [15]. Therefore, false positives cannot be completely avoided and peers need to contribute resource in terms of false positives to ensure scalability. However, allowing individual peers to induce false positives by arbitrarily coarsening their subscriptions without any regularity is unrewarding due to the fact that coarser subscriptions may still intersect instead of being in a containment relationship. We therefore propose to coarsen subscriptions systematically by distinguishing between two levels of subscriptions: user-level and peer-level, as shown in Figure 1. The user-level subscription represents the original subscription as defined by the application. The peer-level subscription is an approximation of the user level subscription and defines which events a peer actually receives. The peer-level subscription is created by spatial indexing [9,14]. The event space is divided into regular sub-spaces which serve as enclosing approximations for user-level subscriptions. The sub-spaces are created by recursive binary decomposition of the event space Ω. The decomposition procedure divides the domain of one dimension after the other and recursively starts over in the created sub-spaces. Figure 2 visualizes the advancing decomposition. Sub-spaces can be identified by dz-expressions. A dz-expression is a bit-string of “0”s and “1”s, which is empty () for Ω. Each time a sub-space is divided, its dz-expression is inherited as prefix for the dz-expressions of the newly created sub-spaces. The peer-level subscription of a peer p can be composed of several sub-spaces and is therefore represented by a set of dz-expressions denoted by DZ(p) with DZ(p) = {dzi | i ≥ 1}. For instance, in Figure 2, the accurate mapping of sub1 = {humidity = [0, 25] ∧ T emp = [0, 100]} requires two sub-spaces in its −→ {000, 010}. peer-level subscription. The mapping is sub1 If the mapping between the subscriptions at user and peer level is identical, the peer will only receive events matching its user-level subscription. In general, however, a peer can coarsen its peer-level subscription in a regular manner so that additional events can occupy a share of its bandwidth. For example, sub1 in Figure 2 can be coarsened by mapping it to the sub-space 0, i.e. sub1 −→ {0}. The regularity of sub-spaces created by spatial indexing is advantageous due to the fact that overlapping sub-spaces are always in a containment relationship, which can be directly mapped to the overlay structure as discussed in Section 4. Additionally, subscriptions can be coarsened or refined in a regular manner. This lesser degree of freedom in the selection of false positives also helps in the

M.A. Tariq et al.

L2=0 L1 =0

Userlevelsubscription Mapping

İ

Filtering Adjust accuracy

OverlayProtocol

Trigger

Ͳ Delayconstraints Ͳ Containmentrelation

Ͳ Delayviolation Ͳ Bandwidthviolation

1

50 100 d1=humidity

100 01

11

010 011 110 111

50

50 00 0

0 0

U1=100 d1=humidity

100

Peerlevelsubscription

0

d2=Temp

Notify

d2=Temp

d2=Temp

Subscribe Advertise Publish

100

U2=100

Application

d2=Temp

462

0

10

100 50 d1=humidity

000 001 100 101 0

0

25 50 75 100 d1=humidity

Sub1 ={humidity=[0,25],Temp=[0,100]}

Fig. 1. Architecture

Fig. 2. Spatial indexing

anticipated bandwidth estimation of the sub-spaces that are considered for use in coarser subscription as detailed in Section 5.

4

Overlay Protocol

Subscribers maintain the overlay in a decentralized manner by connecting and disconnecting other peers. In particular, subscribers satisfy their peer-level subscriptions and delay requirements by connecting to subscribers or publishers that have covering subscriptions and tighter delay requirements. Thereby, subscribers just rely on the subscription and the delay constraints of the peers they are connecting to, and on the fact that these in turn connect to suitable peers. For the satisfaction of its subscription, a peer p needs to discover a suitable parent for each of its dzi in DZ(p). Furthermore, dynamic conditions such as churn, failures and changes in the delay requirements may require a previously suitable parent to be replaced. Therefore, each subscriber maintains a peer view pView 1 that caches information about peers which are relevant because they have covering subscriptions. Overlay maintenance: Periodically, each peer p runs the connectionManagement procedure (cf. Algorithm 1, lines 1-6 ) to check whether each dzi in DZ(p) is covered either by the subscription of a subscriber or by all of the relevant publishers in Fin (p).2 If any dzi is not covered, the findBestParent routine selects a suitable parent from pV iew. Peer p sends a connection request to this potential parent once it is selected. Connection request: Upon reception of a connection (CONNECT) request from a peer p, the potential parent q will normally acknowledge the connection, but it will reject the request if Δ(p) > Δ(q) does not hold. In this case, q sends a hint about the most suitable parent for p according to q’s knowledge. 1 2

In our implementation we modified an epidemic protocol for maintaining pV iew. The set of relevant publishers is maintained similar to pV iew.


463

Algorithm 1. Publish/subscribe overlay maintenance 1: 2: 3: 4: 5: 6:

procedure connectionManagement do while true do if ∃dzi ∈ DZ(p)|dzi is not covered then parent = findBestParent(pV iew, dzi ) pV iew = pV iew − parent trigger Send(CONNECT, p, parent, dzi , Δ(p) )

7: 8: 9: 10: 11: 12: 13: 14: 15: 16: 17: 18: 19: 20:

upon event Receive(CONNECT, p, q, dz(p), Δ(q)) do if Δ(p) > Δ(q) then Fout (q) = Fout (q) ∪ p if |Fout (q)| > m then peer[] = peersToDisconnect() for all t ∈ peer do parent = findBestParent(pV iew ∪ Fout (q), dz(t)) trigger Send(DISCONNECT, t) trigger Send(POTENTIALPARENT, t , parent) if p ∈ / peer then trigger Send(ACK,q) else // Δ(p) ≤ Δ(q) parent = findBestParent(pV iew ∪ Fin (q), dz(p)) trigger Send(POTENTIALPARENT, p, parent)

21: 22: 23: 24: 25: 26: 27: 28: 29: 30:

upon event Receive(ACK, q) do Fin (p) = Fin (p) ∪ q iCon = {dz(ai ) | ai ∈ Fin (p) ∧ Δ(ai ) = 0} // DZ(p) should be covered exactly once, therefore remove unnecessary parents for all dz(ai ) ∈ Fin (p) do for all dz(aj ) ∈ iCon : j = i do if dz(ai ) ≺ dz(aj ) then // dz(ai ) is covered by dz(aj ) iCon = iCon − dz(ai ) Fin (p) = Fin (p) − ai trigger Send(DISCONNECT, ai )

Accepting peer p as a child may violate the out-degree constraints of the peer q. In this case, the peersToDisconnect routine prepares the disconnection from children with a highly selective subscriptions and a large Δ. If p is chosen for disconnection, it will receive a hint (POTENTIALPARENT) message instead of a connection acknowledgement. Upon reception of a hint (POTENTIALPARENT) message, a peer will add the hint to its pView and consider it as a potential parent in its next iteration of the connectionManagement procedure. Connection acknowledgement: Upon reception of an acknowledgement (ACK) message, a peer p ensures that its peer-level subscription is covered exactly once by parent subscribers. This ensures that p’s bandwidth is not wasted in receiving duplicate events. For sub-spaces of p’s subscription that cannot be covered by parent subscribers, coverage must be accomplished by connecting to all relevant publishers. Thus, for each of such sub-spaces that are only covered by one or more publishers, p continues to search for relevant publishers or subscribers. Placement of publishers: Similar to subscriptions, an advertisement of a publisher is represented by a set of dz-expressions (DZ). This allows the automatic discovery and inclusion of the publishers in the overlay network, as a result of connection requests (CONNECT) from subscribers. Publishers maintain their Fout connections similar to subscribers (cf. Algorithm 1, lines 9-17 ).

464

M.A. Tariq et al.

5

Triggers for Changes in Accuracy

Until now we have described the organization and maintenance of the publish/subscribe overlay in the presence of subscriber-specified delay requirements. Nevertheless, we need additional mechanisms to ensure the scalability of the scheme. Sometimes a peer cannot find any potential parent to satisfy its delay constraints. In Figure 3(a), for instance, subscriber S5 has a rather selective subscription and tight delay requirements. If the publisher P1 cannot accommodate more child subscribers, then S5 can only connect to S2 according to Algorithm 1. However, doing so violates the delay constraints of S5 . In this case S5 can coarsen its peer-level subscription according to its bandwidth constraints in order to be placed between P1 and S2 . This is possible because the overlay maintenance strategy places subscribers with less selective subscriptions higher in the dissemination graph (cf. Algorithm 1, lines 7-17 ). Therefore, subscribers can improve the probability to satisfy their delay requirements by agreeing to a coarser subscription as shown in Figure 3(b). Similarly, if changes in the event rate violate the bandwidth constraints, a subscriber refines its subscription accordingly. In this case, however, there will be no change in the set of existing parents as the new subscription is covered by the previous subscription (cf. Algorithm 2, lines 4-6 ). In the following sections, we describe the mechanisms to adjust the accuracy of the mapping between user and peer level subscriptions according to subscriberspecific bandwidth constraints. 5.1

Accuracy of Subscription Mapping

A subscriber can reduce the accuracy of the peer-level subscription by using c −→ from the user-level subscription sub to a smaller set a coarser mapping ǻ=0 P1 DZ={İ}

ǻ=0 P2 DZ={11}

P2

ǻ=0 DZ={11}

100 0101 0111 1101 1111

ǻ=1 S1 DZ={1,00}

ǻ=3 S2 DZ={0}

ǻ=1 S1 DZ={1,00}

ǻ=3 S2 DZ={0}

d2=Temp

75

Sub2 ={humidity=[0,100], Temp=[0,25]}

0100 0110 1100 1110

Subscription

0001 0011 1001 1011

İ

50 25

S5

S3

ǻ=1 DZ={01}

ǻ=2 DZ={1}

(a)

S4 ǻ=5 DZ={100}

S5

S3

ǻ=1 DZ={0}

ǻ=2 DZ={1}

S4 ǻ=5 DZ={100}

(b)

Fig. 3. Example Scenario with m = 2

0000 0010 1000 1010 0 0

25 50 75 d1=humidity

100

DecomposedEventSpace

00

4: 5: 6:

10

DZC

0000 0010 1000 1010 DZ+1 PossibleInternalrepresentations

Fig. 4. Subscriber-defined Accuracy

Algorithm 2. Triggers for change in accuracy 1: 2: 3:

DZͲ1

refinement

ǻ=0 P1 DZ={İ}

upon event TimeOut do if ∃dzi ∈ DZ(p)|dzi is not covered then reduce accuracy of peer-level subscription by coarsening upon event BandwidthViolated do increase accuracy of peer-level subscription accordingly remove subscribers in Fout (p) which are not covered by the new DZ(p).


465

of coarser dz-expressions DZ C . Reduced accuracy causes false positives and c increases bandwidth usage. Therefore, a condition for selecting a −→ mapping on peer s is that the reduction of accuracy does not violate the peer’s bandwidth constraint B(s). The subscriber can ensure this by iteratively selecting another coarse mapping, thereby refining or coarsening individual dz-expressions and thus controlling the overall rate of received events. The bandwidth usage induced by each dz-expression depends on the rate of events matched by the expression. Therefore, for each sub-space represented by a dz-expression in DZ C , the subscriber continuously studies the event rates in the sub-space that is divided once less (DZ −1 ) and in the sub-spaces that are divided once more (DZ +1 ). The latter can be calculated by counting the received messages, while the event rate in the coarser sub-space is estimated by means of statistical aggregation [11]. The estimation of the event rate in the coarser sub-space relies on the measurements of other subscribers that are currently subscribed to the coarser sub-space or a part of it. The measurements appear in the messages of the protocol used to maintain pV iew (Section 4). Figure 4 shows the possible mappings from a user-level subscription. If the subscription is currently mapped to DZ C = {00, 10} then the subscriber keeps track of the event rates in the sub-spaces DZ −1 = {} and DZ +1 = {0000, 0010, 1000, 1010}. If there is a high rate of false positives in a sub-space of the current peer-level subscription, the subscriber will drop it and select the relevant of the finer sub-spaces from DZ +1 instead. Similarly, the subscriber can select one sub-space from DZ −1 instead of multiple previous enclosed sub-spaces and receive additional false positives. If the event rate and the subscriptions in the system remain constant, this strategy will allow the subscriber to converge to a state where its dz-expressions no longer need to be adjusted. 5.2

Optimized Spatial Indexing

For an event space with a large set of attributes, the number of dz-expressions for an accurate subscription representation can be very large. As described in Section 5.1, a coarse subscription mapping reduces the number of dz-expressions. However, it induces false positives and hence its applicability depends on the bandwidth constraints of the subscriber. A simple modification in the representation of dz-expressions can reduce their number without changing their accuracy. A dz-expression is redefined to include the wild-card ∗ which stands for “0 and 1”. Two dz-expressions that differ in only one place can be combined by replacing this place by “∗”. For example, the subscription in Figure 4 can be represented by one dz-expression “∗0∗0”. Dz-expressions of that form are created by a modified spatial indexing mechanism (Section 3). The decomposition procedure works mainly as before. Only if the subscription covers the complete domain of the dimension to be divided, then instead of creating two dz-expression for the smaller sub-spaces (ending with 0 and 1), ∗ is added to the dz-expression. The containment relationship defined on dz-expressions as well as the subscription mapping and bandwidth estimation mechanisms work with the

466

M.A. Tariq et al.

modified technique. Furthermore, the modification allows subscribers to define constraints only on a subset of attributes in the event space.

6

Evaluation

In this section, we evaluate the performance of the presented algorithms according to the following criteria: i) convergence to subscription and delay constraint satisfaction, ii) control overhead, iii) adaptability to dynamic conditions, iv) scalability in terms of number of peers and attributes, and v) effect of bandwidth consumption on the satisfaction of subscribers. Experimental Setup: Simulations are performed using PeerSim [12]. Each peer relies on Gossip-based peer sampling service [13] to maintain its partial view (pV iew) of 5% other peers in the system. Unless otherwise stated, all the simulations are performed for n = 1, 024 peers. The out-degree constraints of the peers are chosen as m = log2 (n). The event space has up to 10 different attributes. The data type of each attribute is Integer, and the domain of each attribute is the range [1, 128]. We evaluated the system performance under uniform (W1 ) and skewed (W2 ) subscription workloads; and with a uniform event distribution. Skew is simulated by twenty randomly chosen hot spots in the event space, around which subscriptions are generated using the widely used 80%-20% Zipfian distribution. We use the following performance metrics in our evaluations: 1) Percentage of converged peers: The fraction of peers out of the total population which have found a suitable set of parents that cover their subscription. 2) Percentage of notified peers: The fraction of peers which are receiving events from all the relevant publishers without violating their delay constraints. 3) Control Messages: The control overhead in terms of number of connection request messages that a peer sent before finding its appropriate set of parents. 4) Construction time: The time needed to complete the construction of the overlay topology. Convergence: In this experiment, moderate delay requirements are assigned to the peers such that convergence can be achieved. Figures 5(a)-(b) show the construction time for the overlay topology. For all of the workloads, the percentage of notified peers is always less than that of converged peers until 100% convergence is achieved. The reason is that the peers opportunistically connect to other peers in order to cover their subscriptions and satisfy their delay constraints. Therefore, during the evolution of the overlay topology, many separate isolated groups of peers may exist. Some of these groups may not have found a connection to the relevant publishers. Eventually, all the groups converge to one overlay topology. The overlay construction time for workload W2 is higher due to the fact that the subscription distribution is highly skewed with very little overlap between the subscriptions of peers assigned to different hot-spots. This results in subscribers with coarser subscriptions occupying all the places near the publishers, forwarding events that only correspond to a portion of the event space. Therefore, the


60 40 20 Converged Peers Notified Peers

80 60 40 20 Converged Peers Notified Peers 0

0

10

20

30

40

50

60

0

10

Time (a) - Workload 1

20

30

40

50

1 2 3 4 5 6 7 8 9 10 11 12 13

60

Time (b) - Workload 2

100

Control Messages (c) 5

100

60

60

Time

80

40

Average in-degree

Workload 1 Workload 2

80

4.5

Notified Peers Converged Peers

40

0.5

1

1.5

2

2.5

2 0

3

1000 2000 3000 4000 5000 6000

% of False Positives in the system

% of Notified Peers

100 80 60 40 Moderate delay req Tight delay req

0 0

3.125 6.25

0

12.5

25

50

% of Allowed False Positives for each peer (g)

20 18 16 14 12 10 8 6 4 2 0

20

40

60

80

100 120

Average Number of Dz-expressions (f)

Number of Peers (e)

Churn Percentage (d) - Workload 1

20

3

2.5

0

0

4

3.5

20

20

0

Workload 1 Workload 2

Percentage of Peers

80

0

% of Notified Peers

50 45 40 35 30 25 20 15 10 5 0

100

Percentage of Peers

Percentage of Peers

100

467

Moderated delay req Tight delay req

3.125 6.25 12.5

25

50

% of Allowed False Positives for each peer (h)

Fig. 5. Evaluations of the presented algorithms

subscribers with finer subscriptions (to uncovered portions of the event space) have to increase their subscription to compete with subscribers with coarser subscriptions. Figure 5(c) shows the control overhead incurred by the peers in order to find suitable parents. It shows the percentage of the affected peers as a function of the number of connection request messages sent by them. Adaptability: This experiment evaluates the dynamic resilience of the system in the presence of continuously joining and leaving subscribers. The percentage of churn is relative to the total of all peers in the system. For instance, a churn of 2.5% means that in each time step, 25 on-line peers leave and the same number of new peers with different subscriptions and delay requirements join the system. Figure 5(d) shows the percentage of converged and notified peers for different percentages of churn along with the standard deviation for W1 . The reason for the gradual degradation in the percentage of notified and converged peers is due to the fact that a high churn rate increases the probability that peers placed near the publishers leave the system, affecting the delay constraint satisfaction of all

468

M.A. Tariq et al.

their descendant subscribers. The evaluation results obtained from W2 shows similar trend. Scalability: First, we study the scalability with respect to the number of peers in the system. In all the experiments the out-degree constraints are chosen as log2 (n) of the total number of peers n. Figure 5(e) shows that up to 5000 peers the overlay construction time almost stays the same. Furthermore, the overlay construction time for W2 is in general higher due to the fact that the subscription distribution is skewed and some subscribers may need to increase their subscriptions as discussed in the convergence evaluations. Next, we study the effect of the number of attributes in the event space on the system’s scalability. The number of dz-expressions needed for the accurate representation of a user-level subscription generally increases with the number of attributes. A peer maintains a suitable parent for each of its dzexpressions. Therefore, we study the effect of an average increase in the number of dz-expressions on the average in-degree for W1 as shown in Figure 5(f). The averages are taken over all the peers in the system and the out-degree constraints of the peers are kept constant during the experiment. The results show a slight increase in the average in-degree with the number of dz-expressions, i.e., increasing the average number of dz-expressions from 8 to 128 increases the average in-degree by just 0.7 to 2.7. Effect of bandwidth on the satisfaction of delay requirements: In this experiment two scenarios are evaluated: one where the subscribers are assigned moderate delay requirements (S1 ) and the other with tight delay requirements (S2 ). In both the scenarios, delay requirements of all the subscribers cannot be satisfied without inducing false positives. All the subscribers are assigned the same bandwidth constraints, specified in terms of allowed false positives as a percentage of the overall event rate. For example, 3.125% of allowed false positives means that subscribers can increase their subscription till they are receiving 3.125% of overall events in the system as false positives. Figure 5(g) shows the percentage of notified peers for different percentages of allowed false positives, and Figure 5(h) shows the actual percentage of false positives in the system for the scenarios S1 and S2 . In case of S1 , only 73.7% of subscribers are notified in the absence of false positives. However, allowing peers to receive up to 12.5% of overall events as false positives increases the percentage of notified peers by 26.3% to 100% with only 7.5% increase in the overall rate of false positives in the system. In contrast, in scenario S2 , even when the subscribers are allowed to increase their false positives up to 50% of overall event rate, the percentage of notified peers increases by only 8.3% to 70.2%. The reason is that the delay requirements of subscribers in S2 are very tight and that it is not possible to satisfy all of them. In this case, the unsatisfied subscribers coarsen their subscriptions to get a better place in the overlay. However, as all the subscribers have similar bandwidth constraints and there are limited places to satisfy delay requirements, coarsening subscriptions does not give them any competitive advantage. It just raises the overall rate of false positives.


7

469

Related Work

Over the last decade, many content-based publish/subscribe systems have been proposed with scalability as the main design criterion [7,10,5,3]. In order to achieve scalability, a large number of unnecessary events (false positives) are clearly undesirable and should be avoided [15]. Many recent systems address scalability by clustering the subscribers with similar interests [1,8]. Sub-2-Sub [19] clusters subscribers with non-intersecting subscriptions into rings and completely avoids false positives. However, even for a moderate number of subscribers, the number of clusters may quickly grow to a very large number, limiting the scalability of the approach [15]. Similarly, techniques from data mining have been used to group subscriptions in a limited number of clusters [16], but this requires central coordination. Apart from the stated drawbacks, existing approaches [2,4] only focus on the overall reduction of false positives without taking into account the heterogeneity of subscribers in terms of QoS requirements to better utilize resources in a publish/subscribe system. Only few publish/subscribe systems address issues related to QoS. IndiQoS [6] addresses individual delay requirements, but it relies on subscription and advertisement forwarding mechanisms to maintain end-to-end delay bounds and to reduce false positives. These mechanisms introduce an extra overhead and have a restricted efficiency with widely dispersed subscribers [15]. Some of the problems stated above are addressed by the system presented in [17] which clusters subscribers into groups in order to reduce false positives. However, within each group, subscription and advertisement forwarding is used to maintain end-to-end delay bounds. The solution presented in the paper at hand goes a step forward, as it avoids advertisement flooding and takes into account the inter-dependencies between peer-specific resource contribution and delay requirements.

8

Conclusion

In this paper we have shown how the individual delay requirements of a large dynamic set of subscribers in a content-based publish/subscribe system can be satisfied without violating their bandwidth constraints. In particular, subscribers are given the flexibility to define their permissible rate of false positives according to their individual bandwidth constraints. Additionally, we propose a subscriber-driven decentralized algorithm to connect publishers and subscribers in an overlay network according to their delay requirements so that subscribers with tight delay requirements are located closer to the relevant publishers. The evaluation shows that the proposed algorithm converges to the satisfaction of subscriber-specific delay constraints even in a very dynamic setting. The ideas presented in this paper are applied to support a peer-to-peer based gaming application in the SpoVNet project where link delay information is provided by a cross-layer information framework [18].

470

M.A. Tariq et al.

References 1. Anceaume, E., Gradinariu, M., Datta, A.K., Simon, G., Virgillito, A.: A Semantic Overlay for Self-Peer-to-Peer Publish/Subscribe. In: ICDCS (2006) 2. Baldoni, R., Beraldi, R., Querzoni, L., Virgillito, A.: Efficient Publish/Subscribe Through a Self-Organizing Broker Overlay and its Application to SIENA. The Computer Journal (2007) 3. Bhola, S., Strom, R.E., Bagchi, S., Zhao, Y., Auerbach, J.S.: Exactly-once Delivery in a Content-based Publish-Subscribe System. In: Intl. Conf. on Dependable Systems and Networks (2002) 4. Bianchi, S., Datta, A., Felber, P., Gradinariu, M.: Stabilizing Peer-to-Peer Spatial Filters. In: ICDCS (2007) 5. Briones, J.A., Koldehofe, B., Rothermel, K.: Spine: Adaptive publish/subscribe for wireless mesh networks. Studia Informatika Universalis 7 (2009) 6. Carvalho, N., Araujo, F., Rodrigues, L.: Scalable QoS-Based Event Routing in Publish-Subscribe Systems. In: Intl. Symposium on Network Computing and Applications (2005) 7. Carzaniga, A., Rosenblum, D.S., Wolf, A.L.: Design and evaluation of a wide-area event notification service. ACM Trans. Comput. Syst. (2001) 8. Chand, R., Felber, P.: Semantic Peer-to-Peer Overlays for Publish/Subscribe Networks. In: Cunha, J.C., Medeiros, P.D. (eds.) Euro-Par 2005. LNCS, vol. 3648, pp. 1194–1204. Springer, Heidelberg (2005) 9. Gaede, V., G¨ unther, O.: Multidimensional access methods. ACM Comput. Surv (1998) 10. Gupta, A., Sahin, O.D., Agrawal, D., Abbadi, A.E.: Meghdoot: Content-Based Publish/Subscribe over P2P Networks. In: Intl. conf. on Middleware (2004) 11. Jelasity, M., Kowalczyk, W., van Steen, M.: An approach to massively distributed aggregate computing on peer-to-peer networks. In: Workshop on Parallel, Distributed and Network-Based Processing (2004) 12. Jelasity, M., Montresor, A., Jesi, G.P., Voulgaris, S.: PeerSim: A Peer-to-Peer Simulator, http://peersim.sourceforge.net/ 13. Jelasity, M., Voulgaris, S., Guerraoui, R., Kermarrec, A.-M., van Steen, M.: Gossipbased peer sampling. ACM Trans. Comput. Syst (2007) 14. Ohsawa, Y., Sakauchi, M.: A New Tree Type Data Structure with Homogeneous Nodes Suitable for a Very Large Spatial Database. In: Proc. of the Intl. Conf. on Data Engineering (1990) 15. Querzoni, L.: Interest clustering techniques for efficient event routing in large-scale settings. In: Intl. Conf. on Distributed Event-Based Systems (2008) 16. Riabov, A., Liu, Z., Wolf, J.L., Yu, P.S., Zhang, L.: Clustering Algorithms for Content-Based Publication-Subscription Systems. In: ICDCS (2002) 17. Tariq, A., Koldehofe, B., Koch, G., Rothermel, K.: Providing probabilistic latency bounds for dynamic publish/subscribe systems. In: Proceedings of the 16th ITG/GI Conference on Kommunikation in Verteilten Systemen (KiVS). Springer, Heidelberg (2009) 18. The SpoVNet Consortium. Spontaneous Virtual Networks: On the road towards the Internet’s Next Generation. it - Information Technology (2008) 19. Voulgaris, S., Rivire, E., Kermarrec, A.-M., van Steen, M.: Sub-2-sub: Selforganizing content-based publish and subscribe for dynamic and large scale collaborative networks. In: Int’l Workshop on Peer-to-Peer Systems (2006) 20. Wang, J., Cao, J., Li, J., Wu, J.: Achieving Bounded Delay on Message Delivery in Publish/Subscribe Systems. In: Intl. Conf. on Parallel Processing (2006)

Combining Hilbert SFC and Bruijn Graphs for Searching Computing Markets in a P2P System Damia Castellà, Hector Blanco, Francesc Giné, and Francesc Solsona Dept. of Computer Science, University of Lleida, Lleida, Spain {dcastella,hectorblanco,sisco,francesc}@diei.udl.cat Abstract. This paper proposes an efficient and scalable computational resource discovery overlay orientated towards P2P computing. Our proposal gathers the peers into markets according to their computational resources. Each market is arranged in an N-tree and the trees are linked by a Bruijn graph. The tree topology allows efficient searching of available resources in a specific market, while Bruijn provides good scalability because search complexity does not depend on the number of markets. A Hilbert function is used to arrange markets in one ordered and mono-dimensional space. This way, the proposed architecture exploits the Bruijn and N-tree topologies together with the Hilbert function. A look-up query mechanism for simple and multiple queries with a low algorithmic cost is also introduced over this architecture. The performance of our proposal was analysed by means of simulation in relation to the widely used Chord overlay with the case of simple queries, and the Baton algorithm with the case of range queries. Furthermore, a large number of experiments demonstrate the proper behaviour of the system. The results obtained reveal the competitiveness of our proposals. Keywords: P2P computing, resource discovery, P2P overlay.

1

Introduction

P2P computing is a distributed computing paradigm that uses Internet to connect thousands or even millions of users into a single large virtual computer based on the sharing of computational resources [8]. Thus, P2P computing can serve for executing a wide variety of parallel applications (i.e. embarrassingly parallel programs or distributed libraries). In order to schedule and execute these applications, a P2P computing platform needs an efficient mechanism to search the set of peers, whose available computational resources (i.e. CPU, Memory and Bandwidth) fit the requirements of the application to be executed. Taking into account both the large-scale size and the unlimited kind of different computational resources spread throughout a P2P network, finding computational resources over a P2P network becomes a research challenge [12].

This work was supported by the MEyC-Spain under contract TIN2008-05913 and the CUR of DIUE of GENCAT and the European Social Fund.


472

D. Castell` a et al.

This paper proposes an efficient and scalable computing resource discovery mechanism orientated to P2P computing. Our proposal classifies the peers into markets according to their computational resources. Thus, a market is made up of a set of peers, whose computational resources (node attributes) fit into a specific range [4]. This classification is done by means of a Hibert multi-dimensional Space-Filling Curve (SFC) [16]. Thus, when a peer requests some kind of computational resources, the Hilbert SFC returns the market key, where the resources can be found. The Hibert was chosen, over other multi-dimensional SFC [15] such as Scan, Peano or Gray, due to the fact that the Hilbert gathers markets of similar resources into nearby keys. Likewise, our proposal arranges the peers belonging to the same market by means of a tree topology, whose root node has knowledge about the available computational resources throughout the tree (market). Note that a tree allows the system-growing capacity to be managed and maintained reliably [5,6,10]. Finally, these markets are tied together into a cohesive system by means of a logical “P2P overlay network”. Several topologies have been suggested by the literature for this purpose [14], including Chord, Baton, CAN, Pastry, Bruijn and Viceroy. Among the existing topologies, the best for our aims is a topology based on the Bruijn graph [11] because it has a balanced congestion, a constant degree of links and a logarithmic diameter. In order to integrate these three layers (Hilbert, Bruijn and Tree) into a cohesive system, some improvements have been introduced into the Bruijn graph. A technique based on the virtualization of nodes has been developed for whenever the Bruijn is not completed. It allows any market with the most similar resources to the one provided by the upper layer (Hilbert), which does not yet exist, to looked for in the graph. Likewise, in order to better map the Hilbert keys (market keys) over the Bruijn graph, some extra links have been added. Thus, the number of hops for a specific query has been optimised. The three layers described, together with the included improvements make up the proposed overlay system architecture, named DisCoP (Distributed Computing in P2P systems), explained in this paper. A look-up query mechanism is also introduced over this architecture. This provides range queries for a specific resource, named range queries in this paper (i.e. 10 nodes of CPU [3-4]Ghz); multiple resource queries of a specific value, named simple query (i.e. 10 nodes of CPU 3Ghz and Memory 2GB), or of a range of values, named multiple queries (i.e. 10 nodes of CPU [3-4]Ghz and Memory [2-3]GB). All these queries are delivered with a low algorithmic cost. The performance of our proposal was analysed by means of simulation in relation to the Chord overlay in the case of the simple queries, and the Baton algorithm in the case of the range queries. The number of hops and and links and the congestion were analysed in both cases. The results obtained reveal the competitiveness of our architecture. In addition, the performance of our proposal was analysed for multiple queries. All the experiments reveal the good behaviour of our system, and that it is a serious candidate to be implemented in a P2P computing platform.

Computing Resource Discovery

473

The outline of this paper is as follows; the related work is described in Section 2. Section 3 explains the main techniques (Hilbert, Bruijn and Tree topologies) used by our proposal. Section 4 introduces the proposed DisCoP architecture. Section 5 presents the discovery mechanism used by DisCoP for simple and multiple queries. The efficiency measurements of our proposals are performed in Section 6. Finally, the main conclusions and future work are explained in Section 7.

2

Related Work

To our knowledge, the CompuP2P architecture proposed by Gupta et al. [9] is the most closely related to our work. CompuP2P creates dynamic markets of network-accessible computing resources, connected by means of a Chord network [18]. Chord is a DHT-based distributed look-up algorithm for locating a node that stores a particular resource. Unlike the Bruijn graph used in our proposal, the Chord overlay network does not have a constant degree and as a consequence, the communication speed is degraded. Likewise, the Chord protocol is not well suited to range queries since hashing destroys the ordering of data. Recent works have proposed some discovery mechanisms addressed to range queries [10,7,1,2,17]. Some works, such as [10], propose P2P overlays oriented to lookup range queries with one attribute dimension. The Baton solution [10] is based on a balanced binary tree overlay network. By adding a small number of links in addition to the tree edges, Baton is able to support an index structure for range queries of a specific resource. Unlike our proposal, Baton does not allow range queries of multiple resources. Other works, such as [7,1], propose resource discovery services optimized to multiple queries. Specifically, P. Costa et. al. [7] presents a fully decentralized resource selection algorithm, where each node is solely responsible for its own attributes. Nodes are mapped to a multi-dimensional space, where each dimension represents a resource-attribute type. In this way, this proposal is similar to our work given that each node in our overlay is also responsible for its own resources. On the other hand, our overlay maps the nodes to a one-dimensional space using SFC curves, while Costa et al. uses a multi-dimensional space. In the same way, J. Albrecht et. al. [1] presents an architecture oriented to Grid environments, called SWORD, organized into a topology of interconnected groups with required intra-group, inter-group and per-node characteristics whose relative importance and sensitivity are expressed using penalty functions. The main difference from our schema is that the authors present a new Grid service supporting multi-attribute range queries over DHTs and scattering the nodes to a virtual multi-dimensional space. Another relevant work is Mercury [2], which handles multi-attribute queries by creating a routing hub for each attribute. Thus, queries are passed to one of the hubs which corresponds exactly to the attributes that are queried. Each routing hub is organized into a circular overlay of nodes and places data contiguously on this ring. Given that Mercury does not use hash functions, data

474


partitioning between nodes can become non-uniform, thus requiring an explicit load-balancing schema. Finally, an interesting and similar work is Squid [17]. Squid uses a locality-preserving indexing scheme based on Hilbert SFC and organizes the nodes into a DHT-based structured system. Thus, this has a similar architecture to our proposal. The main difference is the fact that it is focused on optimizing the searches based on ranges according to the shape of the Hilbert curve and the clustering properties of this. Instead, we use the Hilbert SFC to map the resources and make use of the Bruijn graph for its high connectivity and low diameter.

3

Previous Concepts

DisCoP integrates three layers (Hilbert, Bruijn and Tree) and optimises them to develop a single distributed computing platform. This section introduces Hilbert SFC, Bruijn graphs and Tree topologies as previous concepts. – Hilbert Multi-Dimensional Space-Filling Curve (SFC) is a hash function used to map a multi-dimensional space of data into the 1D domain. As Fig. 1 shows, an SFC is a thread that goes through all the points in a space while visiting each point once. In the P2P computing case, it means that the N-attributes (computational resources) provided by a node are mapped into a single key, which represents the identification of the market where the requested resources are available. Fig. 1 shows an example of a 2-D space (CPU and Memory), where each value of the CPU and Memory axis represents a range of values. There are five SFCs widely used in the literature [15]: Sweep, Scan, Peano, Gray and Hilbert. One of the most desired properties from such an SFC is clustering, which means the locality between objects in the multi-dimensional space being preserved in the linear space. Thus, the following SFC key-mapping step into the Bruijn graph is made easier. Given that it is widely believed that the Hilbert SFC achieves the best clustering [16], this was chosen for our purposes. – Bruijn graph is a directed graph with k outgoing and incoming edges at each node. Each node has a unique and fixed length key of base k. The maximum number of nodes of the graph is N = k D nodes, where D is the diameter (maximum distances between any two nodes). A classic Bruijn for k = 2 and N = 8 is shown in Fig. 1(right). From this Fig., we can see the routing algorithm followed by Bruijn. A node with key H is linked to two nodes with keys 2H and 2H+1, i.e. Fig. 1(right) shows how node 001 is linked to nodes 010 and 011. Bruijn was chosen from among the most widely used P2P topologies, [12,14], such as Chord, Baton, Pastry, Bruijn and Viceroy; because it offers the best diameter-degree and highest connectivity. This is shown in Table 1, where (left) shows the asymptotic degree-diameter properties of the different graphs and (right) its graph diameter for a maximum number of nodes N = 106 . – N-ari Trees are very well appreciated in the database world due to their properties: log(N ) look-up length, where N is the total number of peers,


475

Key length of 3 digits

100

110

010

000

101

111

011

001

Fig. 1. (Left)Example of Hilbert SFC. (Right)Example of Bruijn graph of base 2 and Key length of 3 digits. Table 1. (left) Degree-diameter complexities. (right) Diameter ( N = 106 nodes). Graph Degree Diameter Baton k+1 2logk N Chord log2 N log2 N Pastry (k − 1)logk N logk N Bruijn k logk N Viceroy k 2logk N (1 − o(1))

Graph/k Baton Chord Pastry Bruijn Viceroy

2 20 31

3 40 13 20

4 26 10 16

10 13 6 10

20 10 20 20 5 8

and constant degree. In addition, its hierarchical topology for managing and maintaining the system-growing capacity favours the good scalability of the system. In a previous work [5], a P2P Computing system based on a tree hierarchy, together with its main functionalities (peer insertion, maintenance, job launching and scheduling and peer output) was explained. The proposed maintenance system sends the state information from bottom of the tree to the top. Thus, the root node obtains information about all the available resources in the tree. Likewise, nodes with high reputations are located on the higher levels of the tree (close to the root), whereas nodes with low reputations are located instead in the lower levels. We define reputation as the time elapsed since the entry of a node into the system. Their main drawback is the congestion of the nodes near the root. Because of this, our architecture assumes that the size of the tree (market size) is limited. In the same way, our architecture assumes a huge number of different markets connected through the Bruijn graph. Given that our previous works looked into the managing of peers at the tree level [5,6], the present paper is focused on the interaction between the Hilbert SFC and the Bruijn graph.

4

The DisCoP Architecture

The DisCoP architecture is made up of three layers (see Fig. 2). The top layer is a Hilbert SFC, which allows to be classified into markets according to their

476


computational resources. Note that each node is represented by means of an n-tuple, where n are the computational attributes taken into account (CPU and Memory in the Fig. 2). It is worth pointing out that our system assumes that the attribute values are obtained with a specific benchmark tool. The Hilbert keys are mapped into the second layer by means of a Bruijn graph. Finally, a market of nodes (bottom layer) hangs from each Bruijn node, and the former are arranged by means of a tree topology. The root node of each tree has knowledge of the computational resources available throughout its tree. Fig. 2 shows an example of node insertion. As we can see in Fig. 2(a), a new peer characterised by the 2-tuple {CP U = 3, M em = 1} wishes to be inserted into the system. The Hilbert SFC returns the key = H12, which identifies such a market with nodes characterised by the 2-tuple {3, 1} (see Fig.2(b)). Finally, the market identified by the obtained key is searched for in the Bruijn graph. If such a market exists, the peer is inserted into the market tree in a balanced way (see Fig. 2(c)). A complete explanation of the balancing method used in the tree can be found in [6]. If the market does not exist, then a new Bruijn node (market) is created.

Fig. 2. DisCoP System Architecture. Example of node insertion.

In order to integrate these three layers into a cohesive distributed computing platform, some improvements have been added to the Bruijn layer. A virtualization of nonexistent nodes is proposed to enhance the search mechanism when the Bruijn graph is not completed. Furthermore, a set of extra links was added to improve the mapping of the Hilbert keys into the Bruijn graph.


4.1

477

Virtualization of Nodes

According to the literature, the resource discovery cost over a complete Bruijn graph is optimal. However, whenever the Bruijn graph is not complete, the searching for a specific market can suppose a high number of hops. This is due to the fact that nodes involved in the searching path do not yet exist. In order to improve this, a solution based on the virtualization of nodes is proposed. The virtualization of nodes means that, apart from its own key, one physical node can have a continuous range of keys assigned, and this identifies nonexistent peers (virtual nodes). In this way, each physical node, N odei , stores its own key and all the keys related to the virtual nodes between N odei and its preceding physical node (N odei−1 ), following the Bruijn graph. Thus, as the system increases, fewer virtual keys are assigned to peers. The virtualization ensures that the number of hops in searching for resources will be at most the number of digits of the Bruijn key, that is logk (N ). Nonetheless, each node will have to maintain the links to each key assigned to it. 4.2

Adding Extra Links

The Hilbert function assigns contiguous keys to markets with similar resources. For instance, according to the example of Fig. 2, market with tuple {0,1} has a key H3, whereas the market with tuple {0,2} has a key H4. In order to maintain this locality in the Bruijn graph, a new link (called Hilbert link) was added between two nodes with contiguous Hilbert keys. It means that node H2 will be linked (−→) to node H3 and H3 −→ H4 −→ H5 −→ H6 −→ .... and so on. This way, the platform can perform searches of similar attributes with very few hops.

5

Searching Algorithm

This section explains the simple and multiple query search process carried out by the DisCoP system. The search process can be performed by any node in the system and it returns the Bruijn nodes (markets), where the requested resources are found. Resources are requested by means of an n − tuple. Thus, queries are ndimensional and each term is a resource value, for example, specifying the number of nodes, CPUs per node, CPU power, Memory, Bandwidth, etc. Each n − tuple can be simple or multiple. Simple attends for one value for each resource, i.e. a simple 4-tuple would be {r1 , r2 , r3 , r4 }, where ri is the value of the resource i that the user requests. Multiple query refers to a range of values for each resource i, i.e. {[r11 , r21 ], [r12 , r22 ], [r13 , r23 ], [r14 , r24 ]}, where [r1i , r2i ] is the range of requested values for resource i. Note that the range query case is a particular case of multiple query with a searching range for a single resource {[r1k , r2k ]}, where 1 ≤ k ≤ n. It is worth pointing out that the search for a multiple query can be broken down into multiple simple queries. These multiple simple queries can be performed in a parallel manner due to the design of the DisCoP architecture.

478


Algorithm 1 shows our multiple query-searching proposal. Note that a simple query can be understood as a particular case of a multiple query. The input argument of the algorithm is a multiple n − tuple, which is made up of various simple n − tuples. First of all, Algorithm 1 splits the multiple n − tuple into different simple n − tuples. For each simple n − tuple, its respective Hilbert key is obtained and arranged into the KEY S set (KEY S = Hilbert(n − tuple)) according to its value. Next, for each key belonging to KEY S, the Algorithm looks for the optimal path, in hop terms, from each previous key and for each kind of link, Bruijn or Hilbert. In the case that the optimal path Keyi −→ Keyj was achieved by means of Hilbert links, Keyj will be stored into the HKEY SN odei set, where N odei is the node identified by Keyi . In the other case, Keyj will be stored in the Bruijn set (BKEY SN odei ). Once the BKEY SN ode and HKEY SN ode sets have been built, the Seaching N odes function is called up with both sets as arguments. The Seaching N odes function goes over all the keys belonging to the BKEY SN ode in a parallel manner, making use of the proposed architecture, and stores the nodes identified to such a keys in List. From each N ode, the Searching N odes function is again called up in a recursive mode with the BKEY SN ode and HKEY SN ode sets belonging to such a node. In the same way, whenever the HKEY SN ode is not empty, the function goes over all the nodes starting from N ode, following the Hilbert links until it finds the N ode identified with the Key stored in the HKEY SN ode . Note that all HKEY SN ode sets have one single element as a maximum, given that each node only has only one out-coming Hilbert link. Finally, the list of found nodes is returned. It is worth remarking that the cost of our algorithm is proportional to the number of simple queries that made up a multiple query. Note that the cost of a simple search is key length = logk (Nmax ), where k is the degree of a Bruijn graph and Nmax is the maximum number of nodes of a Bruijn graph.

6

Experimentation

The performance of our proposals was tested by means of GridSim [3] and SimJava [13] simulators. In order to simulate our P2P platform with GridSim, peers were modelled as user entities by means of threads. All entities (peers) were connected through network links, whose bandwidth and latency can be specified at the start time. SimJava features provide the management of events and the mechanism for discovering peers. Each Bruijn node is characterised by 4 different attributes (4-tuple) and a key of 16 bits, varying the degree (k) between 2 to 4. All tests were performed with a maximum value of 1024 peers and a total of N 2 searches, where N is the number of peers of the graph. The searches are created with an uniformly distributed random generator. They can be Simple or Multiple. In this section, we evaluated the search performance of our proposed architecture (DisCoP) with respect to widely used overlays in the literature, such as Chord and Baton (balanced binary tree). In doing so, the number of hops, physical links and congestion are compared.


479

procedure Multiple Searching Algorithm() Input: n − tuple begin KEY S=Hilbert(n − tuple); BKEY SNode = Subset of Keys ∈ KEY S achieved by Bruijn links from N ode; HKEY SNode = Subset of Keys ∈ KEY S achieved by Hilbert links from N ode; List=N odesource .Searching Nodes(BKEY SNodesource ,HKEY SNodesource ); return List; end List Function Searching Nodes(BKEY SNode ,HKEY SNode ) begin List = ∅; Parallel foreach Key ∈ BKEY SNode do Node = Bruijn Search(Key); List = List + Node; List = List + Node.Searching Nodes(BKEY SNode ,HKEY SNode ); end if HKEY SNode = ∅ then Node=This node; while N ode.key = HKEY SNode do Node = Node.Hilbert successor; end List = List + Node; List = List+ Node.searching Nodes(BKEY SNode ,HKEY SNode ); end return List; end

Algorithm 1. Multiple Searching Algorithm 6.1

Number of Hops

Fig. 3 shows the number of hops performed by DisCoP in relation to Chord (key length = 16) for simple queries (left) and Baton for range queries (right). In Fig. 3(left), we can see that the complexity of DisCoP is always below the Bruijn key length, which is the theoretical upper limit of a Bruijn graph. This demonstrates the good behaviour of the virtualization of nodes mechanism used by Bruijn. Furthermore, the DisCoP curve has a logarithmic shape. This corroborates the simple search cost, determined in section 5. In addition, the Bruijn plots depict that the number of hops decreases when k increases. It means that, in any case, the Bruijn graph can be adapted to give better search performance than Chord. In the specific case shown in Fig. 3(left), Bruijn improves Chord by a k = 4. Fig. 3(right) shows the performance of multiple searching, comparing our proposal with respect to Baton. Note that Baton only supports a range search for a single attribute, whereas our proposal can perform multiple attribute searches. We have defined a multiple 4 − tuple, which is made up with a random range


22

Chord Key length=16 bits Bruijn Key length=16, k=2 Bruijn Key length=8, k=4 Theorical Bruijn Key length=16, k=2 Theorical Bruijn Key length=8, k=4

Searching Latency (no. Hops)

20 18 16 14 12 10 8 6 4

400

Bruijn Key length=8, k=4, with Extra Link (Parallel) Bruijn Key length=8, k=4, without Extra Link (Parallel) Bruijn Key length=8, k=4, with Extra Link (Serial) Bruijn Key length=8, k=4, without Extra Link (Serial) Baton

350

Searching Latency (no. Hops)

480

300 250 200 150 100 50

2 0

0 1

128

256

384

512

Nr. Peers

640

768

896

1024

0

128

256

384

512

640

768

896

1024

Nr. Peers

Fig. 3. Number of hops. Bruijn vs. Chord with simple queries (left) and Bruijn vs Baton with range/multiple queries (right).

variability of 10% with respect to the maximum number of values for each attribute. This on average is 63 simple 4 − tuple. Likewise, the Bruijn performance was tested enabling/disabling the extra links described in Sec. 4.2, and enabling/disabling the parallel search used by the Alg. 1. From Fig. 3(right), we can see that, excepting the Bruijn serial case without extra links, the rest of the Bruijn cases behave much better than Baton. Likewise, the Bruijn serial search improves around 150% with the use of extra links, which corroborates the need for adding these extra links in the Bruijn graph. Furthermore, the Bruijn parallel search drastically reduces the latency in relation to the serial case, producing a gain of around 500%. 6.2

Physical Links

Fig. 4 shows the average physical links per node by scaling the system. We can see how Bruijn obtains a lower number of links than Chord and Baton. It can be appreciated that Bruijn links increase with k. Likewise, the Chord and Baton behaviour is logarithmic and that of the Bruijn is almost constant. Note that Bruijn, with fewer physical links than Chord and Baton, is even able to improve the number of hops (see Fig. 3(left and right)). 6.3

Congestion

Fig. 5 shows the congestion obtained per node for a complete graph between Bruijn and Chord for the simple query case (left) and Bruijn and Baton for the range/multiple query case (right). The congestion of each peer has been measured as the number of messages crossing through a peer in relation to the total number of messages in the system. In all cases, the congestion is always lower than 1%. In general, we can see how congestion is better balanced in Bruijn at the expense of scarcely increasing the average. If we consider that the saturation of a portion of nodes can cause networking bottlenecks, dropping the overall performance drastically, the Bruijn option is the best one. It can be


20

481

Chord Key Length=16 bits Bruijn Key length=16, k=2 Bruijn Key length=8, k=4 Baton

18 16

Nr. Links/Peer

14 12 10 8 6 4 2 0 0

128

256

384

512

640

768

896

1024

Nr. Peers

Fig. 4. Number of links for Bruijn, Chord and Baton 1

1

Chord Key length=10 bits Bruijn Key length=10, k=2

0.8

% Congestion

% Congestion

0.8

Baton Bruijn Key length=10, k=2

0.6

0.4

0.2

0.6

0.4

0.2

0

0 0

128

256

384

512

640

768

896

1024

0

Index Peer

128

256

384

512

640

768

896

1024

Index Peer

Fig. 5. Congestion. Bruijn vs. Chord (left) and Bruijn vs Baton(right)

appreciated in Fig. 5(right) as similar results were obtained for the Baton case. To sum up, Bruijn is the best candidate for the upper level of the proposed overlay network.

7


In this paper, a computing resource discovery mechanism oriented to P2P computing is presented. Our proposal is based on an architecture made up of three different layers: the top layer is a Hilbert function, which classifies the nodes into markets according to their computational resources, the medium layer is a Bruijn graph, which links the different markets into a cohesive system and the bottom layer is a set of trees (markets), where each tree gathers those nodes with similar resources. In addition, some improvements have been proposed to optimise the Hilbert-Bruijn interaction. A look-up query mechanism is also introduced over this architecture. This provides simple and multiple query facilities with a low algorithmic cost.

482


The simple query proposal was compared with another simple query algorithm widely used in the literature, Chord. Given that Chord does not implement multiple queries, we also compared our range/multiple query proposal with a binary-tree structure named Baton. For all the topologies, we measured the number of hops, physical links and congestion per node. In general, our results show that our proposal behaves much better than the other topologies. In relation to the number of hops, our proposal can be tuned to improve the Chord case for simple queries, whereas in the multiple case, the DisCoP architecture drastically improves the Baton results due to the improvements introduced into the Bruijn graph (virtualization of nodes and addition of extra links). Likewise, our architecture maintains a constant number of links, lower than Chord and Baton. Finally, the congestion evaluation shows how our proposal balances the congestion throughout the system better than the others. The future trend is directed towards analysing the optimal balance between the number of Hilbert keys (markets), the range of values for a specific attribute (i.e. CPU or Memory) related to those keys and the maximum size of the markets. Likewise, we are interested in proposing a new scheduling mechanism to exploit our architecture efficiently.

References 1. Albrecht, J., Oppenheimer, D., Patterson, D., Vahdat, A.: Design and implementation tradeoffs for wide-area resource discovery. ACM Transactions on Internet Technology (TOIT) 8(4) (2008) 2. Bharambe, A., Agrawal, M., Seshan, S.: Mercury:supporting scalable multiattribute range queries. In: Special Interest Group on Data Communication (SIGCOMM 2004), vol. 34, pp. 353–366 (2004) 3. Buyya, R., Murshed, M.: Gridsim: A toolkit for the modeling and simulation of distributed resource management and scheduling for grid computing. Concurrency and Computation: Practice and Experience (CCPE) 14(13), 1175–1220 (2002) 4. Cabanillas, D., Willmott, S.: Studying viable free markets in peer-to-peer file exchange applications without altruistic agents. In: Joseph, S., Despotovic, Z., Moro, G., Bergamaschi, S. (eds.) AP2PC 2006. LNCS (LNAI), vol. 4461, pp. 151–158. Springer, Heidelberg (2008) 5. Castell` a, D., Barri, I., Rius, J., Giné, F., Solsona, F., Guirado, F.: Codip2p: A peerto-peer architecture for sharing computing resources. In: International Symposium on Distributed Computing and Artificial Intelligence, DCAI 2008, vol. 50, pp. 293– 303 (2008) 6. Castell` a, D., Blanco, H., Giné, F., Solsona, F.: A computing resource discovery mechanism over a p2p tree topology. In: 9th International Meeting High Performance Computing Science, VECPAR 2010 (2010) 7. Costa, P., Napper, J., Pierre, G., Steen, M.V.: Autonomous resource selection for decentralized utility computing. In: The 29th IEEE International Conference on Distributed Computing Systems (ICDCS 2009), Montreal, Canada (2009) 8. Foster, I., Iamnitchi, A.: On death, taxes, and the convergence of peer-to-peer and grid computing. In: Kaashoek, M.F., Stoica, I. (eds.) IPTPS 2003. LNCS, vol. 2735, pp. 118–128. Springer, Heidelberg (2003)


483

9. Gupta, R., Sekhri, V., Somani, A.K.: Compup2p: An architecture for internet computing using peer-to-peer networks. IEEE Transactions Parallel Distributed Systems 17(11), 1306–1320 (2006) 10. Jagadish, H.V., Ooi, B.C., Vu, Q.H.: Baton: A balanced tree structure for peerto-peer networks. In: Very Large Data Base Endowment (VLDB), pp. 661–672 (2005) 11. Loguinov, D., Kumar, A., Rai, V., Ganesh, S.: Graph-theoretic analysis of structured peer-to-peer systems: Routing distances and fault resilience. In: 2003 conference on Applications, technologies, architectures, and protocols for computer communications, pp. 395–406. ACM Press, New York (2003) 12. Lua, E.K., Crowcroft, J., Pias, M., Sharma, R., Lim, S.: A survey and comparison of peer-to-peer overlay network schemes. IEEE Communications Surveys and Tutorials 7, 72–93 (2005) 13. McNab, R., Howell, F.: Simjava (1998), http://www.dcs.ed.ac.uk/home/hase/simjava/ 14. Meshkova, E., Riihij¨ arvi, J., Petrova, M., M¨ ah¨ onen, P.: A survey on resource discovery mechanisms, peer-to-peer and service discovery frameworks. Computer Networks 52(11), 2097–2128 (2008) 15. Mokbel, M.F., Aref, W.G., Kamel, I.: Analysis of multi-dimensional space-filling curves. Geoinformatica 7(3), 179–209 (2003) 16. Moon, B., Jagadish, H.V., Faloutsos, C., Saltz, J.H.: Analysis of the clustering properties of the hilbert space-filling curve. IEEE Transactions on Knowledge and Data Engineering 13, 124–141 (2001) 17. Schmidt, C., Parashar, M.: Squid: Enabling search in dht-based systems. Journal Parallel Distributed Computing 68(7), 962–975 (2008) 18. Stoica, I., Morris, R., Karger, D., Kaashoek, M.F., Balakrishnan, H.: Chord: A scalable peer-to-peer lookup service for internet applications. In: Proceedings of Special Interest Group on Data Communication (SIGCOMM 2001), San Deigo, CA, pp. 149–160 (2001)

Sampling Bias in BitTorrent Measurements Boxun Zhang, Alexandru Iosup, Johan Pouwelse, Dick Epema, and Henk Sips Parallel and Distributed Systems Group Delft University of Technology, Delft, the Netherlands {B.Zhang,A.Iosup,J.Pouwelse,D.H.J.Epema,H.J.Sips}@tudelft.nl Abstract. Real-world measurements play an important role in understanding the characteristics and in improving the operation of BitTorrent, which is currently a popular Internet application. Much like measuring the Internet, the complexity and scale of the BitTorrent network make a single, complete measurement impractical. While a large number of measurements have already employed diverse sampling techniques to study parts of BitTorrent network, until now there exists no investigation of their sampling bias, that is, of their ability to objectively represent the characteristics of BitTorrent. In this work we present the first study of the sampling bias in BitTorrent measurements. We first introduce a novel taxonomy of sources of sampling bias in BitTorrent measurements. We then investigate the sampling among fifteen longterm BitTorrent measurements completed between 2004 and 2009, and find that different data sources and measurement techniques can lead to significantly different measurement results. Last, we formulate three recommendations to improve the design of future BitTorrent measurements, and estimate the cost of using these recommendations in practice.

1

Introduction

Peer-to-Peer file-sharing networks such as BitTorrent serve tens of millions of users daily and are responsible for a significant percentage of the total Internet traffic. Much effort [1, 2, 3, 4, 5, 6] has been put in the last decade into empirical measurements of P2P file-sharing systems including BitTorrent, with the purpose of understanding and improving their use. Similarly to early Internet measurement efforts [7, 8], due to the size of the complete network all BitTorrent measurements have employed data sampling techniques, from periodic measurements to the focus on specific BitTorrent communities. Despite this situation, there currently exists no comprehensive evaluation of the sampling biases, that is, of the ability to objectively represent the characteristics of BitTorrent, introduced by BitTorrent measurements. This work presents the first such investigation. Understanding sampling biases in BitTorrent measurements can benefit the research on BitTorrent in the following ways. First, it can lead to a better understanding of the commonalities and of differences among different parts of the BitTorrent network by explicitly comparing the measurement results. In the Internet community, this ”search for invariants” process [7] fostered many new research opportunities [8]. From the large number of empirical BitTorrent P. D’Ambra, M. Guarracino, and D. Talia (Eds.): Euro-Par 2010, Part I, LNCS 6271, pp. 484–496, 2010. c Springer-Verlag Berlin Heidelberg 2010

Sampling Bias in BitTorrent Measurements

485

measurements [2, 3, 9, 10], few [9, 10] consider even aspects of the sampling bias problem. Second, understanding sampling biases leads to better understanding of the usage of measurement techniques, which is key to designing and improving BitTorernt measurements. It is symptomatic for the current (lack of) understanding of BitTorrent measurement techniques that there is no agreement on the Internet traffic share due to BitTorrent–though caching companies have put forth estimates of over 50% in 2008 [11] and 30% in 2005 [12]. Towards understanding sampling biases in BitTorrent measurements, our main contribution is threefold: 1. We propose a method for exposing the sampling biases in BitTorrent measurements that focuses on both the measured BitTorrent components and the volume of the measured data (Section 3); 2. Using fifteen diverse BitTorrent datasets (Section 4) we show that the measured BitTorrent components and the volume of the measured data can both significantly bias measurement results (Section 5); 3. We formulate recommendations to improve future BitTorrent measurements, and estimate the costs of implementing these recommendations (Section 6). This work is further motivated by the needs of two ongoing initiatives. First, we are continuing our previous work [13] on building a publicly-accessible P2P Workloads Archive, which will include in a first phase the tens of P2P measurement datasets we have acquired since 2003, and in particular the fifteen datasets we use in this work. Second, within the QLectives project1 we are currently conducting new measurements of the BitTorrent network, and need guidelines such as the ones presented in this article.

2

Background

In this section we introduce the background needed to understand the remainder of this work. Much of the P2P-related terminology and BitTorrent description in this section is adapted from our previous work on BitTorrent [3, 9, 13]. A P2P system is a system that uses P2P technology to provide a set of services; this group of services forms together an application such as file sharing. We call peers the participants in a P2P system that contribute to or use the system’s resources and services. A peer is completely disconnected until it joins the system, and is active until it leaves the system. A real user may run several peer sessions; the sessions are not overlapped in time. We call a swarm the group of peers, from all the peers in a P2P system, that interact with each other for a specific goal, such as transferring a file. A swarm starts being active when the first peer joins that swarm, and ends its activity when its last peer leaves. The lifetime of a swarm is the period between the start and the end of the swarm. A community is the group of peers who are or can easily become aware of the existence of each other’s swarms. 1

QLectives, EU FP7 project, http://www.qlectives.eu/

486

B. Zhang et al.

Our view on P2P systems considers three levels of operation. A P2P system includes at least a peer level, but may also include any of the community and swarm levels. The definitions of community, swarm, and peers presented here are general for peer-to-peer systems, though their implementation may differ with the P2P protocol. For example, BitTorrent and eDonkey have different interpretation and thus implementation of the swarm concept. In this work we focus on BitTorrent, which includes all the three levels of operation defined earlier in this section. The files transferred in BitTorrent contain two parts: the raw file data and a metadata (directory and information) part. Peers interested in a file obtain the file’s metadata from a web site (the community level of BitTorrent) and use the peer location services offered by a tracker (the swarm level of BitTorrent) to find other peers interested in sharing the file. The raw file is then exchanged between peers (the peer level of BitTorrent). To facilitate this exchange, the raw data are split in smaller parts, called chunks. Thus, to obtain a complete file a user has to obtain all the chunks by using three application levels. In this paper we distinguish between the complete and the transient swarm population: we define the population of a swarm as the set of all peers ever present in the swarm at any time during the measurement, and a snapshot of a swarm as the set of peers present at a specific time during the measurement.

3

Exposing the Sampling Bias

In this section we introduce our method for exposing sampling bias in BitTorrent measurements. Our method focuses on two main questions that define a measurement process: What is the relationship between the data source and bias? and What is the relationship between the data volume and bias? The first question stems from the complexity and scale of BitTorrent. For example, there currently exist tens of popular BitTorrent communities, many operating independently from the others and having specific usage characteristics. The second question expresses the trade-off between accuracy and measurement cost (see also Section 6). 3.1

Method

We say that a measurement conducted on a BitTorrent component is affected by a sampling bias if the sampled characteristics are significantly different from the real characteristics of the real BitTorrent component.To analyze the sampling bias we need an understanding of the real characteristics (the ground truth), a conceptual framework for understanding the differences between the sampled and real characteristics, and metrics to quantify these differences. Ground Truth: The characteristics of BitTorrent are largely unknown: there currently exists no model of a dynamic and heterogeneous BitTorrent swarm, and scientists do not possess even a single complete measurement dataset comprising every message exchanged between the peers of a BitTorrent community of significant size. Thus, and similarly with the situation of exposing sampling biases


487

for Internet measurements [14,8], we need to trace the presence of sampling bias without a ground truth. Instead, we make the observation that if measurements are unbiased, the measured characteristics should remain the same regardless of the sampling. Following this observation, we define for a measurement the complete dataset, which is the practical equivalent to the ground truth, as the dataset collected with maximal measurement capability. For example, if a real measurement has sampled data every 5 minutes, it can be used to understand the sampling bias resulting from larger sampling intervals, such as 10 minutes. Conceptual Framework: We use the term variability when comparing properties, e.g., the average peer download speed or the empirical distribution of file sizes for a community, as measured across different BitTorrent components when using the same measurement technique. We also use the term accuracy when examining how data collected with different techniques compares with the complete dataset. Metrics: We estimate the sampling bias using two metrics: – The Coverage metric, defined as the percentage of sampled peers or events, from the peers or events comprised in the complete dataset. – The Error/deviation of values metric, which mimics traditional statistical approaches for comparing probability distributions of random variables. The Kolmogorov-Smirnov test [15] uses the D characteristic to estimate the maximum distance between the cumulative distribution functions (CDFs) of two random variables. Similarly, we use the D characteristic to compare the measured and the complete dataset values. Following traditional work on computer workload modeling (see [16] and the references within), we say that measurements resulting in errors above 10% (D metric above 0.1) have very low accuracy, and that measurements with 5–10% error have low accuracy. 3.2

Data Sources

Depending on the selection of the data source, we distinguish three main sources of variability or accuracy: 1. The Measurement level. In Section 2 we have defined three levels for a P2P application, community, swarm, and peer. Measuring at any single level may result in measurement inaccuracy. For example, measurements taken at peer level, with an instrumented peer as the measurement tool, may fail to contact all peers in a swarm, since peers have limited uptime (presence) in the system. 2. The Community type. Many types of communities exist in the BitTorrent world, and this diversity leads to high variability in measurement results. We categorize BitTorrent communities based on the type of content they share, either general or specific content. The specific content may be further divided into content sub-types such as video, operating system, etc.; Garbacki et al. [17] have identified around 200 content sub-types for the SuprNova community. 3. The Passive vs. Active Measurements. Following the terminology introduced in our previous work [9], peer-level measurements are active if the instrumented peers acting as measurement probes initiate contact with other

488

B. Zhang et al.

BitTorrent peers, and passive if they wait for externally initiated contacts. In contrast to passive measurements, active measurements require that the other peers are accessible, for example, they are not behind a firewall. The 2007 measurement by Xie et al. [18] shows that up to 90% of the peers in a live streaming application are firewalled, and that less than 20% of them by-pass the firewalls. 3.3

Data Volume

The data volume is another major discriminant among measurements: 1. Sampling rate and Duration. Since peer-to-peer systems have properties that evolve over time, measurements have to observe the same property repeatedly. Then, the data volume is the product between the sampling rate and the measurement duration; reducing either leads to lower data volumes, but may also lead to inaccuracy. Rates of a sample every 2.5 [3,9] to 30 minutes [10], and durations of a few days [9] to a few months [2] have been used in practice. 2. Number of communities and Number of swarms. BitTorrent communities may share properties, and within a community the most populated swarms may account for most of the traffic. Thus, including in the measurement fewer communities and swarms may reduce the volume of acquired data without reducing accuracy. Until the recent study of four communities [10], measurements have often focused on one community [3, 9], and even on only one swarm [2]. 3. Long-term dynamics. Many BitTorrent communities have changed significantly over time or even disappeared. Thus, measurements should make efforts to catch long-term system dynamics, including seasonal and multi-year patterns. In practice, the only long-term BitTorrent measurements are the five months study of Izal et al. [2] and our own year-long measurement of SuprNova [3].

4

The Collected Traces

To understand sampling bias in BitTorrent measurements, we have acquired 15 long-term traces from 9 BitTorrent communities of hundreds of thousands of peers and responsible for transferring yearly over 13 peta-bytes of data. Overall, this paper investigates the largest number of BitTorrent datasets, to-date, as summarized in Table 1; for a complete description of the traces see our technical report [19]. Traces studied in this work are available at the Peer-to-Peer Trace Archive (http://p2pta.ewi.tudelft.nl); for more details and analysis results of the Archive please refer to our technical report [20]. To ensure heterogeneity among the limited number of traces, we have taken into account the following controllable factors when collecting the traces. The traces cover different community types (sharing from very specific to general content), community sizes (from small communities to the largest in the world of the time of the data collection–T1’04/SuprNova in 2004 and T2’05/PirateBay in 2005), and measurement duration (from a few days to one year). To assess the bias introduced by not following the long-term evolution of BitTorrent communities, several traces include two datasets, acquired in 2005 and 2009 respectively.


489

Table 1. Summary of the datasets used in this work. Only the datasets for traces T1’04 and T2’05 have been previously analyzed [3, 9]. ID

Trace Description Period Sampling Torrents Sessions Traffic (Content Type) (minutes) (GB/day) BT-TUD-1, SuprNova Oct 2003 to Dec 2004 60 32,452 n/a n/a (General) 06 Dec 2003 to 17 Jan 2004 2.5 120 28,423,470 n/a BT-TUD-2, PirateBay 05-11 May 2005 2.5 2,000 35,881,338 32,000 (General) LegalTorrents.com 22 Mar to 19 Jul 2005 5 41 n/a 698 (General) 24 Sep 2009 onwards 5 183 n/a 1,100 etree.org 22 Mar to 19 Jul 2005 15 52 165,168 9 (Recorded events) 24 Sep 2009 onwards 15 45 169,768 143 tlm-project.org 22 Mar to 30 Apr 2005 10 264 149,071 735 (Linux) 24 Sep 2009 onwards 10 74 21,529 15 transamrit.net 22 Mar to 19 Jul 2005 5 14 130,253 258 (Slackware Unix) 24 Sep 2009 onwards 5 60 61,011 840 unix-ag.uni-kl.de 22 Mar to 19 Jul 2005 5 11 279,323 493 (Knoppix) 24 Sep 2009 onwards 5 12 160,522 348 idsoftware.com 22 Mar to 19 Jul 2005 5 13 48,271 19 (Game Demos) 24 Sep 2009 onwards 5 37 14,697 12 boegenielsen.net 22 Mar to 19 Jul 2005 5 15 36,391 308 (Knoppix)

T1’04 T2’05 T3’05 T3’09 T4’05 T4’09 T5’05 T5’09 T6’05 T6’09 T7’05 T7’09 T8’05 T8’09 T9’05

5

The Results

In this section we investigate the effects of the different ways of data source selection and data volume reduction on the variability and accuracy of BitTorrent measurements. 5.1

Data Source Selection

First we will assess the effects of the selection of the data source on the accuracy and variability of BitTorrent measurements. Finding: Measurements performed at a single operational level of BitTorrent can lead to very low accuracy. When measuring swarm dynamics, we observe in T1’04 that in 4 out of 10 swarms, peer level measurements capture 12000 Flashcrowd

T1.003 Swarm Level T1.003 Peer Level

Community Throughput (GB)

Number of Peers

4000

30% difference 3000

2000 infrastructure failure 1000

0 20/Dec 2003

9000 55% difference 6000

3000 T7 ’05 Community Level T7 ’05 Swarm Level

0 27/Dec 2003

03/Jan 2004

10/Jan 2004

Date

(a) Swarm dynamics

17/Jan 2004

23/Apr 2005

30/Apr 2005

07/May 2005

14/May 2005

21/May 2005

Date

(b) Cumulative community throughput

Fig. 1. Comparison of the results obtained from measuring at different levels

B. Zhang et al. 1

1

0.8

0.8

0.6

0.6

CDF

CDF

490

0.4

0.4

T2 ’05 T3 ’05 T4 ’05 T5 ’05 T8 ’05 T1 Dec ’04

0.2 0 0

200

400

600

800

T5 ’09 T6 ’09 T7 ’09 T8 ’09

0.2 0 1000

File Size (MB)

(a) File Size Distributions

1200

0

10

20

30

40

50

Upload speed (KB/s)

(b) Upload Speed Distributions

Fig. 2. Comparison of properties in different BitTorrent communities

many fewer peers than swarm level measurements. For swarm 003 of T1’04, during the flashcrowd (see Figure 1a), the peer-level coverage drops below 70% of the swarm-level coverage. Later, due to the overloaded infrastructure, the peerlevel coverage is even down to 50% for about half the duration of the flashcrowd. Similar effects can be observed from measurements at both the community and the swarm level in many communities. Taking T7’05 as an example, the community throughput as obtained from community-level measurements is more than 50% higher than the swarm-level numbers, and so the latter are very inaccurate (see Figure 1b). Finding: Measuring different BitTorrent communities may lead to very different results. As discussed in Section 3, the diversity in community types contributes to a high variability in the measurement results of different communities. For several BitTorrent communities, we show in Figures 2a and 2b the cumulative distribution functions (CDFs) of the file size and the upload speed, which both differ significantly among communities. Furthermore, for these communities we do not see a correlation between these characteristics and the focus of the community on general versus specific content. We also observe similar differences in the distributions of the swarm sizes and the download speeds in several communities. Finding: The results of passive and active measurements differ significantly, because the presence of firewalled peers is significant in BitTorrent, and the firewalled and non-firewalled peers have different uptime. For example, less than 60% of the peers are non-firewalled in the T1’04 (SuprNova) trace [3]. An in-depth analysis of the impact of (the fraction of) firewalled peers on upload/download performance in four communities was presented by Mol et al. [21]; their analysis also covers the data of T2’05 (The Pirate Bay), which were collected using both active and passive measurements. It turns out that over 65% of the peers discovered using the active measurements are firewalled, and that 96% of the swarms have over 50% firewalled peers. The same study found that, because BitTorrent rewards peers with connectivity, non-firewalled peers exhibit


491

80% less uptime than firewalled peers. However, it is not possible to only perform passive measurements, since in this case it is costly to guarantee coverage and a steady sampling rate. 5.2

Sampling Rate and Duration

In order to understand the effect of the sampling rate on the measurement accuracy, we take the original datasets obtained in our measurements, with their original sampling rates and durations, as the basis. From these datasets we derive new datasets with other (lower) sampling rates by sampling the original datasets at various intervals. Similarly, we derive datasets with shorter durations by simply taking contiguous pieces of the appropriate lengths of the original datasets. We then compare the properties of the original and newly obtained datasets. Finding: When measuring at the peer level, increasing the sampling interval leads to a higher inaccuracy and variance. Figure 3a shows how the average snapshot coverage of several swarms drops when the sampling interval is increased from 2.5 minutes (the original sampling interval) to 30 minutes. Figure 3b shows the statistics (min, max, median, and first and third quartile) of the distribution of the snapshot coverage of a single swarm obtained at different multiples of the original sampling interval. Only when the sampling interval does not exceed 7.5 (15) minutes is the median snapshot coverage at least 90% (80%).

original measurement

80 60 40 20

15 min

0 0

5

T1 ’04 swarm 003 T1 ’04 swarm 005 T1 ’04 swarm 009 T1 ’04 swarm 010 10

15

100 80 60 40

7.5 min 10 min 15 min

20 T1 ’04 swarm 003

0 20

25

Normalized Sampling Interval (unit = 2.5 min)

Population Coverage (%)

100

Snapshot Coverage (%)

Average Snapshot Coverage (%)

Finding: Reducing the measurement duration quickly reduces the coverage of the measurements—a doubling of the sampling interval leads to smaller coverage loss than a halving of the measurement duration. Figure 3c depicts for several datasets the population coverage resulting from various measurement durations, including the original duration of 4 weeks. The different datasets exhibit different losses of accuracy when cutting the original measurement duration in half, but all quickly converge to less than 20% coverage with subsequent duration cuts. Swarm 003 from trace T1’04 is the least affected by halving the original measurement duration at over 80% coverage, but the coverage of the complete community in T5’05 is then already below 40%. This large difference is the result of the system evolution: swarm 003 exhibits a large

2

4

6

8

10

12

14

16

Normlized sampling interval (unit=2.5 min)

(a) Average snapshot cover- (b) Statistics of snapshot age coverage

100

original T5 ’05 T1 ’04 swarm 003 T1 ’04 swarm 009

80 60 40 20 0 4

2

1

1/2

1/4

1/8

1/16

Measurement Duration (weeks)

(c) Population coverage

Fig. 3. The effects of changing sampling interval and measurement duration

492

B. Zhang et al.

flashcrowd [3] in which the peers are caught for at least a week until obtaining the content they want, while in the tlm-project.org community the peers obtain results quickly and then leave the swarm without returning. 5.3

Number of Communities and Torrents

To understand the effect of the number of measured communities on the measurement accuracy, we select one month worth of data from six communities in our datasets. We order these communities by the total amounts of traffic generated by that community, and we compute all the investigated characteristics for all the selected communities together. Then we iteratively remove from the considered datasets the data of the community with the lowest rank (total traffic), and repeat the whole process until only one community is left for analysis. We apply a similar approach to understand the effects of the number of torrents, in this case ordering the torrents by the number of peers in the corresponding swarms. This approach has been taken by many reported measurements [3, 9]. Finding: Measuring only one community or only one swarm is insufficient to obtain accurate results. Figure 4 depicts the CDF of the session length for a varying number of communities. The CDF stabilizes only after at least four communities are considered together. We have obtained similar results for other characteristics, including the download speed and the total amount of data downloaded per peer (see [19] for complete results). Similar results are found when investigating the effects of the number of measured swarms. The upload speed is the only characteristic that we have investigated that does not require multiple communities to be measured to obtain accurate results (see [19]). 5.4

Catching Long-Term Dynamics

To show evidence of the long-term evolution of BitTorrent we first extract from our long-term traces blocks of contiguous data and then compare them.

1

1

0.8

0.8

8% difference 6 communities 5 communities 4 communities 3 communities 2 communities 1 community

0.6 0.4 0.2 0 0

500 1000 1500 Session Length (Minute)

CDF

CDF

Finding: Yearly patterns exist in BitTorrent, but are diverse across different communities. Multi-year evolution is also present but is hard to characterize. Figure 5 depicts the evolution of the file sizes from Dec 2003

8% difference

0.6

T1 ’03 Dec T1 ’04 Feb T1 ’04 Apr T1 ’04 Jun T1 ’04 Aug T1 ’04 Oct

0.4 0.2 0

2000

0

500 1000 1500 2000 2500 3000 3500 4000 File Size (MB)

Fig. 4. Session Length Distributions from Fig. 5. Monthly file size distributions from measuring different numbers of communi- T1 (SuprNova). Only every second month ties is depicted.


493

1

CDF

0.8 0.6 0.4 0.2

World ’04 World ’05

0 0

20

40

EU ’04 EU ’05 60

80 0

Download speed (KB/s)

20

40

NA ’04 NA ’05 60

80 0


20

40

SA ’04 SA ’05 60

80 0


20

40

60

80


CDF

Fig. 6. Change of download speed in different continents and in different years. (Data from trace BT-TUD-1, BT-TUD-2.). 1

1

1

0.8

0.8

0.8

0.6

0.6

0.6

0.4

0.4 T3 ’05 T3 ’09

0.2 0 0

300 600 900 File Size (MB)

0.4 T4 ’05 T4 ’09

0.2 0 1200

0

300 600 900 File Size (MB)

T8 ’05 T8 ’09

0.2 0 1200

0

300 600 900 File Size (MB)

1200

Fig. 7. Change of file size from 2005 to 2009, by community

to Nov 2004; for clarity, the figure only shows curves corresponding to every second month. The results taken in Dec 2003 reveal very different values of this characteristic versus the other measurements. Smaller differences appear between consecutive months, and overall the file sizes decrease slowly over time. We also show the evolution of swarm sizes and community in [19]. In our previous work [9] we have observed that the average download speed has doubled between T1’04 and T2’05. We now show that the evolution is not consistent across all users. Figure 6 depicts the download speed distributions for T1’04 and T2’05 with users grouped by continent. The left sub-graph confirms our previous remark, and the other sub-graphs show that the increase of download speed varies among the different continents. The increase of download speed by community is also found for traces T5-T8 (see [19]). Similarly, we show in Figure 7 evidence that the file size distribution changed from 2005 to 2009, but the actual ”direction” of the change varies greatly by community.

6

Recommendations

Based upon our findings in section 5, we formulate three recommendations for performing BitTorrent measurements with reduced sampling bias: 1. Measure at multiple operational levels of BitTorrent; 2. Use a short sampling interval (below 10 minutes) for peer level measurements;

494

B. Zhang et al.

3. Measure several communities (at least 4) and swarms in each community (at least 50) for long periods of time (months or longer). We now estimate the cost of using these recommendations for new BitTorrent measurements, based on our past experience [9]. Following our recommendations, the new measurements should sample 200 swarms across 4 communities, use a sampling interval of 10 minutes, include both active and passive measurements, and have a duration of one year. To collect the trace T2’05, we have sampled 2,000 swarms from the PirateBay community every 2.5 minutes for 6 days, performing both passive and active measurements. We have used in the process about 300 computers and generated about 300GB of raw data. Assuming that new measurements are performed using an infrastructure with similar capabilities, they will require the use of 30 machines and 500 GB storage space for a period of one year. Excluding human and data curation costs, and using the cheapest cloud resources from the Amazon Web Services (compute resources cost $0.085 per hour, data storage $0.150 per GB, all data transfers free until June 30, 2010, $0.10 per GB afterwards) the total cost is $22,463, of which only $125 are data transfer and storage costs.

7

Related Work

Much previous work has been dedicated to measurements of P2P file-sharing networks [1, 2, 3, 4, 5, 6]. From the few studies that tried to reduce the sampling bias or even recognized its existence, none covers the full set of sampling bias sources addressed in this work. In general, under the assumption that ”more is better”, these studies obtained data over long periods of time [10,22], from more peers [10,6,9], for more files [10, 9, 23] and communities [10, 23], and filtered the raw data before analysis to eliminate some of the measurement biases [10, 6]. Closest to our work, Stutzbach et al. [24] assess the bias incurred by sampling unstructured P2P file-sharing networks, and propose the MRWB technique to sample without bias from such networks. Stutzbach et al. [22] further investigate the bias introduced by the sampling rate. However, their analysis and techniques are valid only for unstructured networks, but do not apply to the tracker-based, centralized peer discovery mechanism in BitTorrent.

8

Conclusion

Despite almost a decade of BitTorrent measurements, little is known about the sampling bias of the measurement results. However, reliable measurements of BitTorrent deployments are needed to improve the quality of service for millions of BitTorrent users. In this paper we have presented an investigation of the factors that cause inaccuracy in and variability among BitTorrent measurements. Towards this end, we have first proposed a method for exposing sample bias. Our method includes a taxonomy of the sources of sampling bias comprising two axes—data source selection and data volume reduction—totaling six sources,


495

and two metrics for quantifying the sampling bias. Then, we have evaluated the effects of the different sources of sampling bias using fifteen real traces taken from nine BitTorrent communities. Our results indicate that current measurement techniques can lead to significant sampling bias in the measurement results. Based on our findings, we have formulated three recommendations to improve future BitTorrent measurements, and analyzed the cost of implementing them in practice. For the future, we plan to extend our work towards developing a complete method for accurate yet low-volume BitTorrent measurements.

Acknowledgements The research leading to this contribution has received funding from the European Community’s Seventh Framework Programme in the P2P-Next project under grant no 216217. The Peer-to-Peer Trace Archive (http://p2pta.ewi.tudelft.nl) provides us with the traces studied in this work.

References 1. Sen, S., Wang, J.: Analyzing peer-to-peer traffic across large networks. In: Proc. of ACM SIGCOMM IMW, pp. 137–150 (2002) 2. Izal, et al.: Dissecting BitTorrent: Five Months in a Torrent’s Lifetime. In: Proc. of PAM, Antibes Juan-les-Pins, France, pp. 1–11 (2004) 3. Pouwelse, J., Garbacki, P., Epema, D., Sips, H.: The BitTorrent P2P file-sharing system: Measurements and analysis. In: Castro, M., van Renesse, R. (eds.) IPTPS 2005. LNCS, vol. 3640, pp. 205–216. Springer, Heidelberg (2005) 4. Bhagwan, R., Savage, S., Voelker, G.M.: Understanding availability. In: IPTPS, pp. 256–267 (2003) 5. Gummadi, K., Dunn, R., Saroiu, S., Gribble, S., Levy, H., Zahorjan, J.: Measurement, modeling, and analysis of a peer-to-peer file-sharing workload. In: ACM Symp. on Operating Systems Principles, SOSP (2003) 6. Handurukande, S.B., Kermarrec, A.M., Fessant, F.L., Massoulié, L., Patarin, S.: Peer sharing behaviour in the eDonkey network, and implications for the design of server-less file sharing systems. In: EuroSys, pp. 359–371 (2006) 7. Arlitt, M.F., Williamson, C.L.: Web server workload characterization: The search for invariants. In: SIGMETRICS, pp. 126–137 (1996) 8. Floyd, S., Paxson, V.: Difficulties in simulating the Internet. IEEE/ACM Trans. Netw. 9(4), 392–403 (2001) 9. Iosup, A., Garbacki, P., Pouwelse, J., Epema, D.: Correlating topology and path characteristics of overlay networks and the Internet. In: Proc. CCGrid, p. 10 (2006) 10. Andrade, N., Santos-Neto, E., Brasileiro, F.V., Ripeanu, M.: Resource demand and supply in bittorrent content-sharing communities. Computer Networks 53(4), 515–527 (2009) 11. ipoque GmbH: Internet studies (2006-2009), http://www.ipoque.com/resources/internet-studies/ 12. Parker, A.: The True Picture of Peer-To-Peer File-Sharing. In: IEEE Int’l. W. on Web Content Caching and Distribution Panel (2005) 13. Zhang, B., Iosup, A., Garbacki, P., Pouwelse, J.: A unified format for traces of peer-to-peer systems. In: LSAP, pp. 27–34. ACM, New York (2009)

496

B. Zhang et al.

14. Lakhina, A., Byers, J.W., Crovella, M., Xie, P.: Sampling biases in ip topology measurements. In: INFOCOM (2003) 15. Lilliefors, H.W.: On the Kolmogorov-Smirnov test for normality with mean and variance unknown. J. Am. Stat. 62, 399–402 (1967) 16. Feitelson, D.G.: Workload modeling for performance evaluation. In: Performance, pp. 114–141 (2002) 17. Garbacki, P., Epema, D., van Steen, M.: Optimizing peer relationships in a superpeer network. In: ICDCS, p. 31 (2007) 18. Xie, S., Keung, G.Y., Li, B.: A measurement of a large-scale peer-to-peer live video streaming system. In: Proc. of ICPP, p. 57 (2007) 19. Zhang, B., Iosup, A., Pouwelse, J., Epema, D., Sips, H.: On assessing measurement accuracy in BitTorrent peer-to-peer file-sharing networks. Tech.Rep. PDS-2009005, TU Delft (2009), http://pds.twi.tudelft.nl/reports/2009/PDS-2009-005.pdf 20. Zhang, B., Iosup, A., Epema, D.: The peer-to-peer trace archive: Design and comparative trace analysis. Technical Report PDS-2010-003, Delft University of Technology (2010), http://pds.twi.tudelft.nl/reports/2010/PDS-2010-003.pdf 21. Mol, J., Pouwelse, J., Epema, D., Sips, H.: Free-riding, fairness, and firewalls in p2p file-sharing. In: P2P, pp. 301–310 (2008) 22. Stutzbach, D., Rejaie, R., Sen, S.: Characterizing unstructured overlay topologies in modern P2P file-sharing systems. IEEE/ACM Trans. Netw. 16(2), 267–280 (2008) 23. Guo, L., Chen, S., Xiao, Z., Tan, E., Ding, X., Zhang, X.: Measurements, analysis, and modeling of bittorrent-like systems. In: Internet Measurment Conference, pp. 35–48 (2005) 24. Stutzbach, D., Rejaie, R., Duffield, N.G., Sen, S., Willinger, W.: On unbiased sampling for unstructured peer-to-peer networks. IEEE/ACM Trans. Netw. 17(2), 377–390 (2009)

A Formal Credit-Based Incentive Model for Sharing Computer Resources Josep Rius, Ignasi Barri, Fernando Cores, and Francesc Solsona Computer Science Department, University of Lleida, Spain {jrius,ignasibarri,fcores,francesc}@diei.udl.cat

Abstract. Peer-to-Peer (P2P) computing, the harnessing of idle CPU cycles through the Internet, offers new research challenges in the domain of distributed computing. This paper presents an incentive mechanism based on credits, designed to operate on different types of shared computing networks such as P2P, P2P Grid, Opportunistic Grid, Desktop Grid, volunteer computing platforms, and so on. The main contribution is a new reinvestment policy called Weighted that increases peer participation significantly. This mechanism reflects P2P user dynamics, penalizes free-riders efficiently and encourages peer participation. Simulation results show that our policy outperforms alternative approaches, maximizing system throughput and limiting free-riding behaviour. Furthermore, a mathematical model has been proposed and analysed in order to formalize the policy and setting up the configuration of the principal parameters of the incentive mechanism. Keywords: P2P, shared and distributed computing, incentive mechanisms, free-riders, social welfare.

1

Introduction

Volunteer and P2P Computing has emerged as a new platform for high throughput computing over the Internet. Such systems harness CPU idle cycles from thousands of computers connected through Internet and aggregate their computational resources to execute huge and massively-parallel distributed applications that cannot be executed by standard supercomputing centres due to their costs and long execution times. P2P and volunteer computing can potentially provide access to a huge volume of cheap computer resources but it still has to deal with important challenges related to cooperation, security, fault-tolerance, availability and quality of service to widen its deployment. One of the most critical aspects in the design of such systems is the development of incentive techniques to enforce cooperation and resource sharing among participants. Cooperation is highly appreciated because users in P2P networks tend to exploit the maximum resources they are able

This work was supported by the MEyC-Spain under contract TIN2008-05913 and the CUR of DIUE of GENCAT and the European Social Fund.


498

J. Rius et al.

to obtain, offering minimum resources in response. This behaviour undermines the goal of P2P in spreading resources and imposes the concept of free-riding. Free-riding is a serious problem with P2P. Experience with peer-to-peer systems shows that in the absence of incentives for donation, a large proportion of the peers only consume the resources of the system. Free-riding leads to the need for social and economic mechanisms to balance resource usage and efforts in P2P systems. Tragedy of the commons is another major problem. It happens when many users misreport their resource features to discourage others from using them. With no cooperation among peers, the network will reach a point where it will be difficult to obtain resources. Therefore, because of their essence as being autonomous and in the absence of any central authority or explicit punishment policies, peers act in a rational selfish way causing a major difficulty in enforcing cooperation among them. Incentive policies try to persuade peers to share their resources. What makes incentive problem particularly difficult is the unique set of challenges that P2P systems pose: large populations, high turnover, asymmetry of interest, collusion, zero-cost identities, and traitors [19]. To deal with these challenges, a typical incentive system is built around two main components: the control mechanism and the incentive techniques. The control mechanism (like trust and reputation) is essential for evaluating the trustworthiness of the participating peers and combating the selfish, dishonest and malicious peer behaviour. Reputation-based systems are proposed to establish trust among peers in P2P systems [7,14]. In a reputation system, each entity’s behaviour is used to predict how it is going to behave in the future. Finally, there are trust-inference mechanisms by which information about users behaviour can be propagated throughout the system [9,21]. Incentive techniques encourage trustworthy behaviour and maximize social welfare. However, they are vulnerable to collusion and rely heavily on the users’ history. To further efficiency, researchers extended the social mechanisms into economic ones by applying monetary based schemes. Pricing has proved to be an effective means to incentivate sharing and control computer networks. It helps in recovering cost for the network operators and providing incentives for users to cooperate [10]. Several approaches have been proposed for P2P: Flat Rate [4,5], Auction-based [13], Micropayment and Stock-Market approaches [17,19]. A considerable amount of research has focused on the exploitation of economic theories through a game-theoretic approach to analyse formally how selfish behaviour impacts the performances of P2P networks. The incentive problem is usually modelled using the Generalized Prisoner’s Dilemma to study the problems of whitewashing (nodes that change their identities to clean their history) and collusion. Game theory is also useful for defining the equilibrium level, the Nash equilibrium, that optimizes the sharing capabilities of a P2P system and reduces free-riding behaviour [4,11,18]. The incentive mechanisms for P2P Computing systems and Desktop grids have been gaining attention recently. Andrade [1] presents an incentive mechanism called the Network of Favours to assemble a large grid (OurGrid P2P

A Formal Credit-Based Incentive Model for Sharing Computer Resources

499

System[3]), which makes it in the interest of each participating peer to contribute their spare resources. H-Trust [20] is a group trust management system inspired by the H-index aggregation technique. It offers a robust personalized reputation evaluation mechanism for both individual and group trusts with minimal communication and computation overheads for Peer-to-Peer Desktop Grids. In a similar way, PowerTrust [14] is also a reputation system focussed on P2P Grids. In [12], a distributed Desktop Grid is presented which promotes fairness and inhibits free-riding through a shared-history reputation mechanism designed for use in dynamic Peer-to-Peer systems. In this paper, we propose a new credit-based scheme to enforce collaboration in P2P computing systems based on the reinvestment of resource payments to incentivate sharing and discouraging free-riders. Our approach differs from others, such as CompuP2P [6], in the sense that we distribute credits non-uniformly among peers based on their contribution to the system, thus increasing system throughput. The incentive mechanism are heavily coupled with the Job scheduling policy, which is based on the reverse Vickrey auction strategy to avoid peers to offering resources at a price lower than their real cost (dumping cheating). To avoid ID-changing cheating, we implement a non-negative credit function supported by an historic term to differentiate between newcomers and old collaborative peers. The incentive scheme is evaluated using simulation, demonstrating its capacity to improve system throughput, i.e. reducing free-riding, for different workloads and user behaviour. Finally, we derive a formal model of the incentive system in order to study the most critical parameter, initial user credits, for setting up our policy. The remainder of the paper is organized as follows. In Section 2, our incentive mechanism is introduced with its deployment on a P2P computing platform. The incentive system is modelled in Section 3. We present the performance of the incentive scheme by simulation and the model to support our assumptions in Section 4 and finally, Section 5 contains the conclusions and future work.

2

Proposed Incentive Mechanism

This paper presents an incentive system based on credits, designed to operate on a shared computing network. An early version of this system was designed for a P2P computing platform with a tree topology called CoDiP2P [2,15]. This paper presents an extension of the model designed to run over any type of structure, provided this can be sub-grouped and managed by a super-peer, such as P2P, P2P Grid, Opportunistic Grid, Desktop Grid, volunteer computing and so on. The outline of the system is a set of peers, called workers, grouped in an area and managed by a master peer called Manager. Our incentive mechanism does not depend on the number of areas nor their interconnection overlay. An area will be made up of one Manager and N workers, where N depends on the network properties, bandwidth and latency. The main goal of the Manager is to manage peers in the same area and schedule jobs to be executed over the workers. At the same time, these workers can also submit jobs to their corresponding Manager. In the system, the Manager is fully responsible for (see Fig. 1):

500

J. Rius et al.

1. Initializing the incentive system, assigning the initial number of credits to each joining peer. Therefore it is necessary to know this number of credits in a suitable way. A mathematical model for solving this problem is proposed in section 3. This fact is represented in Fig. 1 by the top left peer joining the system. 2. Managing the job queue (see label 2 in Fig. 1). Peers in a work area can submit jobs to the system by sending their requests to the Manager, which manages them all and determines whether such requests can be accepted. Algorithm 1 in Section 2.1 explains the procedure for doing so. 3. Scheduling of jobs. Section 2.1 also shows the procedure that maps every part of a job (here on called tasks) to a worker of the area optimally. Due to the dynamism of peers, only best-effort applications can be afforded. QoS is another concern to be developed in the future. The Manager acts as an intermediary node between the sender and the receiving peers (those whose are assigned the jobs tasks). In market terms, the sender should be the buyer and the receiver should be the seller of the computational resources. The Manager earns a commission for this function. Credits labelled with 3 in Fig. 1 represent this operation. 4. Reinvestment. In our incentive mechanism, Managers can reinvest some of their credits, earned on each transaction, into the system to motivate the users to collaborate even more with the network’s aim. Credits labelled with 4 in Fig. 1 represent this reinvestment function.

Fig. 1. Manager functions

2.1

System Features

First of all, it must be drawn up a sharing valuation method according to a prioritizing criterion of resources (CPU, Memory, bandwidth. . . ) or to a combination of them. Notice that in P2P Grid, a resource might even be a cluster. If for example, only CPU and Memory were considered, peer i will have cost Ci = Θi (CP Ui , M emoryi ). This assessment will be in credits and will represent the selling price for system resources. In doing so, we need a broker to manage


501

each market, or in our case, each area. For simplicity, we assign this function to the Managers, but it could be assigned to other peers. Once the cost of peer resources has been defined, it is time to describe the behaviour of the Manager concerning the request for launching applications submitted by the users of the network. For this reason, two algorithms are proposed in order to define the main features of the Manager. The first is called the Job Admission Algorithm (Alg. 1), which takes into account the number of credits of the launching peer in making decisions about accepting/rejecting of an entering job. Algorithm 1. Job Admission Algorithm Require: (Job): Input parameters 1. 2. 3. 4. 5. 6. 7.

M anager receives the Job from the Buyer if (System is idle ) then M anager processes the Job else if (Buyer has enough credits) then M anager queues the Job else M anager rejects the Job

Due to the volatility of the resources shared in the system, if resources are not assigned, they are wasted. Taking this into account, if the system is underused, a job launch request will be accepted, even if the launching peer does not have enough credits (see lines 2-3 of Algorithm 1). Consequently, some peers receive work and credits. These additional earned credits will encourage the possibility of new jobs being accepted by the Manager, who will also earn more credits by doing the job assignment. This mechanism will increase the execution of jobs in the absence of activity and encourage the usage of the system. Algorithm 2, called the Job Scheduling Algorithm, takes two parameters into account, (a) antiquity, the time elapsed for the job, waiting in the Manager queue to be served and (b) historical term, defined as the number of tasks served by the launching peer. Both parameters are stored in the Manager and are used to prioritize the queue. The scheduling policy is based on the reverse Vickrey auction strategy. This method consists on always choosing the worker (W orker1 ) with the lowest cost (V alue1 ). The selected peer is rewarded with the second lowest offered cost (V alue2 ) minus its own, that is, the lowest one (V alue1). It can be said that using the reverse Vickrey auction, the incentive mechanism avoids peer cheating by offering values lower than their respective cost. In the case where one peer wants to be selected and tries to drop its corresponding value (V aluei ), if the second lowest cost is above it, selecting it will cause a negative profit. Thus cheating is discouraged. The Manager will also always choose the best option in the scheduling algorithm, in order to maximize its profit (in credits), defined as the difference between the highest, V aluemax, and the lowest cost of the selected node, V aluesel .

502

J. Rius et al.

Another issue to consider is that, over time, Managers can become large credit hoarders. In our incentive scheme, Managers can reinvest their credits in the system in order to motivate the workers to collaborate with the platform. Reinvestment (calculated in line 9) is the total number of credits to be reinvested by the Manager and it depends on the reinvestment percentatge (δ). Unlike other approaches that distribute their credits uniformly between the peers (called Uniform policy) [6], in this study we propose a more selective distribution policy. The proposed incentive mechanism is responsible for managing reinvestment credits and distributing them among active and collaborative peers (free-rider peers are not rewarded). This new policy, called Weighted, distributes credits non uniformly among peers based on their contribution to the system and especially their computing needs.

Algorithm 2. Job Scheduling Algorithm - Reverse Vickrey Auction Based Require: (Job): Input parameters 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13.

3

M anager gets the next Job from its queue P rof it M anager := 0, Reinvestment := 0, P rof it W orkers := 0 for each T ask in Job do W orker1 := Free W orker with lowest V alue in area V alue1 := Lowest free W orker cost in the area V alue2 := Second lowest free W orker cost in the area P rof it W orker1 := V alue2 − V alue1 P rof it M anager := P rof it M anager + (V aluemax − V alue1 ) ∗ (1 − δ) Reinvestment := Reinvestment + (V aluemax − V alue1 ) ∗ δ P rof it W orkers := P rof it W orkers + P rof it W orker1 M anager sends T ask to W orker1 end for T otal Cost := P rof it M anager + P rof it W orkers

Mathematical Model

The assignment of credits among peers depends on the cost of their computational resources. For this reason, the mechanism needs to know the suitable amount of credits to be assigned to each peer. Optimal values should obtain a maximum throughput of computing resources. For reasons of simplicity, our first approximation does not take free-riding dynamism and user behaviour into account and only one area is considered. Furthermore, it is designed for the Weighted policy. In order to find this optimal value, a mathematical model was developed. It formalizes the proposed incentive mechanism and helps us to find that value. First of all, some notation must be introduced:

A Formal Credit-Based Incentive Model for Sharing Computer Resources Π Ci I T M Li Eij E∗i Ei∗ γ λ Ri βi

: : : : : : : : : : : : :

503

set of peers with |Π| = N . cost of node i. initial number of credits assigned to new-coming peers. total number of credits. That is, T = N · I. credits of the Manager. number of tasks that node i wants to submit to the system. number of tasks that node i executes for node j. number of tasks accepted for node i. number of tasks executed by node i. percentage of credits earned by Managers for each task assignment. percentage of credits that Managers will reinvest in the system. reinvested credits by the Manager into node i. normalized function representing the benefit of node i, defined as: i E∗ if Li > 0 i βi (Li , E∗ , T ) = Li 0 if Li = 0

Let us now put this into mathematical terms. The problem is to maximize expected system utilization (equation 1): maximize E

N

βi (Li , E∗i , T )

≡

maximize E

i=0

N Ei

∗

i=0

Li

(1)

subject to a “Non-negative constraint” (constraint 2), which says that credits initially assigned to a peer, plus the earned ones, minus those paid and plus the reinvested credits must be at least zero: I+

N j=0

(1 − γ) · Eij · Ci − Eji · Cj + Ri ≥ 0

∀i ∈ N

(2)

This constraint 2 along with the historical term, manages to prevent ID-changing cheating because the mechanism avoids prioritizing malicious ID-changing peers over collaborating peers who have consumed more resources than they have contributed. In addition, the mechanism is also able to distinguish between a malicious ID-changing peer who never sells resources and a collaborating peer that has bought at least as many resources as it has sold. Constraint 3 defines the total number of credits that the Manager will have depending on the number of tasks executed in the system and on the commission charged (γ) in each one. M =γ·

N i=0

(Ei∗ · Ci ) + I

(3)

λ·M ≥

N

Ri

(4)

i=0

Finally, the maximum number of credits that the Manager will reinvest in the system is limited to λ · M (constraint 4).

504

4

J. Rius et al.


In this section, the proposed incentive mechanism is evaluated through simulation tests. The results obtained with this simulator are analysed statistically and finally the results of the mathematical model are also presented. 4.1

User Behaviour

Before evaluating the incentive mechanism proposed in section 2, the user behaviour of P2P computing networks has to be defined. Stutzbach [16] showed how session lengths are fitted by Weibull or log-normal distributions, but not by the exponential or Pareto. In our experiments, the Weibull distribution was chosen. Two kinds of peers were taken into account, free-rider and collaborative peers, ρ being the probability that a collaborator becomes a free-rider and σ the probability that a free-rider becomes a collaborator. Probability ρ depends on the ratio (ColRatio) between the profit a peer obtains from the system (LaunchedT asks − RejectedT asks) and the resources of the same peer used by the system (ExecutedT asks). So, ρ probability has an exponential behaviour with the form ρ = β ColRatio , where β is a constant in the range [0..1]. ColRatio =

LaunchedT asks − RejectedT asks ExecutedT asks

(5)

Probability σ depends on the free-rider ratio (FreeRatio) expressed in equation 6. This ratio is the percentage of rejected jobs per peer (i.e. peer penalization). Here, probability σ has a polynomial behaviour with the form σ = F reeRatioα , where α is a constant in the range (0..∞). F reeRatio =

RejectT asks LaunchedT asks

(6)

Peer dynamism is characterized by α and β. 4.2

Simulation Results

In this section, the performance of the proposed system was evaluated through simulations tests with the help of SimJava [8]. SimJava is a process-based discrete event simulation api for Java. The results presented below show the percentage of launched tasks accepted by the system according to the percentage of freeriders, reinvestment percentage applied by the Manager (δ), the initial number of credits assigned to each newcomer and the user behaviour. Each simulation was carried out assuming the same number of peers and workload (100,000 peers, the launching jobs had 500 tasks on average). The results were obtained for the Uniform and Weighted policies. The idea was to show the gain obtained from applying our incentive mechanism and the proposed reinvestment policy.


(a) Uniform (0% free-riders)

(b) Weighted (0% free-riders)

(c) Uniform (50% free-riders)

(d) Weighted (50% free-riders)

505

Fig. 2. Static number of free-riders in the system

Static free-riders. Figure 2 shows the percentage of accepted tasks by fixing the number of free-riders throughout the simulation at 0% and 50% for each policy. In Fig. 2, initial credits must be multiplied by the Average Cost of the system peers. In all the cases, the Weighted policy reach the physical limits of the system much faster (100% of the launched tasks are accepted). Differences in the policy behaviour can be better appreciated when the system has enough resources to accept entering tasks, because the incentive mechanism still has leeway. Weighted, with half the credits, reaches the same percentage of accepted tasks as Uniform. The benefits achieved by applying reinvestment are also significant. Dynamic free-riders. If the same tests are repeated by introducing the user behaviour defined in subsection 4.1, the effects of free-riders can be appreciated. Despite the initial percentage of free-riders, the incentive mechanism tends to equal stabilize by reducing them significantly. This reduction depends on the probabilities of changing the peer role, ρ and σ (from free-rider to collaborator and vice versa). Different user behaviours were tested by varying the parameters α and β. The most important common feature in all the cases was that the Weighted policy achieves social benefits much faster than the Uniform one. Figure 3 shows the results obtained in a favourable situation, i.e., free-riders easily convertible to collaborators and collaborators resistant to becoming free-riders. In this situation, the mechanism stabilizes the free-riders at around 10%. It can clearly be seen that the reinvestment mechanism also plays an important role.

506

J. Rius et al.

(a) Uniform (0% free-riders)

(b) Weighted (0% free-riders)

(c) Uniform (50% free-riders)

(d) Weighted (50% free-riders)

Fig. 3. Dynamic number of free-riders in the system

The Weighted policy optimizes the same number of resources better than the Uniform one. 4.3

Statistical Analysis

The simulated results were analysed with the help of Minitab. A multivariate linear regression equation was estimated by the ordinary least squares method for the Dynamic case (similar results were obtained for the Static environment). The resulting equation in a logarithmic form is: lnA = −5.99 + 0.619lnC + 0.203lnR − 0.0166lnF + 0.333lnP

(7)

where A is the number of accepted tasks, C is the total credits in the system, R the reinvestment rate, F the number of free-riders and P indicates the Policy used (1: Uniform and 2: Weighted). This expression has a goodness of fit 80%. All variables are statistically significant, as their corresponding p-values are 0 and they are also linearly independent since there is a low correlation between them. There are only a 5.37% of atypical samples. The equation is shown in logarithmic form because the weights are the significance (in percentage) of its variables. Thus on doubling the credits or the reinvestment, A increases by 61.9% and 20.3% respectively. Doubling the freeriders, A decreases 1.6%. And finally using the Weighted policy instead of the Uniform one, A increases by an average of 33%.


507

Fig. 4. Initial credits versus accepted tasks. (left) Solver. (right) Linear Regression.

4.4

Mathematical Model Results

Finally, the mathematical model described in Section 4 was implemented with the aid of the CPLEX solver. The objective was to obtain the minimum number of total credits for a certain profit. In the case of the optimal profit, equation N βi (Li , E∗i , T ) is equal to 1. E i=0

Fig. 4 shows the percentage of accepted tasks according to the initial credits assigned to newcomers and for different reinvestment rates applied by the Manager. Fig. 4(left) plots the results of the mathematical model obtained with the solver and Fig. 4(right) shows the linear regression obtained by the Minitab tool for the Static case with 0% of free-riders and for the Weighted policy, that is, the conditions closest to the mathematical model. Taking into account that Fig. 4(right) is a linear regression, the results are quite close. This corroborates the usefulness of our model. It can be seen how benefits increase with the credits and reinvestment. The more reinvestment the Manager applies, the lower the percentage of tasks rejected for lack of credits, so the optimum would allow the Manager to return all profits to the system. By doing so, the Manager loses the motivation to always choose the cheapest workers while assigning tasks, so we have to find a balance between the Manager gains and the benefit of the system. Thus, the selection of initial credits and the reinvestment will be in the hands of the system admin.

5

Conclusion and Future Work

In this paper, we presented a P2P credit-based incentive scheme for different types of shared computing networks. This incentive mechanism implements a non-negative credit function (to prevent ID-changing cheating) with an historic term used to differentiate between newcomers and old collaborative peers. Reinvestment of the credits using a Weighted policy increases peer cooperation enormously. We also propose a user-behaviour model, taking free-riders and collaborator users into account. Simulation results based on this model show that our proposed incentive mechanism outperforms alternative approaches, maximizing system throughput and limiting free-riding, being also tolerant of changes in

508

J. Rius et al.

user behaviour. Moreover, we have presented an accurate mathematical model that helps the system administrator to decide some mechanism parameters, such as the initial credits to assign to newcomers. Future work will look into the interconnection of different areas by fixing some exchange rate representing the cost of sending jobs (basically bandwidth and latency) from one area to another with the aim of decreasing the number of rejected applications due to lack of resources. Furthermore, new reinvesting policies must be developed in order to increase the system performance even more. Finally, bearing user behaviour in mind in the mathematical model will be an important step towards obtaining more realistic results.

References 1. Andrade, N., Brasileiro, F., Cirne, W., Mowbray, M.: Automatic grid assembly by promoting collaboration in peer-to-peer grids. J. Parallel Dist. Comput. (2007) 2. Castella, D., Barri, I., Rius, J., Gine, F., Solsona, F., Guirado, F.: Codip2p: A peer-to-peer architecture for sharing computing resources, vol. 50, pp. 293–303 (2008) 3. Cirne, W., Brasileiro, F., Andrade, N., Costa, L., Andrade, A., Novaes, R., Mowbray, M.: Labs of the world, unite!!! Journal of Grid Computing (2006) 4. Courcoubetis, R., Weber, C.: Incentives for large peer-to-peer systems. IEEE Journal on Selected Areas in Communications 24(5), 1034–1050 (2006) 5. Mironov, I., Golle, P., Leyton-Brown, K., Lillibridge, M.: Incentives for sharing in peer-to-peer networks. In: Fiege, L., M¨ uhl, G., Wilhelm, U.G. (eds.) WELCOM 2001. LNCS, vol. 2232, pp. 75–87. Springer, Heidelberg (2001) 6. Gupta, R., Sekhri, V., Somani, A.K.: Compup2p: An architecture for internet computing using peer-to-peer networks. IEEE Trans. Parallel Distrib. Syst (2006) 7. Zage, D., Hoffman, K., Nita-Rotaru, C.: A survey of attack and defense techniques for reputation systems. ACM Computing Surveys 41(4), 31 (2009) 8. Kreutzer, W., Hopkins, J., van Mierlo, M.: Simjava—a framework for modeling queueing networks in java, pp. 483–488 (1997) 9. Liang, Z., Shi, W.: Analysis of ratings on trust inference in open environments. Perform. Eval. 65(2), 99–128 (2008) 10. Loginova, O., Lu, H., Wang, X.H.: Incentive schemes in peer-to-peer networks. The B.E. Journal of Theoretical Economics 9(1) (2009) 11. Feldman, M., Papadimitriou, C., Chuang, J., Stoica, I.: Free-riding and whitewashing in peer-to-peer systems. IEEE Journal on Selected Areas in Communications 24(5), 1010–1019 (2006) 12. Merz, P., Kolter, F., Priebe, M.: Free-riding prevention in super-peer desktop grids. In: ICCGI 2008: Proceedings of the 2008 The Third International Multi-Conference on Computing in the Global Information Technology (iccgi 2008), Washington, DC, USA, 2008, pp. 297–302. IEEE Computer Society, Los Alamitos (2008) 13. Mondal, A., Kumar-Madria, S., Kitsuregawa, M.: An economic incentive model for encouraging peer collaboration in mobile-p2p networks with support for constraint queries. Peer-to-Peer Networking and Applications 2(3), 230–251 (2009) 14. Rahbar, A., Yang, O.: Powertrust: A robust and scalable reputation system for trusted p2p computing. IEEE Trans. on Par. and Dist. Syst. 18(4), 460–473 (2007) 15. Rius, J., Cores, F., Solsona, F.: A new credit-based incentive mechanism for p2p scheduling with user modeling, pp. 85–91 (2009)


509

16. Stutzbach, D., Rejaie, R.: Understanding churn in peer-to-peer networks. In: IMC 2006: Proceedings of the 6th ACM SIGCOMM conference on Internet measurement, pp. 189–202. ACM, New York (2006) 17. Tan, G., Jarvis, S.A.: A payment-based incentive and service differentiation scheme for peer-to-peer streaming broadcast. IEEE Trans. Parallel Distrib. Syst. 19(7), 940–953 (2008) 18. Turaga, D.S., van der Schaar, M., Sood, R.: Stochastic optimization for content sharing in p2p systems. IEEE Transactions on Multimedia 10(1), 132–144 (2008) 19. Zghaibeh, M., Harmantzis, F.C.: A lottery-based pricing scheme for peer-to-peer networks. Telecommunication Systems 37(4), 217–230 (2008) 20. Zhao, H., Li, X.: H-trust: A robust and lightweight group reputation system for peer-to-peer desktop grid. In: The 28th Int. Conference on Distrib. Comp. Syst. Workshops, Washington, DC, USA, pp. 235–240. IEEE Computer Society, Los Alamitos (2008) 21. Zhou, R., Hwang, K., Cai, M.: Gossiptrust for fast reputation aggregation in peerto-peer networks. IEEE Trans. on Know. and Data Eng. 20, 1282–1295 (2008)

Distributed Systems and Algorithms Pascal Felber2 , Ricardo Jimenez-Peris2 , Giovanni Schmid1 , and Pierre Sens1 1


2

Parallel computing is increasingly exposed to the development and challenges of distributed systems, such as asynchrony, long latencies, network partitions, failures, disconnected operations, heterogeneity and protocol standardization. Furthermore, distributed systems are becoming larger, more diverse and more dynamic (changing topology, highly dynamic number of participants). This EuroPar topic provides a forum for research and practice about new advances in distributed computing and distributed algorithms. Submissions were encouraged across the whole area with emphasis on design and practice of distributed algorithms, scalability, concurrency, performance evaluations, and self-organized distributed systems. This year nineteen papers were submitted to Topic 8 representing a wide spectrum of research areas in distributed computing. Of the nineteen submissions, seven have been accepted for publication. Each paper was reviewed by four reviewers (two papers received only three reviews). The accepted papers cover a wide range of challenges in the distributed system and algorithms topic. Three papers address fundamental distributed algorithm problems: M. Raynal and D. Imbs explore progress conditions in asynchronous read/write systems, R. Baldoni et al. introduce a new shared set object suitable for synchronous dynamic systems, and C. Johnen and F. Mekhaldi propose a self-stabilizing construction of bounded size weight-based clusters. Two papers propose and evaluate new distributed protocols: K-T. Moeller et al. propose an adaptive technique to avoid false sharing in memory transactions whereas T. Ropars and C. Morin evaluate a new event logger on multi-core systems. Finally, two papers focus on specific problems: S. Gorlatch and O. Scharf describe a distributed simulation of the dynamics of crowds inside closed buildings and J. Piquer and J. Bustos-Jiménez propose and evaluate a new frame allocation for networked cameras. The paper “The x-Wait-freedom Progress Condition” by Michel Raynal and Damien Imbs was selected as distinguished paper. We would like to take the opportunity of thanking the authors who submitted a contribution, the Euro-Par Organizing Committee, and the external referees, whose efforts have made this conference and this topic possible.

P. D’Ambra, M. Guarracino, and D. Talia (Eds.): Euro-Par 2010, Part I, LNCS 6271, p. 510, 2010. c Springer-Verlag Berlin Heidelberg 2010

Improving Message Logging Protocols Scalability through Distributed Event Logging Thomas Ropars and Christine Morin INRIA, Centre Rennes - Bretagne Atlantique, Rennes, France {Thomas.Ropars,Christine.Morin}@inria.fr

Abstract. Message logging is an attractive solution to provide fault tolerance for message passing applications because it is more scalable than coordinated checkpointing. Sender-based message logging is a well known optimization that allows to save messages payload in the sender memory and so only the events corresponding to message receptions have to be logged reliably using an event logger. In existing work on message logging, the event logger has always been considered as a centralized process, limiting message logging protocols scalability. In this paper, we propose a distributed event logger. This new event logger takes advantage of multi-cores processors to be executed in parallel with application processes. It makes use of the nodes’ volatile memory to save events reliably. We propose a simple gossip-based dissemination protocol to make application processes aware of new stable events. We evaluated our distributed event logger in the Open MPI library with an optimistic and a pessimistic message logging protocol. Experiments show that distributed event logging improves message logging protocols scalability.

1

Introduction

The size of parallel and distributed computing systems keeps on growing, allowing to execute very large scale applications. In such systems, the mean time between failures is reduced to a few hours [1] so executing distributed applications composed of thousands of processes requires fault tolerance mechanisms. Message passing is a widely used programming model for distributed applications. MPI [2] has become a de facto standard for HPC applications. Fault tolerance for message passing applications is provided by rollback-recovery protocols [3]. Because of its simplicity, coordinated checkpointing is usually used. However coordinated checkpointing does not scale well because the failure of one process makes all application processes rollback to the last coordinated checkpoint. When the failure rate increases, this behavior can prevent the application from progressing. Message logging is an alternative to coordinated checkpointing that can tolerate higher failure rate [4]: message logging protocols don’t require all application processes to rollback in the event of a single failure. In message logging protocols, message exchanges between application processes are logged during failure free execution to be able to replay them in the same order after a failure. Under the piecewise deterministic assumption [5], P. D’Ambra, M. Guarracino, and D. Talia (Eds.): Euro-Par 2010, Part I, LNCS 6271, pp. 511–522, 2010. c Springer-Verlag Berlin Heidelberg 2010

512

T. Ropars and C. Morin

this is enough to ensure that the recovered application state is consistent. Using sender-based message logging [6], messages payload and associated delivery events can be handled separately to improve message logging performance. Messages payload is saved in the volatile memory of the sender while determinants [7] representing the delivery events are saved on reliable storage. The interface between the application processes and the reliable storage is called an event logger [8]. The event logger performance has a major impact on message logging protocols efficiency. However, until now no study focused on the event logger implementation. In recent works the event logger was always implemented as a single centralized process [9, 8, 10], limiting its scalability. In this paper, we propose a distributed event logger that makes use of the volatile memory of the nodes executing application processes to save determinants. One event logger process is executed on every node in parallel with application processes, taking advantage of multi-cores architectures. We study this new event logger in the context of O2P [10], an active optimistic message logging protocol targeting scalability. In that context, we also propose to use a simple gossip-based dissemination protocol to distribute information on new stable determinants to all application processes. We implemented our distributed event logger in the Open MPI library and evaluated it on Grid’5000. Our experiments run with O2P and with a pessimistic message logging protocol show that our distributed event logger improves message logging protocols scalability. The rest of this paper is organized as follows : Section 2 describes the context of this work. It highlights the limits of a centralized event logger and presents the protocols used for evaluation. In Section 3, we detail our new distributed event logger. In Section 4 we present and discuss the results of our experiments on Grid’5000. Finally, conclusions and future work are detailed in Section 5.

2

Context and Motivations

In this section, we first describe the system model we consider. Then we present message logging principles, and focus on the related work studying the impact of an event logger on message logging. Finally, we detail the two protocols we use to evaluate our work. 2.1

System Model

We consider an asynchronous distributed system and a distributed application composed of n processes. Each process has a unique identifier. Application processes communicate by explicitly sending and receiving messages. Communications channels between processes are reliable and FIFO. But there is no bound on message transmission delay and no order between messages sent on different channels. We consider the traditional assumption in message logging protocols that process execution is piecewise deterministic, i.e. it can be seen as a sequence of deterministic state intervals started by a non deterministic event corresponding to a message reception. Message exchanges create dependencies between processes’ state intervals. Processes’ state intervals are partially ordered

Improving Message Logging Protocols Scalability

513

by the Lamport’s happen-before relation [11].Finally, we assume a fail-stop failure model. Data saved in the volatile memory of a node are lost in a failure. 2.2

Event Logging in Message Logging Protocols

Message logging protocols are classified into three families [7]: pessimistic, optimistic, and causal. All of them can be combined with sender-based message logging [6]. In sender based message logging, the payload of a message is saved in the volatile memory of its sender, reducing the amount of data to save reliably. Only a determinant, describing the message’s delivery event, has to be saved reliably. A determinant becomes stable when it is saved on a reliable storage. The state interval started by the corresponding delivery event becomes stable too. If some determinants are lost in a failure, some messages cannot be replayed and some processes might become orphans. An orphan process is a process whose state causally depends on a lost message. After a failure, a message logging protocol has to ensure that the application is recovered in a consistent global state, i.e. a state without orphan processes. Another way to express this is to say that the application has to be recovered in a state that could have been seen during failure free execution. Message logging protocols differ in the way they ensure that a consistent global state can be found after a failure. Pessimistic protocols log determinants synchronously on reliable storage to ensure the always no orphan condition. Optimistic protocols log determinants asynchronously on reliable storage to improve performance on failure free execution compared to pessimistic message logging, with a risk of orphan process creation. Optimistic message logging protocols have to trace causal dependencies between processes during failure free execution to be able to detect orphans. Causal message logging protocols piggyback the determinants required to replay a message on the message itself. Message logging protocols can be combined with uncoordinated checkpointing without the risk of domino effect. Checkpointing can be seen as a mean to reduce logs size. We will not talk about checkpointing in the rest of this paper. Event Logger. The concept of event logger has been introduced in the work on MPICH-V [9]. It’s an interface to the reliable storage, that can execute protocol specific code. It has been shown that using an event logger can dramatically improve performance of causal message logging protocols [8] by reducing the amount of data to piggyback on messages. It has also been shown that it can improve optimistic message logging performance [10]. Message logging protocols performance depends on the event logger performance. However, up to now no study has focused on the event logger: it has always been considered as a single centralized process. We have shown in [10] that a centralized event logger can limit the performance of message logging protocols for large scale applications. Event Logging Model. It has recently been proposed to refine the message logging model for MPI applications to better discriminate non deterministic events [12]. Some MPI communications primitives are actually deterministic (e.g.

514


MPI Send(), MPI Recv, etc.), and so don’t require the corresponding message exchanges to be logged. By taking into account MPI primitives semantics in the message logging model, the number of events to log is reduced and message logging protocols scalability is improved [13]. This solution can complement the solution proposed in this paper. However, in this paper we consider the more generic message logging model. The main reason is that due to the limited size of our testbed, the generic model, that generates more events to log, could better highlight the properties of our solution. 2.3

Evaluated Protocols

To evaluate our work, we focus on two protocols. The first one is O2P [10], an active optimistic message logging protocol. Our distributed event logger has been initially designed for this protocol. We also evaluate the new event logger with a pessimistic message logging protocol to show that it can improve the scalability of any message logging protocol. In this section, we present these two protocols and describe how the event logger impacts them. O2P. O2P is an active optimistic message logging protocol. O2P saves determinants as soon as possible on reliable storage to reduce the risk of orphan process creation compared to other existing optimistic message logging protocols that save determinants periodically [14, 5]. Furthermore O2P takes into account stable determinants to reduce the amount of data to piggyback on application messages, reducing the overhead induced by the protocol on failure free execution. Whereas other optimistic protocols piggyback n-size vector clocks (n being the number of processes in the application) on each application message to track dependencies between processes, O2P only piggyback the timestamps corresponding to the non stable state intervals a message causally depends on. To make processes aware of new stable state intervals, O2P takes advantage of a centralized event logger. Since it is centralized, the event logger knows all the determinants that are saved reliably. So it can maintain a vector representing the latest stable state intervals in the application. This vector is sent to the application processes as acknowledgment for saved determinants. We evaluated how efficiently O2P manages to reduce the amount of data piggybacked on application messages using 4 applications of the NAS Parallel Benchmarks [15]. The experimental setup is described in Section 4.1. The results, illustrated by Figure 1, show that O2P can efficiently reduce this amount of data compared to an optimistic protocol that saves determinants periodically. This experiment also reveals the scalability limit of a centralized event logger. For large number of processes, the event logger becomes overloaded and doesn’t manage to save determinants on time anymore. As a result O2P doesn’t manage anymore to reduce the number of timestamps piggybacked on messages. Since it takes longer to save the determinants, the risk of loosing determinants in the event of a failure increases too. Finally, it can be noticed that with LU, which is the application with the highest communication rate, the event logger crashes because of being too much overloaded from 128 processes.


Number of Timestamps

CG Class C

FT Class C

120

300

100

250

80

200

60

150

40 20

100 50

0

0 16

32

64 128 256

Number of Processes

LU Class C 80 70 60 50 40 30 20 10 0

16

32

64

128 256

Number of Processes

515

MG Class C 250 Optimist O2P

200 150 100 50 0 16

32

64

128 256

Number of Processes

16

32

64

128 256

Number of Processes

Fig. 1. Number of timestamps piggybacked on application messages for 4 applications of the NAS Parallel Benchmarks

Pessimistic Message Logging Protocol. We evaluated our event logger with a pessimistic message logging protocol because the performance of such protocols highly depends on the event logger performance since determinants are logged synchronously. In the pessimistic message logging protocol we use for evaluation, when a process receives a message the corresponding determinant is immediately sent to the event logger. When a process wants to send a message, it has to wait first for all the determinants corresponding to the messages it has already received to be logged.

3

Distributed Event Logging

To improve event logging scalability, we propose a distributed event logger. This event logger makes use of the volatile memory of the nodes hosting application processes to save determinants reliably. It takes advantage of the multi-cores architecture of new processors to be executed in parallel with the application processes. To distribute information on new stable state intervals to all application processes, we use a simple gossip-based dissemination protocol. The distributed event logger is a distributed application. One process of the event logger is executed on each node hosting application processes, as described in Figure 2. In this figure, we consider 4 nodes provided with a 4 cores processor and a distributed application composed of 12 processes, i.e. 3 processes per node. The algorithm executed by the event logger processes is described in Figure 3. We detail it in the rest of this section. Determinants Logging. To save a determinant reliably, it is copied in the volatile memory of other nodes. Copying data on n+1 nodes ensures their availability despite up to n node failures. We define ReplicationDegree as the first parameter of our distributed event logger. In the example of Figure 2, the replication degree is 2 : process p1 sends the determinants corresponding to the messages it receives to the event loggers of node 2 and node 3. When EL2 and EL3 have saved the determinant of an application message locally, they send back the corresponding acknowledgment to EL1 (line 13 of

516


Fig. 2. The Distributed Event Logger

Figure 3). This acknowledgment includes the identifier of the process the application message was sent to (dest) and the receiver sequence number (rsn) of that message, the rsn corresponding to the identifier of the state interval started by the reception of the message by process dest . An event logger handles the acknowledgments for all the processes hosted by the local node. When ReplicationDegree acknowledgments have been received for a determinant, it is considered as stable (Line 20 of Figure 3). To make application processes aware of new stable determinants, a vector called StableV ector is used. Entry i of StableV ectorx is the last stable state interval of process pi known on node X. This vector is shared between the event logger process and the application processes executed on the same node. When a determinant becomes stable, the event logger updates the corresponding entry in the StableV ector (Line 23 of Figure 3). An application process reads this vector to be aware of its new stable state intervals. Gossip-Based Dissemination Protocol. Some message logging protocols, like O2P described in Section 2.3 or causal message logging protocols [8], can exploit information on new stable state intervals in the application to improve performance. To make application processes aware of new stable state intervals, we propose to use a simple gossip-based protocol (line 24-28 of Figure 3). When a new state interval becomes stable for one local process, the event logger randomly selects GossipDegree other event loggers and sends them its updated StableV ector. Thus the event loggers can update their local StableV ector and allow application processes to be aware of new stable state intervals.


1: 2: 3:

517

Variables: ReplicationDegree // Replication degree for the determinants AckListx ← ∅ // List of acknowledgments received by ELx . To each ack is associated the number of times it has been received. StableV ectorx ← [⊥, ..., ⊥] // Vector of stable state intervals of node X GossipDegree // Gossip degree

4: 5: 6: 7: Upon delivery of a message msg by process pi on node X 8: Initialize detmsg // detmsg is the determinant of msg 9: Send (detmsg ,X) to ReplicationDegree remote event loggers 10: 11: Upon delivery of message (detmsg ,Y ) by ELx on node X 12: Save detmsg in volatile memory 13: Send ackmsg to ELy // ackmsg .ps ← detmsg .dest, acqmsg .si ← detmsg .rsn 14: 15: Upon delivery of ackmsg by ELx on node X 16: if ackmsg ∈ AckListx then 17: AckListx [ackmsg ] ← AckListx [ackmsg ] + 1 18: else 19: AckListx [ackmsg ] ← 1 20: if AckListx [ackmsg ] = ReplicationDegree then 21: StableV ectorx [ackmsg .ps] ← ackmsg .si 22: if Gossip is activated then 23: Send StableV ectorx to GossipDegree event loggers randomly chosen 24: 25: Upon delivery of StableV ectory by ELx on node X 26: Update StableV ectorx with StableV ectory Fig. 3. Distributed Event Logger Algorithm

Recovery. The message logging protocols’ recovery is not impacted by the use of a distributed event logger. A deterministic function is used to select the event logger processes each process sends its determinant to so that they can be found again easily after a failure using the same function.

4

Evaluation

This section presents the evaluation of our distributed event logger. We first describe the experimental setup. Then we present and analyze the results of our experiments with an optimistic and a pessimistic message logging protocol. 4.1

Experimental Setup

For evaluations, we used 4 class C applications of NAS Parallel Benchmark Suite [15]. The results are mean values over 5 executions of each test. Prototype Description. We implemented the two protocols described in Section 2.3 in the Open MPI Library1 . A description of our implementation of O2P in Open MPI can be found in [10]. The pessimistic message logging protocol is implemented the same way. Our distributed event logger is implemented as an MPI application. An event logger process is executed on each node where an application process is going 1

http://svn.open-mpi.org/svn/ompi/trunk:r22041

518


to be executed. The user’s MPI application and the distributed event logger are interconnected using MPI Comm connect() and MPI Comm accept() functions, allowing to create an MPI communicator between the two applications. Thus any process of the user’s MPI application can communicate with any event logger process through MPI Send/Receive calls. On a local node, the StableV ector is implemented using a System V shared memory segment. Testbed. All the experiments presented in this paper are run on the Rennes cluster of Grid’5000. All the nodes have a Gigabit Ethernet interface and are connected to a single Cisco 6509 switch. We use 112 nodes including 53 Carri System CS-5393B nodes supplied with 2 Intel Xeon L5420 (4 cores) processors and 32 GB of memory; 30 Dell PowerEdge 1950 supplied with 2 Intel Xeon 5148 LV (2 cores) processors and 8 GB of memory; 29 Dell PowerEdge 1950 supplied with 2 Intel Xeon 5148 LV (2 cores) processors and 4 GB of memory. For the experiments with the centralized event logger, one node is dedicated to the execution of the event logger. 4.2

Results

First we present results highlighting the scalability and performance of the distributed event logger compared to a centralized event logger. Then we show the impact of the gossip degree and the replication degree on the behavior of the distributed event logger. Scalability. We have seen in Section 2.3 that measuring the number of timestamps attached by O2P on application messages shows the scalability of an event logger. We run the same applications as in Figure 1 using our distributed event logger with the parameters ReplicationDegree = 2, and GossipDegree = 1. The results, presented in Figure 4, compare the number of timestamps that would attach an optimistic protocol that saves determinants periodically (Optimist ), with the one attached by O2P using a centralized event logger (C.E.L.) and a distributed event logger (D.E.L.). First, with the distributed event logger O2P always manages to efficiently reduce the number of timestamps piggybacked on application messages. Furthermore, even for large number of processes with LU, the distributed event logger manages to handle the determinants and doesn’t crash. Second, we see the advantage of sharing information on stable state intervals between all the processes hosted on the same node. Since we are using 112 nodes, there is at least 2 processes per node with an application composed of 256 processes. For 3 of the benchmarks, the amount of data piggybacked for 256 processes is less than for 128 processes. We also ran the experiments without gossiping (GossipDegree = 0). The results are not displayed in Figure 4 because the amount of data piggybacked on messages is then always very close to the one piggybacked by an optimistic protocol that saves determinants periodically. This proves that the gossiping protocol has a positive impact on the behavior of O2P.



CG Class C

FT Class C

120

300

100

250

80

200

60

150

40 20

100 50

0

0 16

32

64 128 256

Number of Processes

LU Class C 80 70 60 50 40 30 20 10 0

16

32

64

128 256

MG Class C 250 Optimist C.E.L. D.E.L.

200 150 100 50 0 16

Number of Processes

519

32

64

128 256

16

Number of Processes

32

64

128 256

Number of Processes

Fig. 4. Amount of data piggybacked on application messages with a centralized and a distributed event logger CG Class C

Overhead

2 1.8 1.6

LU Class C

MG Class C

20

Open MPI C.E.L. D.E.L.

4 3.5 3 2.5 2 1.5 1

15 10

1.4 1.2

5

1

0 64

128

256

Number of Processes

64

128

256

Number of Processes

64

128

256

Number of Processes

Fig. 5. Performance of a pessimistic message logging protocol with a centralized and a distributed event logger

Performance. Figure 5 presents the normalized execution time for 3 of the NAS Benchmarks with a pessimist message logging protocol, using a centralized and a distributed event logger. Execution time without message logging equals 1. We choose not to display the results for FT because it’s the application with the lowest communication rate, and so the less interesting case. The distributed event logger’s parameters are ReplicationDegree = 2 and GossipDegree = 0. Since in a pessimistic message logging protocol a process always waits for its current state interval to be stable before sending a message, gossiping information on new stable state intervals is useless. Figure 5 clearly demonstrates the better performance and scalability of the distributed event logger. For CG and MG, the overhead with the distributed event logger remains very small and almost constant (around 10%). Figure 6 presents the same experiments with O2P. The distributed event logger’s parameters are ReplicationDegree = 2 and GossipDegree = 1. The results show again that the distributed event logger outperforms the centralized event logger. For LU, the distributed event logger allows to execute the application with large number of processes. It has to be noticed that in these experiments, the distributed event logger provides a better availability for determinants that the centralized event logger. The centralized event logger saves the determinants in the volatile memory of its node. So with the centralized event logger, the data replication degree is 1 whereas it is set to 2 in our experiments with the distributed event logger: the

520

T. Ropars and C. Morin CG Class C

Overhead

1.3 1.25 1.2 1.15 1.1 1.05 1 0.95

LU Class C

Open MPI C.E.L. D.E.L.

64

MG Class C

2.2 2 1.8 1.6 1.4 1.2 1 128

1.3 1.25 1.2 1.15 1.1 1.05 1 0.95

256

64

Number of Processes

128

256

64

Number of Processes

128

256

Number of Processes

Fig. 6. Performance of O2P message logging protocol with a centralized and a distributed event logger CG Class C 258

CG Class C 258

8 6 4 2 0 0

1 2 Gossip Degree (Replication Degree = 2) (a)

4

CG Class C 258

120

10

100

8

Overhead in %


Overhead in %

10

80 60 40 20 0

6 4 2 0

0

1 2 Gossip Degree (Replication Degree = 2) (b)

4

2

4 Replication Degree (Gossip Degree = 1)

6

(c)

Fig. 7. Impact of the distributed event logger parameters on O2P

distributed event logger actually provides better performance and determinants availability. The overhead induced by the two protocols on LU even when using the distributed event logger is not acceptable. For such applications, with very high communication rate, it needs to be combined with techniques to reduce the number of determinants to log, as described in Section 2.2. Distributed Event Logger Parameters. Figure 7 shows the impact of the replication degree and the gossip degree on the behavior of O2P for CG with 256 processes. On Figure 7(a), we see the variation of the overhead according to the gossip degree with a constant replication degree. Figure 7(b) shows the corresponding number of timestamps piggybacked by O2P on application messages. A gossip degree of 1 improves performance compared to an execution without gossiping. However if we increase the gossip degree, the overhead starts increasing again. Indeed, we can see on Figure 7(b) that increasing the gossip degree higher than 1 does not reduce enough the amount of data piggybacked on messages compared to the number of additional messages generated. Likewise when increasing the replication degree (Figure 7(c)), the increased number of messages between the event logger processes impacts the performance of the application. Therefore the distributed event logger parameters should be selected carefully to fit the fault tolerance needs and optimize performance.


5

521

Conclusion

In this paper, we have proposed a distributed event logger to improve message logging protocols scalability. This event logger takes advantage of multi-cores architectures to be executed in parallel with application processes. It uses the volatile memory of the execution nodes to log determinants. Our experiments, run with an optimistic and a pessimistic message logging protocol, show that our distributed event logger outperforms a centralized event logger both in scalability and performance. For message logging protocols like O2P, that can use information on new stable state intervals to optimize their performance, we have proposed to use a simple gossip-based dissemination protocol to distribute information on new stable determinants to all application processes. The experiments show that this solution manage to efficiently distribute the information. However, when the number of messages generated by the gossiping protocol becomes too high, they start interfering with application execution. As a result, the performance decreases. In the future, we plan to improve the gossiping protocol. In the current version, the whole stable vector is sent to other randomly selected nodes every time one entry in the vector is changed. To limit the amount of data exchanged for the needs of the protocol, we could send only the last updated entry in the vector. Furthermore, the event logger processes could use the determinants they receive to analyze the communication patterns in the application, and make use of this data to better select the nodes to send information on new stable determinants to.

Acknowledgments The authors would like to thank Aurélien Bouteiller for his help on developing the distributed event logger. Experiments presented in this paper were carried out using the Grid’5000 experimental testbed, being developed under the INRIA ALADDIN development action with support from CNRS, RENATER and several Universities as well as other funding bodies (see https://www.grid5000.fr).

References [1] Schroeder, B., Gibson, G.A.: A Large-Scale Study of Failures in High-Performance Computing Systems. In: Proceedings of the International Conference on Dependable Systems and Networks (DSN 2006), Washington, DC, USA, pp. 249–258. IEEE Computer Society, Los Alamitos (2006) [2] Forum, M.P.I.: MPI: A Message-Passing Interface Standard, http://www.mpi-forum.org [3] Elnozahy, E.N.M., Alvisi, L., Wang, Y.M., Johnson, D.B.: A Survey of RollbackRecovery Protocols in Message-Passing Systems. ACM Computing Surveys 34(3), 375–408 (2002) [4] Lemarinier, P., Bouteiller, A., Herault, T., Krawezik, G., Cappello, F.: Improved Message logging versus Improved Coordinated Checkpointing for Fault Tolerant MPI. In: IEEE International Conference on Cluster Computing (Cluster 2004). IEEE CS Press, Los Alamitos (2004)

522


[5] Strom, R.E., Yemini, S.: Optimistic Recovery in Distributed Systems. ACM Transactions on Computing Systems 3(3), 204–226 (1985) [6] Johnson, D.B., Zwaenepoel, W.: Sender-Based Message Logging. In: Digest of Papers: The 17th Annual International Symposium on Fault-Tolerant Computing, pp. 14–19 (1987) [7] Alvisi, L., Marzullo, K.: Message Logging: Pessimistic, Optimistic, Causal, and Optimal. IEEE Transactions on Software Engineering 24(2), 149–159 (1998) [8] Bouteiller, A., Collin, B., Herault, T., Lemarinier, P., Cappello, F.: Impact of Event Logger on Causal Message Logging Protocols for Fault Tolerant MPI. In: Proceedings of the 19th IEEE InternationalParallel and Distributed Processing Symposium (IPDPS 2005), April 2005, vol. 1, p. 97. IEEE Computer Society, Los Alamitos (2005) [9] Bouteiller, A., Cappello, F., Herault, T., Krawezik, G., Lemarinier, P., Magniette, F.: MPICH-V2: a Fault Tolerant MPI for Volatile Nodes based on Pessimistic Sender Based Message Logging. In: Proceedings of the 2003 ACM/IEEE conference on Supercomputing (SC 2003), Washington, DC, USA, p. 25. IEEE Computer Society, Los Alamitos (2003) [10] Ropars, T., Morin, C.: Active Optimistic Message Logging for Reliable Execution of MPI Applications. In: 15th International Euro-Par Conference, Delft, The Netherlands, August 2009, pp. 615–626 (2009) [11] Lamport, L.: Time, Clocks, and the Ordering of Events in a Distributed System. Communications of the ACM 21(7), 558–565 (1978) [12] Bouteiller, A., Bosilca, G., Dongarra, J.: Redesigning the Message Logging Model for High Performance. In: International Supercomputer Conference (ISC 2008), Dresden, Germany (June 2008) [13] Bouteiller, A., Ropars, T., Bosilca, G., Morin, C., Dongarra, J.: Reasons for a Pessimistic or Optimistic Message Logging Protocol in MPI Uncoordinated Failure Recovery. In: IEEE International Conference on Cluster Computing (Cluster 2009), New Orleans, USA (2009) [14] Damani, O.P., Garg, V.K.: How to Recover Efficiently and Asynchronously when Optimism Fails. In: International Conference on Distributed Computing systems, pp. 108–115. IEEE Computer Society, Los Alamitos (1996) [15] Bailey, D., Harris, T., Saphir, W., van der Wilngaart, R., Woo, A., Yarrow, M.: The NAS Parallel Benchmarks 2.0. Technical Report Report NAS-95-020, NASA Ames Research Center (1995)

Value-Based Sequential Consistency for Set Objects in Dynamic Distributed Systems Roberto Baldoni1 , Silvia Bonomi1 , and Michel Raynal2 1

2

Universit´ a La Sapienza, Via Ariosto 25, I-00185 Roma, Italy {baldoni,bonomi}@dis.uniroma1.it IRISA, Université de Rennes, Campus de Beaulieu, F-35042 Rennes, France [email protected]

Abstract. This paper introduces a shared object, namely a set object that allows processes to add and remove values as well as take a snapshot of its content. A new consistency condition suited to such an object is introduced. This condition, named value-based sequential consistency, is weaker than linearizability. The paper also addresses the construction of a set object in a synchronous anonymous distributed system where participants can continuously join and leave the system. Interestingly, the protocol is proved correct under the assumption that some constraint on the churn is satisfied. This shows that the notion of “provably correct software” can be applied to dynamic systems. Keywords: Churn, Consistency condition, Dynamic system, Infinite arrival model, Set object, Synchronous system.

1

Introduction

A set S is a shared object that stores a (possibly empty) finite set of values. A process can acquire the content of S through a get operation while it can add (remove) an element to S through an add (remove) operation. A restricted form of set, namely weak set, has been introduced for the first time by DelporteGallet and Fauconnier in [1]. A weak set is a set without the remove operation. Delporte-Gallet and Fauconnier point out that (due to the semantic of the object itself) a weak set object is not linearizable [2]. More precisely, a get operation does not care about the execution order of two concurrent add operations that happened before the get because the important issue for a get operation is the fact that a value is (or is not) in the weak set (and not the order in which values have been inserted into the set). The authors show that a weak set is a useful abstraction to solve consensus in anonymous shared memory systems. Contribution of the paper. The paper presents a set object that extends the notion of weak set proposed in [1]. A set has operations for joining the computation (i.e. join operation), remove a value from the set (i.e. remove operation) as well as get and add operations defined in the weak set. The paper has two main contributions. P. D’Ambra, M. Guarracino, and D. Talia (Eds.): Euro-Par 2010, Part I, LNCS 6271, pp. 523–534, 2010. c Springer-Verlag Berlin Heidelberg 2010

524

R. Baldoni, S. Bonomi, and M. Raynal

It first introduces a consistency condition for a set object. This new condition is named value-based sequential consistency. While it allows concurrent get operations to return the same output in the absence of concurrent add and/or remove operation, this condition is weaker than linearizability [2]. This is because processes are required to see the same order only of concurrent add and remove operations that are on a same value. Concurrent operations executed on distinct values can be perceived in any order by a process. The second contribution is a protocol that implements a set S on the top of a dynamic anonymous synchronous message-passing distributed system. The implementation uses a copy of S at any process. An important part of that contribution is the proof that the implementation is correct when the churn remains below a given threshold. For the churn we use the characterization given in [3] in which the number of processes in the system is always constant (this means at any time the same number of processes join and leave the system). Roadmap. Section 2 presents the set objects, details the set operations and introduces the value-based sequential consistency. The distributed system model and the model of churn is presented in Section 3. A protocol implementing the set objects is introduced in Section 4. Section 5 presents the related work and section 6 concludes the paper.

2

The Set Object

A set object S is a shared object used to store values. Without loss of generality, we assume that (i) S contains only integer values and (ii) at the beginning of the computation S is empty. A set S can be accessed by three operations: add and remove that modify the content of the set and get that returns the current content of the set. More precisely: the add operation, denoted add(v), takes an input parameter v and returns a confirmation that the operation has been executed (i.e. the value OK). It adds v to S. If v is already in the set, the add operation has no effect. The remove operation, denoted remove(v), takes an input parameter v and returns a confirmation that the operation has been executed (i.e. the value OK). If v belongs to S, it suppresses it from S. Otherwise it has no effect. The get operation, denoted get(), takes no input parameter. It returns a set containing the current value of S. It does not modify the content of the object. Generally, each of these operation is not instantaneous and takes time to be executed; we assume that every process executes operations sequentially (i.e., a process does not invoke any operation before it got a response from the previous one). Hence, given two operations executed by two different processes, they may overlap and the current content of the set may be not univocally defined. Consider, for example, a get() operation overlapping with an add(v) operation while v is not present in the set: has v to be contained in the result of the get()? Moreover, if two processes modify concurrently the set by adding and removing the same value v, has v to be returned by a successive get() operation or not? In the following section the notion of concurrency between operations is defined and the behavior of the get operation in case of concurrency is specified.

Value-Based Sequential Consistency for Set Objects

2.1

525

Basic Definitions

Every operation can be characterized by two events occurring at its boundary: an invocation event and a reply event. These events occur at two time instants (invocation time and reply time). According to these time instants, it is possible to state when two operations are concurrent with respect to the real time execution. For ease of presentation we assume the existence of a fictional global clock (not accessible by processes). The invocation time and response time of every operation are defined with respect to that clock. Given two operation op and op having respectively invocation times tB (op) and tB (op ) and return times tE (op) and tE (op ), we say that op precedes op (op ≺ op ) iff tE (op) < tB (op ). If op does not precede op and op does not precede op then they are concurrent (op||op ). By definition, every get() operation, issued on a set object S, should return the current content of S. If operations occur sequentially, such a content is represented by all the values added by an add() operation preceding the get() and for which there does not exist a remove() that precedes get() and follows the add(). Conversely, if there is concurrency between operations, the values added or removed concurrently may belong to the set. In order to formalize such a behavior, we introduce the notion of admissible sets for a get() operation op (denoted Vad (op)). To that end we first define two sets, namely a sequential set (Vseq (op)) and a concurrent set (Vconc (op)) for an op = get() operation. These sets define admissible values, with respect to a get() operation, in case of sequential access and concurrent access to S. Definition 1 (Sequential set for a get() operation). Given an op = get() operation executed on a set object S, the set Vseq (op) of sequential values for op contains all the values v such that: 1. ∃ add(v) : add(v) ≺ op, and 2. (a) remove(v) or (b) ∃remove(v) : (remove(v) ≺ add(v) ≺ op) ∨ (add(v) ≺ op ≺ remove(v)). Informally, given a get() operation, a sequential set will include all the values that have to be returned by such a get (i.e. all the values such that the add terminates before the get starts and such that there not exists any remove between the add and the get). As an example, let consider the execution of Figure 1(a) and let op be the get() operation represented in Figure 1(a), the sequential set Vseq (op) is equal to {1, 2} because there exist two add() operations, adding values 1 and 2 respectively, that terminate before the get() operation is issued and does not exist any remove() operation starting before the get(). Conversely, in the execution of Figure 1(b) Vseq (op) = ∅ because the only value added to the set is subsequently removed before the get() operation op is issued. Definition 2 (Concurrent set for a get() operation). Given an op = get() operation executed on a set object S, the set Vconc (op) of concurrent values for op contains all the values v such that:

526


GET()

Pi ADD(2)

Pj ADD(1)

Pk

(a) Vseq = Vconc = ∅

Pk

ADD(1)

ADD(1)

{1, 2} and (b) Vseq = ∅ and Vconc = {∅}

REMOVE(1)

Pj

REMOVE(1)

Pj Pk

GET()

Pi

GET()

Pi

GET()

Pi Pj Pk

REMOVE(1) ADD(1)

(c) Vseq = ∅ and Vconc = (d) Vseq = ∅ and Vconc = {1} {1} Fig. 1. Vseq and Vconc in distinct executions

1. ∃ add(v) : add(v) || op or 2. ∃ add(v), remove(v) : (add(v) ≺ op) ∧ (remove(v) || op) or 3. ∃ add(v), remove(v) : add(v) || remove(v) ∧ add(v) ≺ op ∧ remove(v) ≺ op. Informally, given a get() operation, a concurrent set will include all the values that may be returned by such a get due to the concurrency of their addition/removal (i.e. all the values for which the add or the remove operation is executed concurrently with the get or by themself). As an example, let consider the execution of Figure 1(c) and let op be the get() operation represented in that figure, the concurrent set Vconc (op) is equal to {1} since item 1 of the definition is satisfied, while, in the execution of Figure 1(d), Vconc (op) = {1} due to item 3. Now, it is possible to define an admissible set of values for a get() operation. Definition 3 (Admissible set for a get() operation). Given an op = get() operation issued on a set object S, its sequential set Vseq (op) and its concurrent set op Vconc (op), its admissible set Vad (op) is such that 1. Vad (op) contains at least the values in op Vseq (op), 2. and, ∀v ∈ Vad (op)/Vseq (op), we have v ∈ Vconc (op). As an example, let consider the four executions depicted in Figure 1. In Figure 1(a) and Figure 1(b), there exists only one admissible set Vad (op) for each of the get() operations op and is respectively Vad (op) = {1, 2} for the execution of Figure 1(a) and Vad (op) = ∅ for the execution of Figure 1(b). Contrarily, in the executions of Figure 1(c) and Figure 1(d) there exist two different admissible sets for each of the get() operations. In particular these admissible sets (equal for both the executions) are Vad (op) = ∅ and Vad (op) = {1}; the first one contains only the element contained in Vseq (op) while the second one contains also the elements of Vconc (op). Note that in the executions depicted in Figure 1(c) and Figure 1(d) if another get() operation is issued after the add() and remove() operations, the get() may return different admissible sets.


2.2

527

Value-Based Sequential Consistency Condition

A consistency condition defines which are the values that a get() operation is allowed to return. In a shared memory context, a set of formal consistency conditions has been defined [4] as constraints on the partial order of read() and write() operations. In order to specify a condition for a set object, we introduce the concepts of execution history and linear extension of an history. Definition 4 (Execution History). Let H be the set of all the operations ˆ = (H, ≺) is a partial order issued on the set object S. An execution history H on H satisfying the relation ≺ (i.e. precedence relation). Definition 5 (Linear extension of an history). A linear extension Sˆ = ˆ is a topological sort of its partial order where (S, →s ) of an execution history H (i) S = H, (ii) op1 ≺ op2 ⇒ op1 →s op2 and (iii) →s is a total order. Let us introduce now the notion of value-based sequential consistency for a set object. Informally, this consistency condition requires that any group of concurrent get() operations that do not overlap with any other operation, return the same set. Moreover, due to the semantics of the set when considering concurrent operations involving different values (e.g. add(v) and add(v )), these operations can be perceived in different order by different processes. More formally we have the following. ˆ = (H, ≺ Definition 6 (Value-based sequential consistency). A history H ) is value-based sequentially consistent iff for each process pi there exists a linear extension Sî = (S, →si ) such that for any pair of concurrent operations op=add(v) and op’=remove(v ), with v = v , if op →si op for some pi then op →sj op for any other process pj . Note that, if the domain of values that can be written in the set is composed of a single value, then value-based sequential consistency is equivalent to sequential consistency1 [5]. In fact, in this case, any pair of concurrent operations occurs on the same value and thus have to be ordered the same way by all the processes. Since the non-concurrent operations are totally ordered, the result is a unique total order on which all the processes agree. Let now consider the following case: each process can add and/or remove only one specific value (e.g., its identifier). Value-based sequential consistency boils down to causal consistency2 [6] . Since each value is associated with a only one process and each process executes operations sequentially, it follows that the concurrent operations are issued on different values and each process can perceive them in a different order, exactly as in causal consistency. 1

2

ˆ = (H, ≺) is sequential consistent if it admits a linear extension in which A history H all the get() operations returns an admissible set. ˆ from which all get() operations not issued by pi have Let Hˆ i be the sub-history of H ˆ = (H, ≺) is causal consistent if, for every pi , all the been removed. An history H get() operations of Hˆ i return an admissible set.

528

3


System Model

The distributed system is composed, at each time, by a bounded number of processes that communicate by exchanging messages. Processes are uniquely identified (with their indexes) and they may join and leave the system at any point in time. The system is synchronous in the following sense: the processing times of local computations are negligible with respect to communication delays, so they are assumed to be equal to 0. Contrarily, messages take a bounded time to travel to their destination processes. Moreover we assume that processes can access a global clock3 . We assume that there exists an underling protocol, that keeps processes connected each other. 3.1

Distributed Computation

A distributed computation is formed, at each instant of time, by a subset of processes of the distributed system. A process p, belonging to the system, that wants to participate to the distributed computation has to execute a join() operation. Such an operation, invoked at some time t, is not instantaneous: it consumes time. But, from time t, the process p can receive and process messages sent by any other process that belongs to the system and that participate to the computation. Processes participating to the distributed computation implements a set object. A process leaves the computation in an implicit way. When it does, it leaves the computation forever and does no longer send messages. From a practical point of view, if a process wants to re-enter the system, it has to enter it as a new process (i.e., with a new name). We assume that a process does not crash during the execution of add() and remove() operations. In order to formalize the set of processes that participate actively to the computation we give the following definition. Definition 7. A process is active from the time it returns from the join() operation until the time it leaves the system. A(t) denotes the set of processes that are active at time t, while A([t1 , t2 ]) denotes the set of processes that are active during the interval [t1 , t2 ]. Churn rate. The dynamicity of the joins and leaves of the processes is captured by the system parameter called churn. We consider here the churn rate, denoted c, defined as the percentage of the nodes that are “refreshed” at every time unit (c ∈ [0, 1]). This means that, while the number of processes remains constant (equal to n), in every time unit c × n processes leave the system and the same number of processes join the system. It is shown in [7] that this assumption is fairly realistic for several classes of applications built on top of dynamic systems. 3

The global clock is for ease of presentation. As we are in a synchronous system, this global clock can be implemented by synchronized local clocks.


3.2

529

Communication Primitives

Two communication primitives are used by processes belonging to the distributed computation to communicate: point-to-point and broadcast communication. Point-to-point communication. This primitive allows a process pi to send a message to another process pj as soon as pi knows that pj has joined the computation. The network is reliable in the sense that it does not loose, create or modify messages. Moreover, the synchrony assumption guarantees that if pi invokes “send m to pj ” at time t, then pj receives that message by time t + δ (if it has not left the system by that time). In that case, the message is said to be “sent” and “received”. Broadcast. Processes participating to the distributed computation are equipped with an appropriate broadcast communication sub-system that provides the processes with two operations, denoted broadcast() and deliver(). The former allows a process to send a message to the processes currently present in the system, while the latter allows a process to deliver a message. Consequently, we say that such a message is “broadcast” and “delivered”. These operations satisfy the following property. – Timely delivery: Let t be the time at which a process p belonging to the distributed computation invokes broadcast(m). There is a constant δ (δ ≥ δ ) (known by the processes) such that if p does not leave the system by time t + δ, then all the processes that are in the system at time t and do not leave by time t + δ, deliver m by time t + δ. Such a pair of broadcast operations has first been formalized in [8] in the context of systems where process can commit crash failures. It has been extended to the context of dynamic systems in [9]. Assuming that the processing times are negligible, the bound δ and δ makes the system synchronous.

4

Set Implementation in a Synchronous Dynamic Distributed System

This section presents a value-based sequentially consistent protocol implementing a set in a dynamic distributed systems. 4.1

Value-Based Sequential Consistent Protocol

The algorithm implementing a value-based sequential consistent set object is shown in Figure 2. Local variables at process pi . Initially, n processes compose the system and each process pi has the following local variables. – Two variables denoted seti and sni ; seti is a variable that contains the local copy of the set; sni is an integer variable that count update operations executed by process pi on the local copy of the set.

530


– A FIFO set variable last opsi used to maintain an history of recent update operations executed by pi . Such variable contains 4-uples < type, val, sn, id > each one characterizing an operation of type type = {A or R} (respectively for add() and remove()) of the value val, with a sequence number sn, issued by a process with identity id. – A boolean activei , initialized to false, that is switched to true just after pi has joined the system. – Three set variables, denoted repliesi , reply toi and pendingi , that are used in the period during which pi joins the system. The local variable repliesi contains the 3-uples < set, sn, ops > that pi has received from other processes during its join period, while reply toi contains the processes that are joining the system concurrently with pi (as far as pi knows). The set pendingi contains the 4-uples < type, val, sn, id > each one characterizes an update operation executed concurrently with the join. The join() operation. The algorithm implementing the join operation involves all the processes that are currently present (be them active or not). First pi initializes its local variables (line 01), and waits for a period of δ time units (line 02) for possible concurrent add or remove operations. After this waiting period, pi broadcasts (with the broadcast() operation) an inquiry(i) message to the processes that are in the system and waits for 2δ time units, i.e., the maximum round trip delay (line 03). When this period terminates, pi first updates its local variables seti , sni and last opsi to the most up-to-date values it has received (lines 04-05) and then executes all the operations concurrent with the join contained in pendingi and not yet executed as if the update message is just received (lines 06-11). Then, pi becomes active (line 12), which means that it can answer the inquiries it has received from other processes, and does it if = ∅ (line 13). Finally, pi returns ok to indicate the end of the join() reply to operation (line 16). When a process pi receives a message inquiry(j), it answers pj by sending back a reply(< seti , sni , last opsi >) message containing its local variables if it is active (line 18). Otherwise, pi postpones its answer until it becomes active (line 19 and line 13). Finally, when pi receives a message reply(< set, sn, ops >) from a process pj it adds the corresponding 3-uple to its set repliesi (line 21). The get() operation. The get is purely local (i.e., fast): it consists in returning the current value of the local variable seti . The add(v) and the remove(v) operations. Both the add() and the remove() operations have the aim to modify the content of the set object by adding and removing respectively an element. Hence, the structure of the protocols implementing the two operations is the same. In order to assure value-based sequential consistency, all the processes that execute the update operations on the set, have to execute such updates in the same order by applying some deterministic rules. In the proposed algorithm, such deterministic rule is given by the total order of the pairs < sn, id > where sn is the sequence number of the operation and id is the identifier of the process issuing the operation.


531

operation join(i): (01) sni ← 0; last opsi ← ∅ seti ← ∅; active i ← false; pending i ← ∅; repliesi ← ∅; reply toi ← ∅; (02) wait(δ); (03) broadcast inquiry(i); wait(2δ); (04) let < set, sn, ls >∈ repliesi such that (∀ < −, sn , − >∈ repliesi : sn ≥ sn ); (05) seti ← set; sni ← sn; last opsi ← ls; (06) for each < type, val, sn, id >∈ pendingi do (07) < typeval, sn, id >← first element(pending); (08) if (< type, val, sn, id >∈ / last opsi ) (09) then execute update(< type, val, sn, id >); (10) end if (11) end for; (12) activei ← true; (13) for each j ∈ reply toi do (14) send reply (< seti , sni , last opsi >) to pj ; (15) end for; (16) return(ok). (17) (18) (19) (20)

when inquiry(j) is delivered: if (activei ) then send reply (< seti , sni , last opsi >) to pj else reply toi ← reply toi ∪ {j} end if.

(21)

when reply(< set, sn, ops >) is received: repliesi ← repliesi ∪ {< set, sn, ops >}. ————————————————————————————————— operation get(): % issued by any process pi % (22) return(seti ). ————————————————————————————————– operation add(v): % issued by any process pi % (23) sni ← sni + 1; (24) broadcast update(A, v, sni , i); (25) seti ← seti ∪ {v}; last opsi ← last opsi ∪ {< A, v, sni , i >}; (26) wait(δ); (27) return(ok). ————————————————————————————————– operation remove(v): % issued by any process pi % (28) sni ← sni + 1; (29) broadcast update(R, v, sni , i); (30) seti ← seti /{v}; last opsi ← last opsi ∪ {< R, v, sni , i >}; (31) wait(δ); (32) return(ok). (33) when update(type, val, snj , j) is delivered: % at any process pi % (34) if(¬activei ) then pendingi ← pendingi ∪ {< type, val, snj , j >} (35) else execute update(type, val, snj , j) (36) endif. ————————————————————————————————– procedure update(type, val, snj , j) % at any process pi % (37) if (snj > sni ) then last opsi ← last opsi ∪ {< type, val, snj , j >}; (38) if (type = A) then seti ← seti ∪ {val}; (39) else seti ← seti /{val}; (40) endif (41) else temp ← {X ∈ last opsi |X =< −, val, −, − >} (42) if (temp = ∅) (43) then last opsi ← last opsi ∪ {< type, val, snj , j >; (44) if (type = A) then seti ← seti ∪ {val}; (45) else seti ← seti /{val}; (46) endif (47) else if ((type = A)∧ ( < R, −, sn, id >∈ temp | (sn, id) > (snj , j))) (48) then seti ← seti ∪ {val}; (49) last opsi ← last opsi ∪ {< type, val, snj , j >}; (50) endif (51) if ((type = R)∧( < A, −, sn, id >∈ temp | (sn, id) > (snj , j))) (52) then seti ← seti /{val}; (53) last opsi ← last opsi ∪ {< type, val, snj , j >}; (54) endif (55) endif (56) endif (57) sni ← max(sni , snj ).

Fig. 2. The protocol for a set object in a synchronous system (code for pi )

532


When pi wants to add/remove an element v to/from the set, it increments its sequence number sni (line 23 and line 28), it broadcasts an update(type, val, sn, id) message (line 24 and line 29) where type is a flag that identify the type of the update (i.e. A for an add() operation or R for a remove() operation), val is the value that has to be added or removed, sn is the sequence number of the operation and id is the identifier of the process that issues the operation. After, it executes the operation on its local copy of the set and it stores locally in its last opsi variable the tuple < type, val, sn, id > that identifies the last operation executed on the set (line 25 and line 30) . Then pi waits for δ time units (line 26 and line 31) to be sure that all the active processes have received the update message and finally it returns by the operation (line 27 and line 32). When pi receives the update(type, val, snj , j) from a process pj , if it is not active, it puts the current update message in its pendingi buffer and will process it as soon as it will be active, otherwise it executes the update() procedure. In the update() procedure pi checks if the sequence number snj , corresponding to the current operation, is greater than the one stored by pi and if it is so, then pi execute the operation (lines 37-40). Contrary, pi checks if in the set of the last executed operation last opsi there is some operation occurred on the same value val; if there is not such an operation, pi executes the current one (lines 41 - 47) otherwise, it checks, according to the type of the operation to be executed, if the two operations are in the right order and in positive case, pi executes the operation (lines 48 - 56). Finally pi updates it sequence number (line 57). Garbage Collection. Let us remark that the last opsi set variable collects the information related to operations executed on the set. In order to make the protocol working correctly, only the information related to recent operations are needed. Moreover, if the rate of operation is high, each process becomes immediately overloaded of information. To avoid this problem it is possible to define a garbage collection procedure that periodically removes from the last opsi variable the information related to “old” operations. The thread managing the garbage collection is very easy; it is always running and each δ time unit, operations stored more that δ time before are removed. Due to lack of space we omit here the pseudocode of the procedure. ˆ = (H, ≺) be an execution history Theorem 1. Let S be a set object and let H ˆ is always of S generated by the algorithm presented in Section 4. If c < 1/3δ, H value-based sequential consistent. Due to lack of space we omit here the proof that can be found in [10]

5

Related Work

Dynamicity Model. Dynamic systems are nowadays an open field of research and new models able to capture all the aspects of such dynamicity are going to be defined. In [11] are presented models, namely infinite arrival models, able to capture the evolution of the network removing the constraint of having a predefined and constant size n. These models do not address the way the processes


533

join or leave the system. More recently, other models have been proposed that take into account the process behavior. This is done by considering both probabilistic distribution [12], or deterministic distribution [7], on the join and leave of nodes (but in both cases the value of the system size is constant). Registers and Weak-Set. Among shared objects, registers are certainly one of the basic one. A register is a shared variable that can be accessed by processes by means of two operations, namely write() and read(), used to store a value in the register and to retrieve the value from the object. According to the set of values that can be returned by a read() operation, Lamport has defined different type of registers [13] as regular or atomic. In [14] an implementation of a regular register in a dynamic distributed system subject to churn is provided while in [15] an atomic register is implemented in a mobile ad-hoc network. In [1], the authors show how it is possible to implement a weak-set in a static system, by using a finite number of atomic registers, in two particular cases: (i) when the number of processes is finite and known and (ii) when the set of possible values that can be added to the set is finite and show that a weak-set is stronger than a regular register. Unfortunately, in the model considered in this paper, it is not possible to implement a set object by using a finite number of registers. The intuition besides such impossibility is that (i) the domain of the set is possibly infinite and (ii) it is not possible to rely on the number of processes as in the solutions proposed in [1] without using an infinite number of registers. Even if, at each time unit there are always n processes in the system, they change along time and possibly infinite processes may participate to the computation.

6

Conclusion

Shared objects provide programmers with a powerful way to design distributed applications on top of complex distributed systems. This paper has introduced a set object suited to dynamic systems. The paper has presented a consistency condition for set objects that is weaker than sequential consistency (by exploiting the semantic of the set object and allowing, at the same time, concurrent readings to return the same set in absence of other operations). The paper also presented a value-based sequentially consistent implementation of the set object in a dynamic, synchronous and anonymous distributed systems. A proof has been given that shows that the proposed protocol is correct when the churn remains below a given threshold. This shows that “provably correct software” can be extended to dynamic systems.

Acknowledgement This work is partially supported by the European project SM4All and SOFIA.

References 1. Delporte-Gallet, C., Fauconnier, H.: Two consensus algorithms with atomic registers and failure detector ω. In: Garg, V., Wattenhofer, R., Kothapalli, K. (eds.) ICDCN 2009. LNCS, vol. 5408, pp. 251–262. Springer, Heidelberg (2008)

534


2. Herlihy, M.P., Wing, J.M.: Linearizability: a correctness condition for concurrent objects. ACM Transactions on Programming Languages and Systems 12(3), 463– 492 (1990) 3. Baldoni, R., Bonomi, S., Raynal, M.: Regular Register: an Implementation in a ˇ Churn Prone Environment. In: Kutten, S., Zerovnik, J. (eds.) SIROCCO 2009. LNCS, vol. 5869, pp. 15–29. Springer, Heidelberg (2010) 4. Schiper, A., Raynal, M., Raynal, M.: A suite of formal definitions for consistency criteria in distributed shared memories. In: ISCA Proceedings of the International Conference PDCS, Dijon, France, pp. 125–130 (1996) 5. Lamport, L.: How to make a multiprocessor computer that correctly executes multiprocess programs. IEEE Transaction on Computers (TC) 28(9), 690–691 (1979) 6. Ahamad, M., Neiger, G., Burns, J.E., Kohli, P., Hutto, P.W.: Causal memory: Definitions, implementation, and programming. Distributed Computing 9(1), 37– 49 (1995) 7. Ko, S.Y., Hoque, I., Gupta, I.: Using tractable and realistic churn models to analyze quiescence behavior of distributed protocols. In: SRDS 2008: Proceedings of the 2008 Symposium on Reliable Distributed Systems, Washington, DC, USA, pp. 259–268. IEEE Computer Society, Los Alamitos (2008) 8. Hadzilacos, V., Toueg, S.: Reliable broadcast and related problems. Distributed Systems, 97–145 (1993) 9. Friedman, R., Raynal, M., Travers, C.: Two abstractions for implementing atomic objects in dynamic systems. In: Anderson, J.H., Prencipe, G., Wattenhofer, R. (eds.) OPODIS 2005. LNCS, vol. 3974, pp. 73–87. Springer, Heidelberg (2006) 10. Baldoni, R., Bonomi, S., Raynal, M.: Value-based Sequential Consistency for Set Objects in Dynamic Distributed Systems. Technical report, MIDLAB 3/2010 (2010) 11. Aguilera, M.K.: A pleasant stroll through the land of infinitely many creatures. Distributed Computing 35(2), 36–59 (2004) 12. Leonard, D., Rai, V., Loguinov, D.: On lifetime-based node failure and stochastic resilience of decentralized peer-to-peer networks. SIGMETRICS Perform. Eval. Rev. 33(1), 26–37 (2005) 13. Lamport, L.: On interprocess communication, part 1: Models. Distributed Computing 1(2), 77–101 (1994) 14. Baldoni, R., Bonomi, S., Kermarrec, A.M., Raynal, M.: Implementing a Register in a Dynamic Distributed System. In: 29th International Conference on Distributed Computing Systems, ICDCS 2009 (2009) 15. Dolev, S., Gilbert, S., Lynch, N.A., Shvartsman, A., Welch, J.: Geoquorum: Implementing atomic memory in ad hoc networks. In: Fich, F.E. (ed.) DISC 2003. LNCS, vol. 2848, pp. 306–320. Springer, Heidelberg (2003)

Robust Self-stabilizing Construction of Bounded Size Weight-Based Clusters Colette Johnen1 and Fouzi Mekhaldi2 1

Université Bordeaux, LaBRI UMR 5800, F-33405 Talence, France 2 Université Paris-Sud, LRI UMR 8623, F-91405 Orsay, France

Abstract. We propose the first robust self-stabilizing protocol building 1-hop clusters whose size is bounded, moreover the clusterhead selection is weight-based. The protocol reaches quickly (in 4 rounds) a safe configuration, where the safety property is satistfied: network nodes are partitionned into bounded clusters (clusterheads are not the most suitable nodes). During the convergence to a legitimate configuration, where more desired properties are guaranteed, the safety property is preserved, ensuring then the continuity functioning of hierarchical protocols. Keywords: Clustering, Self-stabilization, safety property, robustness.

1

Introduction

Clustering: An Ad Hoc or sensor network consists of wireless hosts that communicate without any pre-installed infrastructure. The clustering is introduced in such networks to facilitate the network management and increase the scalability. Clustering is a hierarchical organization which consists of partitioning network nodes into groups called clusters. Each cluster has a single clusterhead, and eventually a set of ordinary nodes. Clustering is attractive hence it allows the use of hierarchical routing which reduces the amount of stored routing informations; and decreases the transmission overhead. However, for more efficiency, the hierarchical structure should be established as soon as possible, and must be maintained over time. A well maintenance ensures the continuity functioning of protocols using the hierarchical organization like hierarchical routing protocols. Bounded clusters: If a certain zone becomes densely populated with nodes, the clusterhead might not be able to handle all the traffic generated by nodes of its cluster. In addition, the power consumption of a clusterhead is proportional to the number of nodes in its cluster; thus the lifetime of a clusterhead is inversely proportional to its cluster’s size. Therefore, controlling the number of nodes in a cluster will extend its clusterhead’s lifetime, and will improve the stability of the cluster. Furthermore, keeping the number of nodes in a cluster smaller than a threshold facilitates the operation of the medium access control (MAC) protocol. Self-Stabilization: A technique for designing solutions that tolerate transient faults is the self-stabilization. A self-stabilizing system guarantees that regardless of the current system’s configuration, it will converge to a legitimate configuration in a finite number of steps. This convergence implies three important P. D’Ambra, M. Guarracino, and D. Talia (Eds.): Euro-Par 2010, Part I, LNCS 6271, pp. 535–546, 2010. c Springer-Verlag Berlin Heidelberg 2010

536

C. Johnen and F. Mekhaldi

properties of self-stabilization: embedded tolerance of arbitrary transient failures, unneeded of correct initialization (as the initial state does not have to be legitimate) and obvious adaptivity to dynamic changes of the network topology: ex. mobility of nodes, or node crashes (as any of them can be treated as a transient fault). This is why the self-stabilization property is attractive. Robustness: In the other hand, there are some disadvantages of self-stabilizing protocols. They are in particular: tolerance of only non-permanent faults, and no explicit detection of accomplishing convergence. Moreover, self-stabilizing systems do not guarantee any property during the convergence period, so the system behaves arbitrarily along the convergence to a legitimate configuration. In addition, the convergence time may be proportional to the size of the network; particularly, in weight-based clustering protocols (see Sect. 5.2). Thus, self-stabilizing weight-based clustering protocols are not scalable. In order to overcome these drawbacks, we are interested to the robust self-stabilization. The robust selfstabilization guarantees that from an illegitimate configuration and without occurrence of faults, the system reaches quickly a safe configuration, in which the safety property is satisfied. During the convergence to a legitimate configuration, the safety property is preserved. The safety property has to be defined in such a way that the system still performs correctly its task once a safe configuration is reached. The safety property for our protocol is defined in Section 4.3. The problem studied is the clustering of a network such that the number of nodes per cluster is bounded by a pre-defined threshold. This structure is also known as capacitated dominating set (dominating set with node capacities). Related works: We are interested to protocols building 1-hop clusters (1-dominating sets) in which ordinary nodes are neighbor of their clusterhead. [1] presents a self-stabilizing protocol that constructs a minimal dominating set under synchronous scheduler. In [2,3], self-stabilizing protocols building a connected dominating set are provided. In [4], a self-stabilizing protocol to construct a maximal independent set (MIS) is presented. In [5], a probabilistic self-stabilizing MIS protocol is presented. A self-stabilizing protocol building a minimal k-dominating set under synchronous scheduler is presented in [6]. A self-stabilizing clustering protocol is presented in [7]; the density criteria (defined in [8]) is used to select the clusterheads. In [9], a robust self-stabilizing protocol building a minimum connected dominating set is proposed. In a safe configuration, the built set is just dominating set. In [10], a robust self-stabilizing version of DMAC [11] under synchronous scheduler is presented. A robust self-stabilizing weight-based clustering protocol is proposed in [12]. It is a robust self-stabilizing version of GDMAC [13] (an extended version of DMAC). In robust self-stabilizing protocols [10,12], a configuration is safe if each node belongs to a cluster. To our knowledge, the only protocols building bounded size clusters are [14,15,16]. In [15], the obtained clusters have a size bounded by a lower and an upper bound. This solution cannot be applied to one-hop clusters, because the degree of nodes may be less than the lower bound. [14,15] are not self-stabilizing. Although [16] is self-stabilizing, it is not robust. During the convergence period, a node may not belong to a cluster even if it belongs initially to a well-formed cluster.

Robust Self-stabilizing Construction of Bounded Size Weight-Based Clusters

537

Contributions: We propose the first robust self-stabilizing protocol building 1-hop clusters having bounded size, where the clusterhead selection is weightbased. Our protocol ensures the fault-tolerance, load-balancing, best choice of clusterheads, and reliability. The load balancing is achieved by building bounded size clusters. The number of nodes that a clusterhead handles is bounded by a threshold (SizeBound). Thus, none of clusterheads are overloaded. The faulttolerance and the reliability are achieved by the robust self-stabilization property. Our protocol reaches a safe configuration, in at most 4 rounds, where a minimum service is guaranteed: the network is partitioned into bounded size clusters having an effectual leader (it is not the most suitable node within the cluster). From that, the protocol converges to a legitimate configuration in at most 7∗|V | 2 +5 rounds (|V | is the size of network). Any self-stabilizing protocol building weight-based clusters needs O(V ) rounds (see subsection 5.2) from some configurations and some networks. During the convergence to a legitimate configuration, the safety property stays verified. Thus, the minimal service is continuously provided. In a legitimate configuration, a best quality of service is achieved: clusterheads are the most suitable nodes in their cluster, and their number is locally minimized.

2

Model and Concepts

A distributed system S is modelled by an undirected graph G = (V, E) in which, V is the set of (mobile) nodes and E is the set of edges. There is an edge {u, v} ∈ E, if and only if u and v can communicate between them (links are bidirectional). In this case, we say that u and v are neighbors, and we note by Nv the set of neighbors of the node v. Every node v in the network is assigned a unique identifier id. The state of a node is defined by the value of its local variables. A configuration of the system S is an instance of the node states. The program of each node is a set of rules. Each rule has the following form: Rulei : Guardi −→ Actioni . The guard of a rule of a node v is a Boolean expression involving the local variables of v, and those of its neighbors. The action of a rule of v updates one or more variables of v. A rule can be executed only if it is enabled, i.e., its guard evaluates to true. A node is said to be enabled if at least one of its rules is enabled. In a terminal configuration, no node is enabled. A computation step ci → ci+1 consists of one or more enabled nodes executing a rule. A computation is a sequence of consecutive computation steps. We say that a computation e is maximal if it is infinite, or if it reaches a terminal configuration. A computation is fair, if for any node v which is continuously enabled along this computation, eventually performs an action. In this paper, we study only fair computations. We note by C the set of all configurations, and by E the set of all (fair) computations.The set of (fair) computations starting from a particular configuration c ∈ C is denoted Ec . EA is the set of computations where the initial configuration belongs to A ⊂ C. A node v is neutralized in the computation step e, ci → ci+1 , if v is enabled in ci and not enabled in ci+1 , but v did not execute any action during e.

538


We use the round notion to measure the time complexity. The first round of a computation e = c1 , ..., cj , ... is the minimal prefix e = c1 , ..., cj such that every node v enabled in c1 , either executes a rule or becomes neutralized in cj . Let e be a suffix of e such that e = e e . The second round of e is the first round of e , and so on. We use the attractor notion to define the self-stabilization. Definition 1 (Attractor). Let B1 and B2 be subsets of configurations of C. B2 is an attractor from B1 , if and only if the following conditions hold: • Convergence: ∀e ∈ EB1 (e = c1 , c2 , ...), ∃i 1 : ci ∈ B2 . ∀c ∈ B1 , If (Ec = ∅) then c ∈ B2 . • Closure: ∀e ∈ EB2 (e = c1 , ...), ∀i 1 : ci ∈ B2 . Definition 2 (Self-stabilization). A distributed system S is self-stabilizing if and only if there exists a non-empty set L ⊆ C, called set of legitimate configurations, such that the following conditions hold: • L is an attractor from C. • All configurations of L satisfy the specification problem. Definition 3 (Robustness under Input Change [12]). Let SP be the safety predicate, that stipulates safe configurations. Let IC be a set of input changes that can occur in the system. A self-stabilizing system is robust under any input changes of IC if and only if the set of configurations satisfying SP is: • closed under any computation step. • closed under any input changes of IC.

3

Weight-Based Bounded Size Clustering

The problem studied consists to build 1-hop clusters having bounded size. Both problems of finding the minimum number of 1-hop clusters (i.e., a minimal dominating set), and the minimum number of bounded clusters (i.e., a minimal capacitated dominating set) are NP-hard [17,18]. The goal of our protocol is not to give a distributed solution to these problems, but to propose a robust self-stabilizing protocol building an useful clustering. Our solution satisfies some desired properties like: a best choice of clusterheads, number of nodes per cluster is bounded, and the number of clusterheads neighbor is minimal. Specifications: The final clusters provided by our protocol satisfy the wellbalanced clustering properties, informally defined as follows: • Affiliation condition: each ordinary node affiliates with a neighbor clusterhead, such that the weight of the clusterhead is greater than its weight. • Size condition: each cluster contains at most SizeBound ordinary nodes. • Clusterheads neighbor condition: if a clusterhead v has a neighbor clusterhead u such that wu > wv , then the size of u’s cluster is SizeBound (v cannot join u’s cluster without violating the size condition).


539

As clusterheads have more tasks to ensure than ordinary nodes, then each clusterhead must be more suitable than ordinary nodes inside its cluster. This is the goal of the Affiliation condition. Our protocol selects clusterheads according to their weight value. Each node has an input variable, its weight, named w, representing its suitability to be clusterhead. The higher the weight of a node, the more suitable this node is for the role of clusterhead. A significant node’s weight can be obtained by a sum of different normalised parameters like: node mobility, memory and processing capacity, bandwidth, battery power, and so on. The computation of the weight value is out the scope of this paper. Nevertheless, we consider that the weight value of a node can increase or decrease, reflecting changes in the node’s status. We assume that nodes weight are different (the id of nodes breaks the tie). The proposed protocol provides bounded size clusters; at most SizeBound ordinary nodes are in a cluster. This limitation on the number of nodes that a clusterhead handles, ensures the load balancing: no clusterhead is overloaded. As clusters have bounded size, several clusterheads may be neighbors. To limit locally the number of clusterheads, the clusterheads neighbor condition is used. A node v stays clusterhead only if it cannot join any neighbor cluster: all neighbor clusters are full (it contains SizeBound members), or v’s weight is bigger than all neighbor clusterhead’s weight. Notice that clusterhead neighbor condition ensures that the clusterhead set S, is minimal : there is not clustering structure satisfying the affiliation and size conditions where the clusterheads set, S is a proper subset of S. Let we note that, a trivial configuration in which each node of the network is a clusterhead does not satisfy the specification of the problem: it does not satisfy the clusterheads neighbor condition. Because, some clusterheads can become ordinary without violating the affiliation and size conditions.

4 4.1

Robust Self-stabilizing Protocol for Bounded Size Weight-Based Clusters Variables and Macros

The variables, and macros are presented in Protocol 1. Each node v has three possible status. It can be a clusterhead (HSv = CH ), a nearly ordinary node (HSv = N O), or an ordinary node (HSv = O). A node v which is clusterhead or nearly ordinary, is the leader of its cluster, and it is responsible to manage it. To prevent the violation of the size condition, a node u cannot freely join a cluster: u needs the permission of its potential new clusterhead. More precisely, only nodes belonging to the set CDv may join v’s cluster. The set Nv+ indicates the clusterhead neighbors of v that are more suitable than v’s current clusterhead. If the set Nv+ is not empty, then v must change its clusterhead in order to choose a more suitable one: in v’s neighborhood, there is a clusterhead u having a bigger weight than v and v’s clusterhead, and u accepts the v’s affiliation (i.e., v ∈ CDu ).

540


Protocol 1: Variables and macros on node v. Constants wv ∈ R; Weight of node v. SizeBound ∈ N; Maximum number of ordinary nodes that may belong to a cluster. Local variables HSv ∈ {CH, O, N O} ; Hierarchical status of node v. Headv ∈ {IDs} ; Identity of v’s clusterhead. whv ∈ R; Weight of v’s clusterhead. CDv ⊆ {IDs}; List of nodes that can join v’s cluster. Sv ∈ N; Locale value about the size of v’s cluster. Macros Size of v’s cluster: Sizev := | {z ∈ Nv : Headz = v} |; v’s neighbors could be clusterheads of v: Nv+ := {z ∈ Nv : (v ∈ CDz ) ∧ (HSz = CH) ∧ (wz > wHeadv ) ∧ (wz > wv )}; Computation of CD2v : Begin CD0v := {z ∈ Nv : whz < wv ∧ wz < wv }; If |CD0v | SizeBound − Sizev then CD1v := CD0v ; Else CD1v contains the SizeBound − Sizev smallest members of CD0v ; If CDv ⊆ CD1v ∪ {z ∈ Nv : Headz = v} then CD2v := CD1v ; Else CD2v := ∅; End

4.2

Predicates and Rules

The predicates and rules are illustrated in Protocol 2. The election rule allows a node to become clusterhead. The affiliation rule allows an ordinary or a nearly ordinary node to affiliate with an existing clusterhead. The resignation rule allows a clusterhead to become nearly ordinary node. The correction rules update if necessary the value of v’s local variables: Headv , CDv , Sv , and whv without changing its hierarchical status. Election and Affiliation processes: When an ordinary node v does not satisfy the affiliation or size conditions, the predicate Change is satisfied. In this case, v has to change its cluster (it will affiliate with another clusterhead or it will become clusterhead). The rule executed by v depends on Nv+ value. If Nv+ = ∅, then no node can be clusterhead of v. So, v must become clusterhead (Election = ∅), v has a neighbor that could be its new clusterhead. rule). Otherwise (Nv+ So, v affiliates with the best clusterhead of Nv+ (Af f iliation rule). Resignation process: A clusterhead v has to resign its status when it does not satisfy the clusterheads neighbor condition, i.e. Nv+ = ∅. In this case, v executes the Resignation rule. In order to maintain the hierarchical structure over the network, the clusterhead v does not take directly the ordinary status: v takes the nearly ordinary status, and still performs correctly its task of clusterhead. Nevertheless, HSv = N O and CDv = ∅, i.e., no node can join v’s cluster. All members of v’s cluster verify the predicate Change. So, they will quit the v’s cluster. Thus, the v’s cluster will eventually be empty (Sizev = 0). Then,


541

either v affiliates with an existing clusterhead (Af f iliation rule) if Nv+ = ∅, or it becomes again a clusterhead (Election rule). This mechanism guarantees that during the construction/maintenance of clusters, no clusterhead abandons its leadership. Thus, the hierarchical structure of the network is continuously available even during its reorganization. Protocol 2: Robust Self-Stabilizing Clustering Protocol on node v. Predicates /* true if a node has to change its clusterhead */ / Nv ∪ {v})∨ Change(v) ≡ (Headv ∈ (wv > wHeadv ) ∨ (HSHeadv = CH) ∨ (SHeadv > SizeBound) /* The guard of Election rule */ Election-g(v) ≡ [(HSv = O) ∧ (Nv+ = ∅) ∧ Change(v)] ∨ [(HSv = N O) ∧ (Nv+ = ∅)] /* The guard of Affiliation rule */ = ∅)]∨[(HSv = N O)∧(Sizev = 0)∧(Nv+ = ∅)] Affiliation-g(v) ≡ [(HSv = O)∧(Nv+ /* The guard of resignation rule */ Resignation-g(v) ≡ (HSv = CH) ∧ (Nv+ = ∅) /* The guards of Correction rules */ Cor-guardCH(v) ≡ (HSv = CH)∧ [(Headv = v) ∨ (CDv = CD2v ) ∨ (Sv = Sizev ) ∨ (whv = wv )] Cor-guardNO(v) ≡ (HSv = N O) ∧ [(Headv = v) ∨ (CDv = ∅) ∨ (Sv = 0) ∨ (whv = wv )] Cor-guardO(v) ≡ (HSv = O) ∧ [(CDv = ∅) ∨ (Sv = 0) ∨ (whv = wHeadv )] Rules /* Clustering Construction rules */ Election : Election-g(v) −→ HSv := CH; Headv := v; CDv := CD2v ; Sv := Sizev ; whv := wv ; Affiliation : Affiliation-g(v) −→ HSv := O; Headv := maxwz z ∈ Nv+ ; CDv := ∅; Sv := 0; whv := wHeadv ; Resignation : Resignation-g(v) −→ HSv := N O; Headv := v; CDv := ∅; Sv := 0; whv := wv ; /* Correction rules */ Correction-CH : ¬Resignation-g(v) ∧ Cor-guardCH(v) −→ Headv := v; CDv := CD2v ; Sv := Sizev ; whv := wv ; Correction-NO : ¬Election-g(v) ∧ ¬Affiliation-g(v) ∧ Cor-guardNO(v) −→ Headv := v; CDv := ∅; Sv := 0; whv := wv ; Correction-O : ¬Election-g(v) ∧ ¬Affiliation-g(v) ∧ Cor-guardO(v) −→ CDv := ∅; Sv := 0; whv := wHeadv ;

4.3

Safety Predicate

A safe configuration is a configuration satisfying the safety predicate SP. The safety predicate ensures that the following properties are satisfied: • each node belongs to one cluster having an effectual leader (no condition on leader’s weight, but its status is not ordinary node);

542


• each cluster has less than SizeBound ordinary members. Definition 4. The safety predicate SP is defined as follow: • SP ≡ ∀v, SP v = T rue • SP v ≡ (Headv ∈ Nv ∪ {v}) ∧ (HSHeadv = O) ∧ Ps (v) • Ps (v) ≡ |Clusterv ∪ CDv | SizeBound • Clusterv = {z ∈ Nv : Headz = v}. Clusterv is the v’s cluster (i.e., the set of nodes having chosen v as their clusterhead). Our protocol is robust under the following input changes: (i.e., the safety predicate SP is preserved) (i) the change of node’s weight, (ii) the crash of ordinary nodes, (iii) the failure of a link between (1) a clusterhead and a nearly ordinary node, (2) two clusterheads, (3) two nearly ordinary nodes, or (4) two ordinary nodes, (iv) the joining of a sub-network that verifies the predicate SP. The difficulty is to preserve the size condition after any computation step. A cluster whose clusterhead v satisfies the predicate Ps (v), verifies the size condition in the current configuration and after any computation step. On the contrary, a cluster whose clusterhead v does not satisfy the predicate Ps (v), may not verify the size condition after the specific computation step in which all nodes of CDv join v’s cluster (this feature is illustrated in Figure 1). In the initial configuration, Cluster6 = {1}, and CD6 = {2, 3, 4}. Thus, the size condition is satisfied, but the Ps (6) predicate is not satisfied: |CD6 ∪ Cluster6 | = | {1, 2, 3, 4} | > 3. After the computation step where all nodes of CD6 join 6’s cluster, the size condition is no more satisfied. Nearly ordinary node

Clusterhead

Ordinary node

Wid = id

Head1 = 6 CD6 = {2, 3, 4}

1

Head6 = 6

6

2

SizeBound = 3

Head1 = 6

Affiliation(2), Affiliation(3), Affiliation(4)

CD6 = {2, 3, 4} Head6 = 6

1

6

2 Head2 = 6

Head2 = 2

5 Head5 = 5

4

3 Head3 = 3

Head4 = 4

5 Head5 = 5

4

Head4 = 6

3 Head3 = 6

Initial configuration

Fig. 1. Violation of the size condition from a configuration not satisfying Ps (v)

The variable CDv is computed in such a way that the predicate Ps (v) stays verified after any computation step. For each clusterhead v, the macro CD2v is used to compute CDv value. CD2v is computed in 3 steps. CD0v is the set of v’s neighbors that want to enter into v’s cluster, i.e., their weight and their clusterhead’s weight are smaller than v’s weight. The size of CD0v can be greater than SizeBound − Sizev: CD1v is a subset of CD0v , containing at most SizeBound−Sizev elements. The set CD2v is a subset of CD1v ensuring that the predicate Ps (v) stays verified by v after any computation step from the current configuration (assuming that Ps (v) is verified in the current configuration).

Robust Self-stabilizing Construction of Bounded Size Weight-Based Clusters Nearly ordinary node

Clusterhead

Ordinary node

Wid = id

Head1 = 6 CD6 = {4, 5}

1

Head6 = 6

6

2 3

5

CD6 = {}

Affiliation(4), Affiliation(5), Correction-CH(6)

Head5 = 5

1

Head6 = 6

6

2 Head2 = 2

4 Head4 = 6

3 5

Head3 = 3

Initial configuration : c1 .

SizeBound = 3 Head1 = 6

Head2 = 2

4 Head4 = 4

543

Head3 = 3

Head5 = 6

Configuration c2 .

Fig. 2. Illustration of CD value computation

The Figure 2 illustrates the computation of CDv value. In the initial configuration, there are 5 clusters satisfying the size condition, and CD6 = {4, 5}. For simplicity, the weight of a node is its identity. Thus, the clusterhead 6 has the highest weight in its neighborhood. Nodes 2, 3, 4 and 5 want to belong to Cluster6 (the node 1 is already in Cluster6 ); so, CD06 = N6 − Cluster6 = {2, 3, 4, 5}. CD16 contains only two nodes, because SizeBound = 3 and |Cluster6 | = 1; so, CD16 = {2, 3}. In the reached configuration c2 , CD6 = ∅ because CD6 (c1 ) {CD16 (c1 ) ∪ Cluster6 (c1 )}. Notice that in c2 , Ps (6) is still verified: |CD6 (c2 ) ∪ Cluster6 (c2 )| SizeBound.

5

Convergence

Proofs of convergence to a safe and legitimate configurations, and robustness are omitted due to lack of space, they can be found in [19]. A legitimate configuration is a terminal configuration in which the well balanced clustering properties are verified. The convergence process from an arbitrary configuration to a legitimate configuration is done in two steps. First, the system converges quickly to a safe configuration; and after that, it progresses to reach a legitimate configuration. 5.1

Illustration of the Convergence Process

The convergence process from an unsafe to a safe configuration, is straightforward: once an ordinary node locally detects an unsafe situation (ex: its cluster has more than SizeBound members), it becomes clusterhead if it cannot join a neighbor cluster without violating the affiliation and size conditions. Along any computation, the time to reach a safe configuration is at most 4 rounds. The convergence to a legitimate configuration is explained informally in follows (technical proofs are in [19]). Figure 3 illustrates the convergence process from an arbitrary configuration to a legitimate configuration. For simplicity in this example, the weight of a node is its identifier. The initial configuration 3.a is not safe, because the cluster of 5 does not have a leader. Node 5 eventually performs the Affiliation rule to affiliate with the clusterhead 7. The reached configuration, 3.b, is safe.

544

C. Johnen and F. Mekhaldi Nearly ordinary node

Clusterhead 6 9

7

Affiliation(5)

6

5

9

7

Ordinary node Election(9)

6

5

9

Wid = id

SizeBound = 3

Resignation(6) Resignation(7)

5

7

CD9 = {6, 7}

Configuration (a)

Configuration (b)

Configuration (c)

6

Affiliation(6) Election(5) 6 9

Correction-CH(9) 5

7

CD9 = ∅

CD5 = ∅

Configuration (g)

6 9

Affiliation(7) 7

CD9 = {6, 7}

5 CD5 = ∅

Configuration (f)

7

5

Configuration (d)

6 9

9

7

5 CD5 = ∅

Configuration (e)

Fig. 3. Illustration of convergence to a legitimate configuration

From a safe configuration, the stabilization is done in phases. At the end of ith phase, nodes of Seti have their final state (have stabilized). Notation 1. We denote the set of safe configurations by A3 , and we denote by Vi = V − Seti the set of nodes not having their final state after the ith phase. • Set0 is the set of nodes having initially their terminal state (they will not do any action). Usually, Set0 = ∅. • We name vh0 , the node having the highest weight in V0 = V − Set0 . In Figure 3, vh0 = 9. The node 9 has to be clusterhead (it verifies Election-g). Once node 9 has performed the Election rule, the system reaches a configuration of L1 = A3 ∩ {c ∈ C | HSvh0 = CH} where vh0 will never change its status. In Figure 3, the configuration 3.c belongs to L1 . We prove, in [19], that L1 is an attractor, and it is reached from A3 in at most one round along any computation. • Let L1 = L1 ∩ {c ∈ C| |Clustervh0 | = M in(SizeBound, |Nvh0 ∩ V0 |)} be the set of configurations where the vh0 ’s cluster is stable (no node will quit or join this cluster). In Figure 3, the configuration 3.f belongs to L1 . We prove that L1 is an attractor, and along any computation, it is reached from L1 in at most five rounds. • In the last step of phase 1, all nodes of vh0 ’s cluster reach their final state (a configuration of L1 ). In Figure 3, the configuration 3.g belongs to L1 , where L1 = L1 ∩ {c ∈ C| ∀ v ∈ Set0 ∪ {vh0 } ∪ {Clustervh0 } , CDv = ∅ }; We prove that L1 is an attractor, and it is reached from L1 in at most one round. At the end of first phase, the set of nodes having their final state is Set1 = Set0 ∪ {vh0 } ∪ {Clustervh0 }. Each phase i is similar to the first one: the node of Vi = V − Seti having the highest weight, named vhi , becomes clusterhead. vhi ’s cluster is filled out, and the members of vhi ’s cluster get their final state. At the end of ith phase, the set of nodes having their final state is Seti = Seti−1 ∪ {vhi } ∪ {Clustervhi }. Each phase building a cluster with at least one


545

ordinary node requires at most 7 rounds, whereas the construction of clusters having just the clusterhead requires 1 round. The number of phases is equal or less than number of clusters. Thus, the convergence time to a legitimate | configuration is at most 7∗|V rounds from a safe configuration. 2 5.2

Upper Bound of Stabilization Time

Theorem 1. The convergence time of a self-stabilizing weight-based clustering protocol is intrinsically proportional to the network size. Proof. Let us study the example network presented in Figure 4. In the initial configuration 4.a, there are (|V | − 1)/2 clusters where |V | is the network size. For any value of i that is old, Xi is a clusterhead, Xi+1 affiliates with Xi , and X0 is member of X1 ’s cluster. The node weights are ordered as follows: Xi > Xi+1 . The legitimate configuration (assuming that SizeBound 1) is defined as: for any value of i that is even, Xi is a clusterhead, and Xi+1 affiliates with Xi . To reach a legitimate configuration, each node has to change its role. Xi+1 detects that it has to change its role only after a change on Xi ’s role. Clearly, Xi can change its role only after i rounds. Therefore, the convergence time is O(V ) rounds. Ordinary node

Clusterhead X0

X1

X2

X3

(a) Initial configuration

X4

X0

X1

X2

X3

X4

(b) Legitimate configuration

Fig. 4. Convergence time

6

Conclusion

Compared to the self-stabilizing protocol presented in [16], our solution is wellsuited for large-scale modern distributed systems such as mobile ad hoc and sensor networks. Our protocol unlike [16], is scalable and during the reorganization of clusters, it ensures the maintain of hierarchical organization. The first benefit is due to the constant time (4 rounds at most) required to reach a configuration ensuring a minimum useful service. The second one is due to the robustness property which ensures that this minimum useful service still provided during the convergence to a legitimate configuration. The cost of robustness property is the require of more time to ensure the stabilization (i.e., the convergence to a legitimate configuration). In fact, the upper bound on stabilization time for [16] is |V | rounds, whereas for our protocol | + 5 rounds. is 7∗|V 2 Our protocol is designed for the state model. Nevertheless, it can be easily transformed into a protocol for the message-passing model. Each node v broadcasts periodically a message containing its state. Based on this message, v’s neighbors decide whether to update their states or not.

546


References 1. Xu, Z., Hedetniemi, S.T., Goddard, W., Srimani, P.K.: A synchronous selfstabilizing minimal domination protocol in an arbitrary network graph. In: Das, S.R., Das, S.K. (eds.) IWDC 2003. LNCS, vol. 2918, pp. 26–32. Springer, Heidelberg (2003) 2. Drabkin, V., Friedman, R., Gradinariu, M.: Self-stabilizing wireless connected overlays. In: Shvartsman, M.M.A.A. (ed.) OPODIS 2006. LNCS, vol. 4305, pp. 425–439. Springer, Heidelberg (2006) 3. Jain, A., Gupta, A.: A distributed self-stabilizing algorithm for finding a connected dominating set in a graph. In: PDCAT 2005, pp. 615–619 (2005) 4. Goddard, W., Hedetniemi, S.T., Jacobs, D.P., Srimani, P.: Self-stabilizing protocols for maximal matching and maximal independent sets for ad hoc networks. In: IPDPS 2003, vol. 162(2) (2003) 5. Dolev, S., Tzachar, N.: Empire of colonies: Self-stabilizing and self-organizing distributed algorithm. Theoretical Computer Science 410, 514–532 (2009) 6. Datta, A., Devismes, S., Larmore, L.: A self-stabilizing o(n)-round k-clustering algorithm. In: SRDS 2009 (2009) 7. Mitton, N., Fleury, E., Guérin-Lassous, I., Tixeuil, S.: Self-stabilization in selforganized multihop wireless networks. In: WWAN 2005, pp. 909–915 (2005) 8. Mitton, N., Busson, A., Fleury, E.: Self-organization in large scale ad hoc networks. In: MED-HOC-NET 2004 (2004) 9. Kamei, S., Kakugawa, H.: A self-stabilizing approximation for the minimum connected dominating set with safe convergence. In: Baker, T.P., Bui, A., Tixeuil, S. (eds.) OPODIS 2008. LNCS, vol. 5401, pp. 496–511. Springer, Heidelberg (2008) 10. Kakugawa, H., Masuzawa, T.: A self-stabilizing minimal dominating set algorithm with safe convergence. In: APDCM 2006 (2006) 11. Basagni, S.: Distributed clustering for ad hoc networks. In: ISPAN 1999, pp. 310– 315 (1999) 12. Johnen, C., Nguyen, L.H.: Robust self-stabilizing weight-based clustering algorithm. Theoretical Computer Science 410(6-7), 581–594 (2009) 13. Basagni, S.: Distributed and mobility-adaptive clustering for multimedia support in multi-hop wireless networks. In: VTC 1999, pp. 889–893 (1999) 14. Chatterjee, M., Das, S.K., Turgut, D.: WCA: A weighted clustering algorithm for mobile ad hoc networks. Journal of Cluster Computing 5(2), 193–204 (2002) 15. Tomoyuki, O., Shinji, I., Yoshiaki, K., Kenji, I., Kaori, M.: An adaptive maintenance of hierarchical structure in ad hoc networks and its evaluation. In: ICDCS 2002, pp. 7–13 (2002) 16. Johnen, C., Nguyen, L.H.: Self-stabilizing construction of bounded size clusters. In: ISPA 2008, pp. 43–50 (2008) 17. Dom, M., Lokshtanov, D., Saurabh, S., Villanger, Y.: Capacitated domination and covering: A parameterized perspective. In: Grohe, M., Niedermeier, R. (eds.) IWPEC 2008. LNCS, vol. 5018, pp. 78–90. Springer, Heidelberg (2008) 18. Kuhn, F., Moscibroda, T.: Distributed approximation of capacitated dominating sets. In: SPAA 2007, pp. 161–170 (2007) 19. Johnen, C., Mekhaldi, F.: Robust self-stabilizing construction of bounded size weight-based clusters. Technical Report No 1518, LRI (2009), http://www.lri.fr/~ bibli/Rapports-internes/2009/RR1518.pdf

Adaptive Conflict Unit Size for Distributed Optimistic Synchronization Kim-Thomas Rehmann, Marc-Florian M¨ uller, and Michael Sch¨ ottner Heinrich-Heine-Universit¨ at D¨ usseldorf Universit¨ atsstraße 1, D–40223 D¨ usseldorf, Germany [email protected]

Abstract. Distributed and parallel applications often require accessing shared data. Distributed transactional memory is an emerging concept for concurrent shared data access. By using optimistic synchronization, transactional memory is simpler to use and less error-prone than explicit lock-based synchronization. However, distributed transactional memories are particularly sensitive to phenomena such as true sharing and false sharing, which are caused by correlated data access patterns on multiple nodes. In this paper, we propose a transparent technique that adaptively manages conflict unit sizes for distributed optimistic synchronization in order to relieve application developers from reasoning about such sharing phenomena. Experiments with micro-benchmarks and an on-line data processing application similar to Twitter (using the MapReduce computing model) show the benefits of the proposed approach.

1

Introduction

In recent years, numerous data sharing techniques emerged, such as in-memory data grids, cloud storage, and network-attached memory, used by distributed and parallel applications. As opposed to earlier distributed sharing techniques such as file sharing and distributed shared memory, these new techniques aim at being versatile and dynamic while at the same time guaranteeing consistency and reliability. Nonetheless, research on scalable, transparent distributed data sharing to complement existing message passing techniques is still under way. Sharing techniques benefit from the locality principle [1], which allows to improve data access performance based on correlated access patterns. For example, caching is a special form of replication where a processor keeps data that it has used earlier, thereby exploiting temporal locality. In addition, caches operate on cache lines larger than a single machine word, because processors often use adjacent words together, a phenomenon known as spatial locality. In distributed systems, locality of reference is particularly important. Misprediction or lack of locality cause excessive messaging overhead, which is expensive because of higher communication latencies and, depending on the network infrastructure, reduced network bandwidth. In order to formalize the principle of locality, researchers have defined the notions true sharing and false sharing [2]. True sharing is a situation where two P. D’Ambra, M. Guarracino, and D. Talia (Eds.): Euro-Par 2010, Part I, LNCS 6271, pp. 547–559, 2010. c Springer-Verlag Berlin Heidelberg 2010

548

K.-T. Rehmann, M.-F. M¨ uller, and M. Sch¨ ottner

or more nodes access the same object using read or write operations. If the object is accessed by read-only operations, replication improves access performance. If one or more nodes modify this object, all other nodes must be notified of these modifications. Coherence protocols such as update-on-write or invalidateon-write require network communication, if the nodes expect to view the system in a consistent state. The false sharing phenomenon results from nodes being unable to distinguish object accesses. In order to take advantage of spatial locality and to minimize bookkeeping overhead, some systems aggregate objects into consistency units. For example, the memory management unit (MMU) that virtualizes random access memory detects accesses at the granularity of virtual memory pages. If two or more nodes access indistinguishable but different objects and at least one node modifies a single object, all objects appear to be modified. Obviously, if false sharing accesses occur often, they will slow down applications as much as true sharing does. As opposed to true sharing, it is possible to avoid false sharing without modifying the application by choosing fine-grained consistency units. However, true sharing and false sharing are time-dependent phenomena. When access patterns change, true sharing can turn into false sharing and vice versa. The contribution of this paper is an adaptive management concept of consistency unit sizes in the context of a distributed transactional memory (DTM) system. DTM extends optimistic synchronization [3], the idea behind transactional memory (TM) [4,5], to distributed systems such as (federated) clusters [6,7,8,9,10,11]. DTM uses transactions to keep replicas consistent, avoiding complicated lock management and deadlocks. Beyond these benefits, speculative transactions bundle operations, allowing bulk network transfers while at the same time providing strong consistency. False sharing in TM leads to false conflicts, causing unnecessary transaction serializations. In contrast to related work, our approach is transparent for the application programmer. Internally we use larger consistency unit sizes whenever possible to allow bulk network transfers (for efficiency reasons). As soon as false sharing situations are detected during runtime, we transparently reduce the granularity of affected consistency units. If false sharing vanishes, we transparently aggregate smaller consistency units. The remainder of this paper is structured as follows. Section 2 reviews a static mechanism to avoid false sharing. In Section 3 we present a DTM that adapts its conflict unit sizes to avoid false sharing while supporting spatial locality. In Section 4 we evaluate our dynamic sharing technique. Section 5 discusses related work, and Section 6 concludes with an outlook on further improvements.

2

Static False Sharing Avoidance

Distributed systems that guarantee consistency of shared objects must control accesses to objects. The granularity of access detection influences the performance of distributed and parallel applications. On the one hand, coarse-granular

Adaptive Conflict Unit Size for Distributed Optimistic Synchronization

549

access detection allows bulk network transfers improving performance but may run into false sharing situations drastically degrading performance. On the other hand, fine-granular access detection is not prone to false sharing, but it does not support spatial locality, and it may incur additional run-time overhead. We classify access detection mechanisms as either being object, attribute or page based. Object-based access detection eliminates false sharing among different objects [12], but it requires either annotations by the programmer or instrumentation by a compiler. For example, the distributed system might rely on applications to notify object accesses by means of operations such as openObject [13]. At a finer granularity, attribute-based access detection completely avoids false sharing. However, allowing applications to modify different attributes of the same objects concurrently is counter-intuitive. In order to detect object accesses transparently with respect to both application and programmer, page-based access detection uses memory protection mechanisms built in hardware. 2.1

Page-Based Access Detection

We have decided to base our implementation of a DTM on page-based access detection for several reasons. First, using pages as consistency units benefits from locality of access and allows bulk network transfers. Second, although the MMU detects object accesses during address translation at hardware level, a user-level library can conveniently control access detection. Therefore, page-based access detection neither depends on the language in which the application was written, nor does it require any special markup for accesses. If a page lacks the requested access privilege, the MMU generates an exception, which the user-level library handles. Furthermore, in a distributed system, high communication latencies mitigate the overhead for local page-based access detection. Third, page-based access detection integrates well with transaction semantics, as we will discuss in Section 3.2. Although most modern processors support multiple page sizes (e.g. 4 KB and 2 MB respective 4 MB on x86 processors), operating systems usually do not allow applications to select the hardware page size. If objects are smaller than the page size, page-based access detection is prone to false sharing, because accesses to different objects on the same virtual page cannot be distinguished. Page diffing [14] permits locating write accesses at byte granularity. Writable implies readable on x86 processors, such that diffing cannot preclude false sharing, unless it reveals that a page has not been modified at all. A primitive approach to counteract false sharing would be to allocate objects sparsely in the virtual address space. However, placing each object in a distinct consistency unit trades exact access detection in for internal fragmentation. Although modern machines usually have plenty of physical memory available, internal fragmentation can increase memory consumption by a factor of thousand in extreme cases, for example when wasting a 4 KB page for a 4 Byte

550


object. Moreover, padding irrevocably eliminates the potential benefits of spatial locality. Even worse, the approach cannot adapt to different object usage patterns. 2.2

Multiview/Millipage Address Space Layout

The Multiview/Millipage approach proposed by Itzkovitz and Schuster [15] constructs special virtual-to-physical mappings, allowing for access detection at object granularity, nonetheless avoiding internal fragmentation as with primitive false sharing avoidance. A Millipage region divides a physical page frame into 2n disjoint Millipages. If the hardware page size is 2p , one Millipage covers 2p−n bytes. Each Millipage has a distinct mapping in the virtual address space, such that accesses to objects that reside on the same physical page frame are detected independently. A privileged mapping allows to circumvent access detection, thereby enabling atomic updates in multithreaded applications. Figure 1 illustrates Millipage layout with two Millipages per physical page frame.

Fig. 1. Millipage mappings (n = 2)

In our implementation of Multiview, one region’s Millipages reside in consecutive virtual memory pages, such that the region spans a range of 2np bytes in which only a total of 2p bytes belongs to valid Millipages. This simple convention about memory layout enables cheap access validation: An address’s offset in the virtual page must match the page’s index in its Millipage region, e.g. the application must reference the first object only through the first virtual page, the second object only through the second virtual page and so on. Failure of access validation indicates a corrupt memory pointer. The Multiview technique effectively decouples page size and consistency unit size. If each consistency unit holds at most one object, every access uniquely identifies a single object. Therefore, Multiview completely avoids false sharing. 2.3

Handling Huge Objects

The Multiview approach applies in case the consistency unit size is smaller than the page size. In contrast, some objects may be larger than page size, such that these objects must be allocated on consecutive full pages. False sharing cannot occur in these cases, set aside false sharing between attributes of the


551

same object. Nevertheless, it is interesting to note that there are two modes how a sharing service can synchronize access to the pages a large object consists of. Either the service synchronizes each page in stand-alone manner, that is, it allows parts of the object to be modified concurrently, or it handles the object as an aggregation of pages, that is, it automatically ascribes access to all pages simultaneously. If the operating system supports different page sizes in userspace, the sharing service can allocate huge objects on larger pages to improve the performance of access detection. Our DTM supports concurrent modification of object attributes.

3

Dynamic Conflict Size Management

The transaction concept allows concurrent activities to access multiple objects with implicit synchronization. A transaction bundles several read and write operations. Well-known in the context of databases, the so-called ACID properties [16] guarantee atomicity, consistency, isolation and durability of transactions. To ensure atomicity and isolation, transactions execute speculatively. After speculative execution, a subsequent validation phase ensures that a transaction only commits its modifications if speculative execution did not violate the consistency requirement. Durability of database transactions asserts that committed transactions cannot be undone. Transactional memory (TM) applies the transaction concept to in-memory data [17,5]. By integrating application logic and data access mechanism, TM avoids any potential database overhead and enables specific optimizations. For example, TM relaxes the durability of transactions, such that distributed state need not be written to disk. The object sharing service (OSS) [18] implements a DTM for XtreemOS [19] but also runs on any x86-based Linux system. The OSS provides shared objects as ranges in virtual memory, such that references to objects are simply memory pointers. For heterogenous setups we plan a pointer swizzling technique like for example implemented in Interweave [20]. Every TM requires application developers to define transaction boundaries by identifying code sections that access shared data concurrently. In the OSS, begin and end of transactions are specified by calling the corresponding library functions. In order to simplify application development, the OSS controls read and write operations transparently by using page-based access detection. As discussed in the introduction, a consistency unit size of one page (4 KB) is prone to false sharing. In a DTM system, false sharing causes false conflicts between transactions, leading to unnecessary transaction aborts. The adaptive conflict unit size management we propose in this section is flexible and transparent for the application programmer, relieving him of reasoning about data allocation and memory layouts causing false sharing situations. By providing an adaptive approach, we can support large consistency units and bulk network transfers whenever possible, but we can switch to a fine-grained Multiview consistency unit management in case false sharing shows up.

552

3.1


Page-Based Access Detection for Transactional Memory

At the beginning of a transaction, the transaction management requests access notification for all objects by revoking read and write permission for the corresponding virtual memory pages. The first read operation to a page causes Linux to report the object’s address to the OSS, which in turn inserts the address into the transaction’s read set and grants access to the corresponding page. The first write operation proceeds in a similar manner. In addition, the OSS creates a shadow copy containing the page’s original content, such that transaction management can restore the page in case the transaction cannot commit and must restart. Repeated read or write accesses to a page within the same transaction are not monitored and can run without any overhead. However, the first write operation on a previously read page causes a shadow copy to be created and moves the address from the read set to the write set. 3.2

Integration of Multiview into Transactional Memory on Linux

Our OSS supports using Millipages of different granularities and full (nonMillipage) pages side-by-side. When allocating an object, the OSS automatically chooses the Millipage granularity coarse enough to hold the object. The Millipage granularity is stored in the virtual page’s attributes that are themselves distributed objects with fixed granularity. The Multiview allocation scheme and the privileged mapping require multiple mappings of the same memory segment. Therefore, the OSS constructs memory mappings using System V shared memory segments, which can be attached repeatedly to a single address space. We have identified several synergies between memory transactions and the Multiview approach. First, Multiview restrains object size for read and write accesses, such that false aborts are eliminated. Second, Multiview speeds up shadow copy operations. When creating a shadow copy for a Millipage, the OSS needs to backup only a fraction of a full page, at most one physical page frame for an entire Millipage region. Similarly, Multiview restrains the range to compare for diff generation. Third, the privileged mapping allows transactions to run multithreaded in the same process. Otherwise, during non-atomic updates, all of a process’s threads would have to be halted. 3.3

Monitoring of Object Accesses

The Multiview technique completely avoids false conflicts if objects are always allocated on distinct pages, but degrades performance for access patterns that do not cause conflicts but could benefit from spatial locality. Moreover, transaction conflicts are dynamic phenomena, which depend on object access patterns. Consequently, we have implemented an access detection technique that dynamically adapts to the degree of false conflicts.


553

Our technique monitors object accesses to determine whether Millipages should be handled seperately or conjointly. In the context of TM, we designate consistency units as conflict units. A Millipage region that serves as coarse conflict unit is called object access group (OAG). To avoid exponential state-keeping and limit memory overhead, the monitoring mechanism considers only objects located in the same Millipage region. These objects have been allocated by the same node during some time interval, such that a semantical relationship among these objects is likely. Furthermore, a single system call can set the access protection for a contiguous region of virtual memory, such that using OAGs does not increase the number of costly switches between user and kernel mode. The dynamic adaptation mechanism bases its decisions only on local information in order to avoid network communication. Each node receives write sets from other committing nodes. Nodes need not transmit read sets, because remote read operations are not relevant to identify false conflicts, given that transactions in the OSS commit using a first-wins strategy. During the validation phase, transaction management determines whether a transaction conflicts with already committed transactions. In addition, for nonaggregated Millipage regions, our transaction management calculates whether OAGs would have caused hypothetic false conflicts. 3.4

Adapting Sharing Granularity

When aggregating objects into OAGs, it may happen that some objects in the group are not accessed during a transaction. Thus, the transaction’s read or write set might contain false positives. For objects in the write set, generating a diff between the actual object and its shadow copy reveals whether the object has been modified. Given that writable implies readable on our target architecture x86, transaction management must not ignore unmodified objects, but it can relocate them from the write set to the read set. For objects in the read set, it is impossible to detect whether they have actually been accessed in the transaction. As a consequence, false positives in read sets increase the probability of false transaction aborts but do not cause inconsistencies. The dynamic adaptation mechanism handles both the aggregation of objects to OAGs and the division of OAGs to objects with individual access detection. A sharing situation with spatial locality among objects in a Millipage region is characterized by few hypothetic conflicts. If hypothetic conflicts are rare and a read set contains several objects from the same region, the adaptation mechanism combines the Millipages into an OAG. To avoid oscillation, OAGs are formed no sooner than several transactions after splitting the region. We determined empirically that a reasonable stabilization interval is equal to the number of Millipages in the region. An OAG that causes a conflict during validation is subject to false conflicts or even true conflicts. Thus, the adaptation mechanism splits the OAG immediately.

554

3.5


Hints for the Application Developer

Monitoring of object accesses also assists the developer in identifying those objects that frequently cause true aborts. Conflict rates are aggregated among all participating nodes and published in the built-in name service, including information about which node and which function created the object. The developer can extract true sharing hotspots from the name service either periodically or manually, for example before terminating the application.

4


To evaluate the performance of our adaptive sharing technique, we have run micro-benchmarks under different sharing and allocation strategies. An on-line data processing application demonstrates that transactional memory benefits from adaptive sharing for realistic workloads. We ran our experiments on dualcore nodes equipped with AMD Opteron 244 processors running at 1.8 GHz under Linux 2.6.26. The nodes were connected via Gigabit Ethernet over Broadcom NetXtreme NICs. 4.1

Micro-Benchmarks

We have run synthetic workloads with four different allocation schemes. The dlmalloc allocator is a general-purpose allocator, similar to the one used by the GNU standard C library. We use its MSpaces variant, which enables multithreaded allocations of transactional memory. MSpaces is quite space-efficient but prone to false sharing. The Page allocator places each object in a separate physical page frame. It implements the primitive approach against false sharing and causes internal fragmentation for object sizes that are not a multiple of page size. The Millipage allocator statically places all objects on Millipages and does not aggregate objects. The Adaptive allocator is based on the Millipage allocator and implements adaptive sharing based on OAGs. To express an allocation scheme’s reaction on an access pattern, we have measured the number of detected accesses. For the first test, the setup consists of two nodes accessing two objects that have been allocated consecutively. The examined node reads both objects, the second node writes to one object, causing frequent transaction restarts on the examined node. A simple fairness strategy in OSS ensures that a transaction will commit after restarting once. Figure 2 impressively demonstrates that the MSpaces allocator is susceptible to false sharing, whereas the other allocators enable to distinguish both objects. In the second test, a single node accesses two objects conjointly in a loop of 216 transactions. The Page and Millipage allocator detect each access separately, as depicted in Figure 3. The MSpaces and Adaptive allocator only detect one access per transaction because of spatial locality between the objects.


555

Fig. 2. Access detections induced by accessing distinct objects

Fig. 3. Access detections induced by accessing objects conjointly

Fig. 4. Memory consumption (logarithmic scale)

Fig. 5. Dynamic adaptation from OAGs to Millipages and vice versa

For a synthetic workload with 1024 4-Byte objects, the memory consumption of the Page allocator is severe, whereas the other allocators allocate only the requested object size plus some allocation meta-data (see Figure 4). We have also evaluated how well our technique adapts to varying object access patterns. The setup consists of two nodes, one of which is running transactions in a loop for 220 times, reading from two objects. The other starts up about six seconds later, runs transactions in a loop for 219 times, writing to one of the objects. Initially, the second node does not run transactions at full speed, which causes the first node to switch several times between coarse-granular and finegranular access detection. Figure 5 subdivides the number of access detections for OAGs and for Millipages. 4.2

MapReduce

MapReduce [21] is a computing model for processing large amounts of data. The model applies to certain problems where mapping the input data to a different domain allows highly parallelized computations. Being easy to understand, MapReduce has reached widespread use. For example, Google uses MapReduce for different search and extraction problems in more than 4000 applications. The common introductory example for MapReduce is word frequency analysis. For determining the frequency of words in an input text, the map phase emits each individual word with a count of 1. The reduce phase afterwards collects all identical words from map phase’s output and sums up their counts, yielding the total count per word.

556


Fig. 6. Example trie storing the frequency of the words tree, trie (2) and try

The word frequency problem exemplifies that MapReduce is well-suited for analyzing static data. Dynamic, interactive information sharing, such as Web 2.0 applications that are currently emerging, needs processing facilities for continuous data streams. To achieve good scalability, continuous data should be processed in parallel. As a consequence, the computing model must efficiently support concurrent access to shared data. We have applied MapReduce to processing of continuous data streams. Our implementation bases on our DTM. Extending the word frequency example, we illustrate the effectivity of adaptive conflict size management with continuous analysis of text data streams, using a scenario resembling the well-known Web 2.0 application Twitter. The continous word counting example operates on a trie [22] where each word is represented by a path from the tree’s root to a node. The node at the end of a word stores the frequency of the word it terminates, possibly other statistical information such as time stamps too. Intermediate nodes represent prefixes of a word, storing at most 26 references to next prefix characters (see Figure 6). In our implementation, each node has a size of 216 Bytes, which equals 26 references to child nodes plus a 64-Bit counter. When allocating nodes for the trie, the Adaptive allocator splits a physical page frame in 16 Millipages, each 256 Bytes large, causing 16% internal fragmentation. The trie representation of words already counteracts false sharing by enforcing a high fan-out, e.g. compared to a representation of words in a binary tree. Our implementation serves back-to-back allocations from the same Millipage region, if space allows so. Therefore, nodes tend to reside in the same region as their ancestors and descendants, such that grouping adjacent objects makes sense.

Fig. 7. Access detections for word frequency analysis

Fig. 8. Transaction aborts for word frequency analysis


557

In our experiment, we connected two nodes using our DTM. Each node simulated a user who entered some text (the novel Kim written by Rudyard Kipling). The text consisted of 107585 words in total, thereof 10636 different words. Again, we measured the number of access detections representing how well an allocator makes use of locality (see Figure 7). Additionally, we determined the number of transaction restarts, which indicates how much access detection suffers from false sharing (see Figure 8). Our adaptive access detection mechanism triggers only 60% access detections compared to the Millipage allocator, and it causes less than 25% transaction restarts compared to the MSpaces allocator.

5

Related Work

The implications of sharing phenomena and their interdependencies have been discussed mainly in the context of caching hardware and distributed shared memory (DSM). Several consistency models that take account of sharing have been defined, for example scope consistency [23] and view-based consistency [24]. These models provide weaker consistency than TM. The Region-trap library [25] combines pointer swizzling and virtual memory protection to trap accesses to individual objects, requiring region pointer annotation. Amza et al. [26] describe the dynamic aggregation of pages for lazy release consistency [14]. Our work has some similarities with ComposedView [27]. ComposedView provides transparent aggregation of small consistency units for sequential consistency, but to our knowledge the technique has not been applied to TM yet. The impact of false sharing on TM has been discussed recently, for example in the VELOX project [28]. Burcea et al. [29] propose to vary access tracking granularity. In contrast to our approach, the authors focus on per-object granularity that does not adapt dynamically to access patterns. Bocchino et al. [7] implement a DTM for large-scale clusters. They define eight design dimensions for their TM, one dimension is the static size of conflict detection units.

6

Conclusion

We have presented an approach for the adaptive management of conflict unit sizes for a distributed transactional memory system. The combination of a smart allocation strategy and transparent access monitoring avoids false sharing and thus unnecessary transaction aborts caused by false conflicts. At the same time we support locality whenever possible, allowing bulk network transfers to speed up distributed processing. The proposed solution is transparent for the application programmer and is able to adapt its strategy to changing access patterns. The evaluation using micro-benchmarks and a MapReduce application demonstrate the benefits of the adaptive conflict unit size management while at the same time introducing only minimal overhead. Clearly, transactional memory is attracting a lot of people in research and industry, and recently some of these ideas have also shifted to distributed systems. Therefore, we expect more and more transactional applications, also for distributed environments.

558


We plan to study more flexible object access groups containing objects from different Millipage regions. In this context, Bloom filters [30] are a promising data structure for statistical monitoring of large data sets. Finally, we have started with large-scale experiments on the Aladdin-Grid’5000 platform.

References 1. Denning, P.J., Schwartz, S.C.: Properties of the working-set model. ACM Commun. 15(3), 191–198 (1972) 2. Torrellas, J., Lam, M.S., Hennessy, J.L.: False sharing ans spatial locality in multiprocessor caches. IEEE Trans. Computers 43(6), 651–663 (1994) 3. Kung, H.T., Robinson, J.T.: On optimistic methods for concurrency control. ACM Trans. Database Syst. 6(2), 213–226 (1981) 4. Herlihy, M., Moss, J.E.B.: Transactional memory: architectural support for lockfree data structures. SIGARCH Comput. Archit. News 21(2), 289–300 (1993) 5. Felber, P., Fetzer, C., Guerraoui, R., Harris, T.: Transactions are back—but are they the same? SIGACT News 39(1), 48–58 (2008) 6. Kotselidis, C., Ansari, M., Jarvis, K., Luj´ an, M., Kirkham, C., Watson, I.: DiSTM: A software transactional memory framework for clusters. In: ICPP 2008: Proceedings of the 37th IEEE International Conference on Parallel Processing, September 2008, IEEE Computer Society Press, Los Alamitos (2008) 7. Bocchino, R.L., Adve, V.S., Chamberlain, B.L.: Software transactional memory for large scale clusters. In: PPoPP 2008: Proceedings of the 13th ACM SIGPLAN Symposium on Principles and practice of parallel programming, pp. 247–258. ACM, New York (2008) 8. Herlihy, M., Sun, Y.: Distributed transactional memory for metric-space networks. Distributed Computing 20(3), 195–208 (2007) 9. Manassiev, K., Mihailescu, M., Amza, C.: Exploiting distributed version concurrency in a transactional memory cluster. In: PPoPP 2006: Proceedings of the eleventh ACM SIGPLAN symposium on Principles and practice of parallel programming, pp. 198–208. ACM, New York (2006) 10. Romano, P., Carvalho, N., Rodrigues, L.: Towards distributed software transactional memory systems. In: LADIS 2008: Proceedings of the 2nd Workshop on Large-Scale Distributed Systems and Middleware, pp. 1–4. ACM, New York (2008) 11. Couceiro, M., Romano, P., Carvalho, N., Rodrigues, L.: D2STM: Dependable distributed software transactional memory. In: PRDC 2009: Proc. 15th Pacific Rim International Symposium on Dependable Computing (November 2009) 12. Bal, H.E., Bhoedjang, R., Hofman, R., Jacobs, C., Langendoen, K., R¨ uhl, T., Kaashoek, M.F.: Performance evaluation of the Orca shared-object system. ACM Trans. Comput. Syst. 16(1), 1–40 (1998) 13. Herlihy, M., Luchangco, V., Moir, M., Scherer, I.W.N.: Software transactional memory for dynamic-sized data structures. In: PODC 2003: Proceedings of the twenty-second annual symposium on Principles of distributed computing, pp. 92– 101. ACM, New York (2003) 14. Keleher, P., Cox, A.L., Zwaenepoel, W.: Lazy release consistency for software distributed shared memory. In: Proc. of the 19th Annual Int’l Symp. on Computer Architecture (ISCA 1992), pp. 13–21 (1992)


559

15. Itzkovitz, A., Schuster, A.: MultiView and Millipage – fine-grain sharing in pagebased DSMs. In: OSDI 1999: Proceedings of the third symposium on Operating systems design and implementation, pp. 215–228. USENIX Association, Berkeley (1999) 16. Haerder, T., Reuter, A.: Principles of transaction-oriented database recovery. ACM Comput. Surv. 15(4), 287–317 (1983) 17. Dias, R.J., Louren¸co, J.M.: Unifying memory and database transactions. In: Sips, H., Epema, D., Lin, H.-X. (eds.) Euro-Par 2009 Parallel Processing. LNCS, vol. 5704, pp. 349–360. Springer, Heidelberg (2009) 18. M¨ uller, M.F., M¨ oller, K.T., Sonnenfroh, M., Sch¨ ottner, M.: Transactional data sharing in grids. In: PDCS 2008: Proceedings of the International Conference on Parallel and Distributed Computing and Systems (2008) 19. Christine, M.: XtreemOS: A Grid operating system making your computer ready for participating in virtual organizations. In: ISORC 2007: Proceedings of the 10th IEEE International Symposium on Object and Component-Oriented Real-Time Distributed Computing, Washington, DC, USA, pp. 393–402. IEEE Computer Society, Los Alamitos (2007) 20. Chen, D., Dwarkadas, S., Parthasarathy, S., Pinheiro, E., Scott, M.L.: Interweave: A middleware system for distributed shared state. In: Languages, Compilers, and Run-Time Systems for Scalable Computers, pp. 207–220 (2000) 21. Dean, J., Ghemawat, S.: MapReduce: simplified data processing on large clusters. ACM Commun. 51(1), 107–113 (2008) 22. Knuth, D.E.: The art of computer programming. In: sorting and searching, 2nd edn., vol. 3. Addison Wesley Longman Publishing Co., Inc., Redwood City (1998) 23. Iftode, L., Singh, J.P., Li, K.: Scope consistency: a bridge between release consistency and entry consistency. In: SPAA 1996: Proceedings of the eighth annual ACM symposium on Parallel algorithms and architectures, pp. 277–287. ACM, New York (1996) 24. Huang, Z., Sun, C., Purvis, M., Cranefield, S.: View-based consistency and false sharing effect in distributed shared memory. SIGOPS Oper. Syst. Rev. 35(2), 51–60 (2001) 25. Brecht, T., Sandhu, H.: The region trap library: handling traps on applicationdefined regions of memory. In: ATEC 1999: Proceedings of the annual conference on USENIX Annual Technical Conference, p. 7. USENIX Association, Berkeley (1999) 26. Amza, C., Cox, A., Rajamani, K., Zwaenepoel, W.: Tradeoffs between false sharing and aggregation in software distributed shared memory. In: PPOPP 1997: Proceedings of the sixth ACM SIGPLAN symposium on Principles and practice of parallel programming, pp. 90–99. ACM, New York (1997) 27. Niv, N., Schuster, A.: Transparent adaptation of sharing granularity in MultiViewbased DSM systems. Softw. Pract. Exper. 31(15), 1439–1459 (2001) 28. Harmanci, D., Felber, P., Gramoli, V., Fetzer, C.: TMUNIT: Testing transactional memories. In: TRANSACT 2009: 4th Workshop on Transactional Computing, Feburary (2009) 29. Burcea, M., Steffan, J.G., Amza, C.: The potential for variable-granularity access tracking for optimistic parallelism. In: MSPC 2008: Proceedings of the 2008 ACM SIGPLAN workshop on Memory systems performance and correctness, pp. 11–15. ACM, New York (2008) 30. Bloom, B.H.: Space/time trade-offs in hash coding with allowable errors. ACM Commun. 13(7), 422–426 (1970)

Frame Allocation Algorithms for Multi-threaded Network Cameras José Miguel Piquer1 and Javier Bustos-Jiménez2 1

Departamento de Ciencias de la Computacion (DCC), Universidad de Chile [email protected] 2 Escuela de Ingenier´ıa Informática, Universidad Diego Portales [email protected]

Abstract. This paper presents a first attempt to solve a challenging problem, proposing novel and successful algorithms to efficiently distribute video frames from network cameras to many concurrent clients. The usual scenario studied is composed of a camera generating video frames at a given rate and distributing them over a network to several concurrent clients. In general, the idea is to allocate one thread per client at the camera, sharing a pool of one-frame buffers. The algorithms studied consider the allocation of buffers to new frames and the allocation of frames to clients. We study different combinations of algorithms, buffers and clients in order to find an optimal solution for the usual scenarios we face when the network camera is under heavy use. The main conclusion is that frame allocation algorithms have a strong impact on system performance: under the same conditions, client performance improves from 4 to 25 frames per second with the best algorithm combination at the camera. Keywords: Video over IP, frame allocation, network camera.

1 Introduction The Video over IP transmission is becoming popular with the advent of high-speed networking, good video encoding algorithms (reducing bandwidth usage while preserving the image quality [12,13]), and the massification of inexpensive video hardware. This has created a lot of new challenges for the Internet, particularly related to congestion control [8]. Also, a lot of research has been done on video compression [4,5], transmission with packet loss recovery [10,2] or using Quality of Service extensions [9,6]. This has enabled the advent of network cameras: digital video cameras integrated with a network connection, accepting client connections from the Internet and sending them a real-time video stream. A typical network camera includes a web server to authenticate users and a video server to send the video frames to the clients using a proprietary protocol (running over UDP or TCP). A network camera is expected to support concurrent clients with limited hardware resources, namely memory and processor. A typical commercial Axis camera limits the number of clients to a maximum of 20. Our objective is to develop P. D’Ambra, M. Guarracino, and D. Talia (Eds.): Euro-Par 2010, Part I, LNCS 6271, pp. 560–571, 2010. c Springer-Verlag Berlin Heidelberg 2010

Frame Allocation Algorithms for Multi-threaded Network Cameras

561

algorithms to support more than 100 concurrent clients with less memory than a typical Axis camera, using only 500 Kbytes of RAM. The configuration used is with color 640x480 resolution frames, JPEG encoded giving usual sizes of around 50 Kbytes each. The main goal is to support as many clients as possible with a given hardware capacity. Another important objective of our algorithms is to be able to deliver 30 frames per second to a fast client in a high-speed network. So, we propose algorithms preserving that performance for fast clients even if many clients are much slower than that. As an example, a home user connected at 256 Kbps will be able to receive only one frame every two seconds whereas a local 100 Mbps ethernet user will be able to receive 30 frames per second using almost 10 Mbps bandwidth. To provide this service, the camera needs to store video frames in its own memory while they are being transmitted to their clients. However, we cannot afford to allocate one frame per client, as memory is a scarce resource in the camera. The problem studied in this paper is how to manage a bounded pool of frames in memory, choosing which frames to keep and which frames to send to the different clients running at different speeds. This problem, to our knowledge, has not been studied previously in the literature. In our experiments, we send a complete frame to each client, using Motion JPEG [1] as the video codec. If a differential encoder is used (as H.261 [4]), the frame allocation problem remains the same: we need to choose a frame to calculate the differences with the last one, and to keep a pool of frames to choose from. However, we have not tested differential encoders, and all the experiments performed for this paper were carried out sending complete JPEG video frames. The paper studies the server side of the problem (camera). Of course, the client can provide some buffering or even to store the whole video to play it afterwards. However, if we want to stream the video from the camera in real time, buffering does not help to solve the different capacities between clients, it only helps to conceal variance in point to point bandwidth. In this scenario, only the server can solve the problem. The main contributions of this paper are: a comprehensive study of a new area of research for network cameras, a list of new algorithms to manage frames and clients, and a performance analysis based on simulation, and a proposal of efficient combinations of algorithms for frame allocation. The paper is organized as follows: Section 2 presents the problem and the algorithms involved. Section 3 explains each algorithm in detail. Section 4 presents the designed experiments and the obtained results. Finally, Section 5 presents our conclusions.

2 Frame Allocation and Replacement A network camera with only one client is always running an infinite loop getting the next frame from the hardware, copying it into memory, and then sending it to the client over the network. The main performance measures considered are the frames per second (FPS) that the camera hardware can generate and the FPS that can be displayed at the client. The FPS depends on several factors, such as frame size, resolution, color depth, network bandwidth, protocol used, etc. In this paper we concentrate only on the parameters that are directly controlled by the camera itself and leave out the network protocol used.

562

J.M. Piquer and J. Bustos-Jiménez

If a network camera can generate a maximum of x FPS, and we have a client that can display up to y FPS, we suppose that the network camera with only this client is always able to deliver the min(x, y) FPS at the client. We call a client that is able to display at the camera FPS or higher (y ≥ x), a Fast Client. We define as Slow Client a client that is only able to display at an FPS five times lower than the camera FPS (5y ≤ x). In a real environment, it means a client receiving at less than 6 FPS. 2.1 Concurrent Clients When a network camera has more than one connected client, it must send the same video stream over different connections. The simplest algorithm is to read the next video frame into memory and then send it to every client. However, in this case, a new frame cannot be read into memory (and thus must be discarded) until the slowest client has finished receiving the previous one. Thus, all clients are forced to receive at the same frame rate, determined by the slowest client. In practice, all clients perform at the slowest client frame rate (Figure 1(a)). If we denote by yi the FPS of the client i, every client is receiving the video at min(x, yk ) FPS, where yk = min(yi ), ∀i. network camera network camera

client 1

client 1 client 2

client 2 client n client n

(a) Only one shared buffer

(b) One buffer per client Fig. 1. Camera buffering

A better solution is to develop a multi-threaded server, with one thread per client, a thread for the camera, and a separate buffer allocated in memory for each client (see Figure 1(b)). The camera thread must copy each new frame to all free client buffers. This solution provides each client with an optimal frame rate, as we have every client i receiving the video at min(x, yi ) FPS. This is the optimal solution, as it is equivalent to the case where each client is connected alone to the camera, as long as the camera is able to copy the frames to all the buffers without degrading its own FPS. However, memory is a scarce resource on the camera and in practice we are forced to limit the number of buffers in memory to a fixed maximum number to ensure that we do not run out of memory. On the other hand, we do not want to limit the number of clients to the number of buffers in memory. In general, we want to be able to support a number of clients much higher than the number of buffers (around 10 times), without degrading the overall performance too much.


563

2.2 The Frame Pool We designed a solution that is in between the two previous algorithms: sharing a pool of frames between clients to have more than one frame allocated in memory but less than one frame per client. To enable frame sharing, the server is divided in two parts: a camera thread reading the frames from the hardware and putting them into a frame pool, and several client threads (one per client) retrieving a frame from the frame pool and sending it to the corresponding client (see Figure 2). network camera rc=2

client 1

rc=1

client 2

t=2

camera thread

t=1

client n

t=0 rc=0

buffer pool

client threads

Fig. 2. Many shared buffers

The frame pool is implemented using a fixed-length array of buffers (depending on the camera memory size), so the camera thread and the client threads must synchronize between them to block when the buffer pool is either full or empty. This synchronization problem is very similar to the well-known readers/writers problem [11], but with a few stronger restrictions1 : Each buffer holds a frame with a timestamp of the time at which the hardware generated it and a reference count which is a counter of the number of client threads currently using that buffer (see Figure 2). Each time the camera thread gets a new frame from the hardware, it looks for an available buffer where to write the frame. In general, all buffers are holding a frame, so one must be selected to be replaced by the new one. A buffer that has one or more client threads reading it (i.e. has a reference count greater than zero) cannot be replaced. So, the only candidates for replacement are the frames with reference count zero. If there is none, the camera thread blocks until such a frame is available. Each time the client thread needs a new frame to send to its client, it looks for a buffer holding a usable frame. A usable frame is any of the frames in buffers that have a timestamp newer than the last frame it sent to its client (to ensure the monotony of time for the video). In pseudo-code, the camera thread and the client threads can be seen in Figure 3. Where camera frame replacement (get free buf()) is the algorithm executed by the camera thread to select a buffer to hold a new frame received from the hardware. As 1

This problem is a very good example of process synchronization for an Operating Systems or Concurrent Programming course. We have been using it successfully: it is complex yet understandable.

564


camera_thread() { for(;;) { buf = get_free_buf(); buf.frame = read_camera(); buf.timestamp = time(); put_buf(buf); } }

client_thread(socket) { t=0; for(;;) { buf = get_next_buf(t); send(buf.frame, socket); t = buf.timestamp; free_buf(buf); } }

Fig. 3. Pseudo-code for camera and client threads

all the buffers hold a frame in steady state, a frame will be deleted by this operation. The frame replacement algorithm is implemented by get free buf(), a function that finds buffers with a reference count zero and selects the best candidate discriminating by the timestamp value. If no buffer with reference count zero exists, the function blocks until one is available (signaled by free buf()). And, client frame allocation (get next buf()) is the algorithm executed by all the client threads each time they have finished sending a frame to select the new frame to transmit. The client frame allocation algorithm is implemented by get next buf(), a function that finds a buffer with a timestamp greater than the last one sent by the client thread. If many buffers with suitable timestamps are available, it can use the reference counts and the timestamps to choose one. If none is available, the function blocks until a new frame is generated by the camera thread (signaled by put buf()). Different client threads can share the same buffer, as they are only reading it. The reference counts indicate how many client threads are using each buffer; put buf() initializes the reference count of the buffer to zero, get next buf() increments it and free buf() decrements it. While the camera thread is blocked waiting for a buffer where to put the next frame, the system is losing frames from the hardware which continues to capture at the hardware FPS (x) but no thread is reading. All performance differences on the client side are related to this effect: if the camera blocks for more than 1/x seconds, the FPS at fast clients immediately drops. If the camera never blocks, fast clients receive the maximum FPS. Therefore, all good algorithms improve the probability of finding a free buffer to put a new frame to avoid the blocking of the camera thread.

3 Proposed Algorithms We designed three frame replacement and five frame allocation algorithms that can be combined freely. We implemented all of them in order to compare their final performance. 3.1 Camera Frame Replacement We studied three different algorithms for the camera thread to choose a frame to be replaced, given a list of all the frames stored in buffers with reference count zero.


565

Oldest First (OF): Selects the frame with the oldest timestamp. This algorithm replaces the frames one by one, erasing always the oldest frame in the pool, supposing it is the less useful for the clients. Newest First (NF): Selects the frame with the most recent timestamp. This algorithm tends to renew the last frame generated if only slow clients are present, thus using only a few buffers and minimizing the number of useful frames in memory for every client. Any Frame (ANY): Selects the first available frame. This algorithm searches sequentially the buffers and selects the first frame that has reference count zero. Hence, it chooses a frame essentially arbitrarily and serves mainly as a benchmark to test if using the timestamp information improves the performance. 3.2 Client Frame Allocation We studied five different algorithms for the client threads to choose the next frame to be sent, given the list of all the frames stored in buffers with timestamps greater than the last frame they sent. Oldest First (OF): Sends the frame with the oldest timestamp. This algorithm sends the frames in order without skipping frames that are still available in memory. Newest First (NF): Sends the frame with the most recent timestamp. This algorithm sends the new frames as quickly as possible, skipping all the intermediate frames and catching up with the newest available. Maximum Reference Count (MAX): Sends the frame with the maximum reference count. This should increment the number of available buffers with reference count zero at any given time. Ties are resolved using the timestamp, picking the newest. This algorithm improves buffer sharing, maximizing reference counts. Maximum Reference Count/Oldest First (MOF): Sends the frame with the maximum reference count. Ties are resolved using the timestamp, picking the oldest. Any Frame (ANY): Sends an arbitrary usable frame. This algorithm provides an arbitrary behavior. As before, it serves mainly as a baseline to compare with the others.

4 Experiments and Results We carried out two parallel experiments: one was executed using a real network and many clients connecting to a camera server and the other was a simulation ran on the scientific simulator Scilab. The first experiment was run mainly to ensure that our simulated results were consistent with the real ones. The Internet is a complex system to be simulated [3,7], so it is important to verify our results. The real experiment was deployed up to 15 clients and the simulation up to 100. We will only show here the results from the simulation, as the real experiment was fully consistent with it. For this work, we do not consider the network and protocol behaviors, and we focus only on the efficient management of the frames in the camera memory.

566


4.1 Frames per Second We first tested all combinations of algorithms using one main measure: frames per second obtained by a Fast Client (see section 2), while we keep adding Slow Clients to the camera. We designed a realistic scenario for the experiment: we fixed the number of buffers allocated in memory, we ran two Fast Clients permanently, and we added Slow Clients one by one, measuring the FPS received by the fastest client. The two Fast Clients were capable of receiving 34 FPS and 24 FPS respectively. The Slow Clients were able to display between 3.5 and 5.5 FPS. To avoid synchronization between the arrivals of clients requests, we forced all the clients to have different speeds. Whenever asked by read camera(), the camera always returns a new frame, so there is no limitation on the FPS from the camera itself. The fastest client FPS depends only on the FPS that the camera thread is able to put into the buffer pool: a client thread will only block waiting for a frame if no frame newer than the last it has sent is available, and this only happens if the camera thread was not able to put that frame in the pool (the only exception would be if the client is faster than the camera, but this is not the case here). Frame Allocation. To select the best Frame Allocation algorithms, we run each experiment during 10 minutes, generating 20,000 frames at the camera. The results for 4, 6 and 8 buffers fixing Newest First as the Frame Replacement algorithm, are shown in Figure 4. Each point in each graph is the mean FPS measured in the fastest client during 20,000 frames generated at the camera. For the frame allocation algorithm, we can separate three groups from the graphs: the best group (MAX and MOF), the intermediate group (ANY and NF), and the bad one (OF). Note that with 100 slow clients and only 8 buffers, the best group can still keep the fastest client at 25 FPS while the worst has it already at 5 FPS (actually this is equivalent to the slowest client we have). MAX performs better than MOF at the end. During the very early stages of the test, with only a few clients, MOF tends to be a little better. Anyhow, MAX is clearly better at high loads, when performance is crucial. OF is interesting because it behaves very well at the beginning but degrades fast and finishes even worst than ANY. We can conclude that the best algorithm (MAX) allows us to serve 100 clients in our experiment with 8 buffers, degrading the fastest client only from 34 to 26 FPS. So we are able to serve 12 clients per buffer with a degradation of only 24% in the frame rate. If we look at the 6-buffer case, we are able to serve 45 clients with 25 FPS, giving 7 clients per buffer. In the 4-buffer case, we get 12 clients with 25 FPS, giving us 3 clients per buffer. Even though these are preliminary results, it seems that for MAX, the number of clients increases better than linear with respect to the number of buffers. We ran again all these expermients with different Frame Replacement algorithms fixed as default (to see if the use of NF played a role in the results) and the curves were exactly the same. Frame Replacement. To select the best Frame Replacement algorithm, we re-ran each experiment for 10 minutes, generating 20,000 frames at the camera. The results for 4, 6 and 8 buffers using MAX as the Frame Allocation algorithm are shown in Figure 5. Each point in the graph is the mean FPS measured in the fastest client during 20,000 frames generated at the camera.

Frame Allocation Algorithms for Multi-threaded Network Cameras Fast Client, Buffer = 4 ANY NF OF MAX MOF

30

frames / sec

25

20

15

10

5

0 10

20

30

40

50

60

70

80

90

100

Slow clients

Fast Client, Buffer = 6 ANY NF OF MAX MOF

30

frames / sec

25

20

15

10

5

0 10

20

30

40

50

60

70

80

90

100

70

80

90

100

Slow clients Fast Client, Buffer = 8

30

frames / sec

25

20

15

10

5

0

ANY NF OF MAX MOF 10

20

30

40

50

60

Slow clients

Fig. 4. Frame Allocation and FPS, 4,6 and 8 buffers

567

568

J.M. Piquer and J. Bustos-Jiménez Camera Algorithms, Fast Client

buf=8

30

frames / sec

25

20

buf=6

15

10

5

0

ANYMAX, buf=4 ANYMAX, buf=6 ANYMAX, buf=8 NFMAX, buf=4 NFMAX, buf=6 NFMAX, buf=8 OFMAX, buf=4 OFMAX, buf=6 OFMAX, buf=8 10

20

buf=4

30

40

50 Slow clients

60

70

80

90

100

Fig. 5. Frame Replacement FPS, 4, 6, 8 buffers

Note that once the number of buffers and the Camera Frame Allocation algorithm are fixed, the Frame Replacement algorithm is irrelevant with respect to the FPS received at the fastest client. This can be expected, as different algorithms do not affect the FPS put into the buffer pool by the camera, because they only replace different frames. This behaviour is confirmed by measures taken with all the other Camera Frame Allocation algorithms. 4.2 Distance between Frames We noted in previous experiments that Frame Replacement algorithms do not affect the FPS received by the fastest client. However, the algorithms do select different frames to replace, so the clients are getting the same FPS but they are not receiving the same frames. This difference may be perceived by the user, as some algorithms generate jumps in the frame sequence and at certain point this could become annoying. To study this effect, we measured the perceived smoothness of the video stream. The smoothness of a video stream can be measured calculating the distance (number of skipped frames) between consecutive frames received at a given client. If the frames generated at the camera are numbered sequentially, each client knows exactly how many frames is skipping each time it receives a new frame. We call this number frame distance. As the FPS is the same for all algorithms, the mean frame distance will also be the same in the long run. However, the standard deviation of the frame distance can be different depending on the algorithm combination. The idea is that a small deviation should be better than a big deviation, as a sequence of equally spaced frames is better than a sequence where some frames are very near in time and then a big skip occurs. We


569

suspected that the camera replacement algorithms could make a difference here, as they replace different frames in the buffer pool. Also, we suspected that the reference-count based client algorithms (MAX and MOF) could be worse in terms of distance variation as they select the frames to send primarily based on their reference count instead of their timestamps. To measure the distance variation, we picked one Slow Client in the same experiments, measuring its frame distance variation using 8 buffers. The Fast Clients are not very interesting as they are able to receive the full stream of frames, so the frame distance is too low. We measured with 8 buffers, as a high number of buffers generates more options for the algorithms and thus should generate more variation. Frame Allocation Algorithms. Figure 6 shows, for the NF Frame Replacement algorithm, combined with the five Frame Allocation algorithms, the standard deviation of frame distance divided by the mean distance, to normalize it to be between 0 and 1. In this case, a lower value is better than a high value: 0 corresponds to a constant stream without variance on the frame distance, while a value 1 indicates that we cannot predict the distance of the next frame based on history, i.e. a very variable stream of frames. Note that the different algorithm combinations generate very different results in terms of frame distances. As expected, ANY is the worst and the reference-based algorithms are worse than the time-based algorithms. It should be noted than a variance near 1 means that there is almost no correlation between a pair of frames received in sequence. How disturbing is a level of variance around 0.5 to a human user is something to be studied in the future. If it is a serious problem, maybe a tradeoff between FPS and frame variance will be needed. Frame distance, Newest First Camera, Slow Client 0.7

0.6

std dev/mean distance

0.5

0.4

0.3

0.2

ANY NF OF MAX MOF

0.1

0 0

10

20

30

40

50

60

70

Slow clients

Fig. 6. Frame Allocation and Variance

80

90

100

570

J.M. Piquer and J. Bustos-Jiménez Camera algorithms, buffer=8 0.7

0.6

stdev / mean

0.5

0.4

0.3

0.2

0.1 ANY Camera NF Camera OF Camera

0 0

20

40

60

80

100

Slow clients

Fig. 7. Frame Replacement and Variance

Frame Replacement. To see if the Frame Replacement algorithms play a role in the variance between frames, we fixed the Frame Allocation to MAX (the best algorithm in terms of FPS) and we compared the obtained variance between the three Frame Replacement algorithms (see Figure 7). Note that in the first part of the graph, NF generates a lower variance. After 25 clients, the algorithms are equivalent.

5 Conclusions In this paper we presented a complete study of different algorithms to distribute video to many concurrent point-to-point clients with a limited number of buffers in memory. The different algorithms tested showed that the best choice for the client Frame Allocation algorithm, in terms of FPS, is the Maximum Reference Count (MAX algorithm). However, MAX generates high variance in the frame distance, although using Newest First (NF) as the camera Frame Replacement algorithm we can alleviate this in the first stages. This solution can handle 12 clients per buffer with 8 buffers, with a degradation of only a 24% for the fastest client (from 34 to 26 FPS) and a frame distance standard deviation over the mean of 0.5 (around +/- 10 frames). If FPS maximization is not the main goal and the frame variance is the most important measure, the combination NF/NF seems to be the best choice, but at an enormous cost in FPS: with 12 clients per buffer and 8 buffers, we get a degradation of 74% for the fastest client (from 34 to 9 FPS) and a frame distance standard deviation over the mean of 0.2 (around +/- 1 frame). In our experiments, we increased the numbers of buffers from 4 to 8 and the performance of the MAX algorithm gave that the number of clients increased better than


571

linear with respect to the number of buffers. As memory will continue to be an issue in the future on all network cameras, as the number of clients keeps growing, the solutions proposed here are crucial to limit the memory consumption in the camera. A lot of future work remains to be done: to study the curves beyond 100 clients and 8 buffers, to adapt these algorithms to other codecs (particularly for time-compression) and to study the relevance of variance in the user perception. The main conclusion of this paper is that the Frame Allocation algorithms are crucial to provide high-performance video delivery to fast clients under heavy load. Note also that this approach could be used on P2P/Ad-Hoc video streaming, where faster/better equipped peers/nodes could download frames to serve their slower neighbors.

References 1. Smith, B., Rowe, L.: Compressed domain processing of jpeg-encoded images. Real-Time Imaging 1(2), 3–17 (1996) 2. Bolot, J., Turletti, T.: Experience with control mechanisms for packet video. ACM Communication Reviews 28(1) (1998) 3. Floyd, S., Paxson, V.: Difficulties in simulating the internet. IEEE/ACM Transaction on Networking (Feburary 2001) 4. ITU-T: Video coding for low bitrate communication. Tech. rep., ITU-T Recommendation H.263, version 2 (January 1998) 5. JTC1, I.: Coding of audio-visual objects - part 2: Visual. Tech. rep., ISO/IEC 14496-2 (MPEG-4 Visual Version 1) (April 1999) 6. Zhang, L., Deering, S., Estrin, D., et al.: Rsvp: A new resource reservation protocol. IEEE Network 5, 8–18 (1993) 7. Paxson, V.: End-to-end internet packet dynamics. In: ACM SIGCOMM 1997, Cannes, France (April 1997) 8. Floyd, S., Fall, K.: Promoting the use of end-to-end congestion control in the internet. IEEE/ACM Trans. on Networking 7(4), 458–472 (1999) 9. Raghavan, S., Tripathi, S., Prabhakaran, B.: On qos parameters and services for multimedia applications. Tech. Rep. 3167, Dept. of Computer Science, Univ. of Maryland, MD (1994) 10. Varadarajan, S., Ngo, H., Srivastava, J.: Error spreading: A perception-driven approach to handling error in continuous media streaming. IEEE/ACM Trans. on Networking 10(1), 139– 152 (2002) 11. Silberschatz, A., Galvin, P.B.: The Readers and Writers Problem. In: Operating System Concepts, 5th edn., pp. 173–175. Addison Wesley Longman, Inc., Amsterdam (1998) 12. Wiegand, T., Sullivan, G.J., Bjontegaard, G., Luthra, A.: Overview of the h.264/avc video coding standard. IEEE Trans. on Circuits and Systems for Video Technology 13(7) (July 2003) 13. Wenger, S.: H.264/avc over ip. IEEE Trans. on Circuits and Systems for Video Technology 13(7), 645–656 (2003)

Scalable Distributed Simulation of Large Dense Crowds Using the Real-Time Framework (RTF) Ole Scharf, Sergei Gorlatch, Felix Blanke, Christoph Hemker, Sebastian Westerheide, Tobias Priebs, Christoph Bartenhagen, Alexander Ploss, Frank Glinka, and Dominik Meilaender University of M¨ unster, Germany [email protected]

Abstract. The simulation of large groups (crowds) of individuals is a complex and challenging task. It requires creating an adequate model that takes into account psychological features of individuals, as well as developing and implementing efficient computation and communication strategies in order to manage the immense computational workload implied by the numerous interactions within a crowd. This paper develops a novel model for a realistic, real-time simulation of large and dense crowds, focusing on evacuation scenarios and the modeling of panic situations. Our approach ensures that both global navigation and local motion are modeled close to reality, and the user can flexibly change both the simulation environment and parameters at runtime. Because of the high computation intensity of the model, we implement the simulation in a distributed manner on multiple server machines, using the RTF (Real-Time Framework) middleware. We implement state replication as an alternative to the traditional state distribution via zoning. We show that RTF enables a high-level development of distributed simulations, and supports an efficient runtime execution on multiple servers.

1

Introduction

The simulation of the behavior of large and dense human crowds is a socially important and technologically challenging task. If attempted with real people, the re-enactment of a scenario such as the evacuation of a fully occupied sports arena poses substantial problems: (1) the costs of involving several thousand people can be enormous; (2) the safety of persons participating in the re-enactment of a potentially life-threatening situation can not be fully guaranteed; (3) simulations involving panics cannot be reproduced at all, since people behave differently when they are aware that the danger is not real; and (4) the inflexibility of reenactments limits the usability: parameters cannot easily be changed, planning is extensive, change of environment is close to impossible. A computer-based simulation overcomes these drawbacks: sophisticated scenarios and large crowds can be set up in reasonable time, simulations can be changed at runtime and repeated with many variations of parameter sets. P. D’Ambra, M. Guarracino, and D. Talia (Eds.): Euro-Par 2010, Part I, LNCS 6271, pp. 572–583, 2010. c Springer-Verlag Berlin Heidelberg 2010

Simulation of Crowds Using the Real-Time Framework (RTF)

573

Crowd modeling and simulation is a comparatively young and vigorously studied research area. Purely mathematical approaches or analytic models are not adequate in characterizing the dynamics of a crowd, because it is not simply a collection of individuals, but may exhibit highly complex dynamics due to social and psychological factors. To represent the behavior of a crowd, different kinds of models have been proposed: flow-based, entity-based, and agent-based models. This paper enhances the agent-based approach as the most accurate one due to the cognitive and reasoning capabilities embedded into the model, in contrast to flow-based models that ignore the individual properties altogether and to entitybased models that express them only in a restricted manner. However, accuracy comes at a price: agent-based models are very time- and space-consuming, thus posing a challenge and an opportunity for parallel and distributed computing. We develop a novel model for computer-based simulations that reproduces the motion of a crowd of individuals by a combination of psychological and geometrical rules with social and physical forces. We describe a new parallelization approach based on state replication rather than on traditional zoning and implement this approach using the Real-Time Framework (RTF) [5] developed at the University of M¨ unster In Section 2, we extend the recent approach [9,12] in which persons in a crowd are regarded as individual entities, so-called agents. We describe how the simulation is distributed among several machines in Section 3 by using a high-level approach based on RTF. We outline our simulation environment and its graphical user interface for interactively changing the simulation at runtime. Section 4 reports first experimental results that demonstrate the scalability of the approach.

2

The Model for Crowd Simulation

In developing our model for crowd simulation, we aim at the following properties: (a) capability to simulate the motion of and within large crowds, in particular dense crowds which pose special challenges; (b) high flexibility in the creation of scenarios and complex surrounding areas; (c) possibility to manipulate the simulation interactively at runtime; (d) no inherent design elements in the model that could prevent the distribution or parallelization of computations. Our approach is based on the HiDAC (High-Density Autonomous Crowds) model [9,12] which is a multi-agent system without a central controlling unit. Each agent corresponds to a simulated person with its own individual behavior. As illustrated in Figure 1, crowd behavior is computed on two levels: 1. The high-level module simulates behavior from a global perspective. Based on the environment information, the agent navigation within its surroundings is computed and passed on in form of waypoints (intermediate targets), represented by coordinates in virtual space, to the low-level module.

574

O. Scharf et al.

2. The low-level module determines the agent’s perception of its surroundings and derives the necessary motion to reach its current waypoint. Both modules take into account an agent’s physiological and psychological status while at the same time updating it. We made several modifications to the original HiDAC architecture, which are described in the following sections.

Fig. 1. Modified HiDAC architecture

2.1

The High-Level Module of the Model

We use a Cell-Portal-Graph (CPG) to represent the interrelation between rooms and doors of a building used by the crowd as cells and portals correspondingly. Rooms carry the information about enclosing walls and agents inside. Nodes represent rooms, edges represent their connections, the portals. In the indoor scenarios, the latter are commonly doors or bottlenecks, therefore inducing the generalized term ’door’ for that kind of connection. Our CPG implementation allows for multiple edges between any two nodes. Figure 2 shows an example translation of an indoor scenario with eight rooms and ten doors into a CPG. Agents pursue a global goal, e.g., leaving the building, by following a sequence of waypoints at those doors that lead to the exit. Agents react dynamically to changes in their environment (e.g., if a door is interactively closed during simulation) and can select alternative routes. As an extension to HiDAC, we implemented the capability to subdivide overlarge simulated rooms on graph level. This allows us to balance the CPG, thus increasing the performance and facilitating the modeling of route-selection preferences. By permitting arbitrary orientation and quantities of walls, we enable a flexible, unrestricted modeling


(a) A building with rooms and doors.

575

(b) The model of the building as CPG.

Fig. 2. Translation of a complex building scenario into a CPG

of the surroundings and an improved waypoint system, empowering agents to navigate even in very complex environments. We use the Boost Graph Library (BGL) [3] for implementing the CPG, building upon the library’s class adjacency list which offers methods for accessing specified nodes or edges. The room-related data of nodes is held in a std::vector list, the door-related data of the edges is held in a std::multiset, thus allowing direct access to looked-up edges between known nodes by implicit sorting. Pathfinding. If the surroundings are known, the agent’s pathfinding is reduced to finding the shortest path from its starting room to its target room, i.e. between the corresponding nodes in the graph. Finding this path is strongly influenced by the weighted edges in our graph implementation: Crowding agents in front of a door add to the weight of the associated edge in the graph. As each agent increases the weight by the amount of space it requires (in our model, this is represented by a circular area defined by the agent’s individual radius), the weight of an edge can thus be understood as a measure of the occupied space in front of the door. With da denoting the shortest distance of an agent to the door, w the current weight, and ra the agent’s radius, the agent increases the weight by 2 · ra , if da < w + ra . An agent is excluded from the weight calculation if it has passed through the door or decides to follow an alternative route. Utilizing the Dijkstra’s algorithm (dijkstra shortest paths in BGL), a list of N nodes and corresponding rooms is generated which the agent has to traverse on its way. In O(|N | · log |N |) time, this algorithm yields a minimal path w.r.t. the weighted sum of its edges. Alternative routes. Two kinds of events can trigger the necessity to reconsider the currently followed path of an agent: A door it was going to pass through turns out to be closed or jammed by other agents. Encountered closed doors in immediate perception range, as seen in Figure 3, are kept in the agent’s memory and trigger a re-run of pathfinding. When an agent wishes to pass through an open, though jammed door, this agent’s impatience is the decisive factor.

576

O. Scharf et al.

Fig. 3. Agents take alternative routes because of a closed (left) or jammed door (right)

Collision Detection and Assisting/Local Waypoints. Our model improves upon HiDAC by circumnavigating the hindering parts of a room’s geometry obstructing an agent’s way. To do so, we introduce assisting waypoints, dynamically set within a room itself, in contrast to the waypoints used in the global navigation. An assisting local waypoint is derived in order to guide the agent around the obstacle, taking into account all walls an agent is currently perceiving. The derivation is illustrated in Figure 4. Let nw be the normal vector, ow the orientation vector, S and E the starting point and the endpoint of a wall, correspondingly. Moreover, let T be the agent’s current (global or local) target, P its position, dw the shortest distance to the wall and L a potential local waypoint at one end of the wall. Then a sequence of tests is performed (e.g., on which side of the wall is the agent positioned, is there an intersection between the direct path and the wall, is this the closest wall, etc.) which allow the agent to keep its motion towards the overall target, e.g., an exit of the building.

Fig. 4. Boundaries of agent’s perception and assisting waypoint L


2.2

577

The Low-Level Module: Perception and Motion

While the high-level module is responsible for global navigation and determining waypoints, the low-level module deals with the simulation of perception and local motion of an agent, as illustrated in Figure 1. Our implementation extends and modifies the original HiDAC [9] as follows. Local motion is based on a combination of psychological, physiological and geometrical rules with physical forces. Agent movement derives from the weighted sum of these forces, in particular representing the agent’s motion-related goals: reaching its current waypoint, avoiding collision with other agents, obstacles and walls, collision handling, and assuming a certain motion inertia towards maintaining its current direction of movement. Agents possess an individual perception range which is of significance for the computational distribution over a network: collision avoidance takes place for agents, obstacles and walls within this perception range only. Collision handling is enforced regardless of the perception range. While collision avoidance is only applied to the room the agent is currently residing in, collision handling additionally accounts for any agents from other rooms passing through connecting doors. Thereby, we prevent unnatural oscillating behavior of agents in crowded spaces. Furthermore, the most important behavioral patterns are simulated: queuing, pushing through crowds, falling down, panicking with panic propagation, and anticipatory agents considering crowd density by means of dynamically adjusting their perception range. On top of our own implementation of structural elements described in [9], we introduced a more situation-aware, intelligent collision avoidance, as well as modifications regarding the avoidance of walls. Another novel feature is that agents look ahead through doors into other rooms when accounting for elements to avoid. The handling of collisions has also been modified to improve agent behavior in collision loops (when agents repeat the same steps again and again, that might eventually lead to a collision).

3

Distributed Multi-server Implementation

High-performance simulation is a prerequisite for a broad class of immersive applications where the user participates in the simulation from the first person perspective. We achieve this by employing a multi-server distribution approach and by providing a comfortable interface for interactions. For distributing the computations over a network of computers, we use the Real-Time Framework (RTF) [5]. The RTF middleware was originally designed to support a high-level development of Internet-based interactive applications like multi-player online computer games. In our simulation system, RTF is responsible for efficient parallel computations, communication and synchronization. 3.1

Serialization and Area of Interest Management

The distribution of tasks between the servers and clients in our system is as follows: a client is processing user interactions, it relates them to the simulation,

578

O. Scharf et al.

and renders a 3D visualization, while the computation of the actual simulation state takes place in a distributed manner over multiple servers. This task separation and distributed computations require a communication interface for transferring data between clients and servers throughout the system. The Real-Time Framework offers an efficient and comfortable means to implement the data transfer: in particular, it supports automatic serialization of objects, i.e. transforming them into a a network-transmittable form. User-defined serialization is offered optionally. Area of Interest (AoI) Management describes the process of distinguishing between relevant and irrelevant information within the simulation process. We apply the AoI concept of RTF [5] on both client- and server-side, dismissing some unnecessary data transfers in order to reduce network traffic. The cost of a potential transfer is weighed against its benefit: (a) information is rated valuable for a client if it makes a noticeable difference to the visualization; (b) information is rated valuable for a server if it supports a consistent simulation state throughout the distributed system. Such AoI management allows us to limit visualization updates transmitted over the network to only those agents which reside inside the user’s field of view, thus saving resources on the servers and the network bandwidth. 3.2

Distribution: Zoning and Replication

The intuitive technique traditionally used in many distributed applications is ’zoning’: the environment is split into disjoint zones, in which computations are handled by different servers, see Fig. 5 (left). When a moving entity (agent or avatar) leaves a zone and enters another zone, this changes the server’s responsibility for this entity. For crowd simulations, the ’zoning’ approach has several drawbacks. First, agent interaction over zone borders is prevented, since information is exclusively available only to one responsible server. Thus, an agent cannot make a decision based on observing other remote agents, which is often necessary in practical scenarios. Moreover, when simulating dense crowds, we cannot distribute the

Fig. 5. Concepts of ’zoning’ (left) and ’replication’ (right)


579

computational workload where it is especially needed: zone borders can only be placed in sparsely populated areas, thus eventually leaving the simulation of a very densely populated area to one server. Finally, strict separation of data among servers requires the client, responsible for visualization, to communicate frequently with every single server in order to render a complete picture of the simulation state. The novelty of this paper is to explore the use of ’replication’ rather than ’zoning’ for computation distribution. Replication means that each server holds the complete simulation data, see Fig. 5 (right). Each server is computing updates only for its so-called active agents; all other agents are called shadowed on this server, and their updates are computed by other servers (every agent is active on exactly one server) and received from them. This allows us to distribute the workload evenly between servers, even in densely crowded scenarios, without hindering agent interaction as with ’zoning’. Additionally, a client now only needs to connect to one server to receive a complete picture of the simulation state for visualization. Replication in our implementation is implemented using RTF which supports both replication and zoning and advanced combinations thereof (the latter will be studied in our future work). The simulation environment is described on a high level of abstraction in an RTF-specific ’map’ which determines the distribution of geometrical space on available servers. Our current system employs a single area replicated over the network: each server comes with its own HiDAC unit. Using mechanisms offered by RTF, agents can be added to a unit, removed from it, and migrated to a different unit at runtime. Agents, doors and obstacles are initialized on one server and subsequently replicated on the others. For example, a newly created obstacle on server A, at first only exists inside the HiDAC unit on server A and inside its RTF Communication and Computation Parallelization (CCP) Module. RTF automatically recreates this obstacle in the CCP Module of server B, which is then to be inserted into server B’s HiDAC unit in the final step of the replication process. RTF manages replication across all participating servers, such that eventually each server keeps an instance of the modeled obstacle. In the terminology of RTF, the obstacle is ’active’ on server A, whereas it is ’shadowed’ on all other servers. This means that server A has sole authority for manipulating this obstacle’s state, while the remaining servers receive updates from A on state changes. An essential feature of replication is that the state of all agents must be known to all servers. Thus, servers need to update the state of their active entities on all other servers. These state updates are automatically performed by RTF and comprise three main steps: 1) serialize agent states and prepare messages, 2) transmit messages over the network, and 3) de-serialize agent states. For a particular server, the complexity of these steps depends linearly on two factors: the total number of agents and the number of servers. To reduce the overhead of replication, RTF employs several optimizations. The serialization of active agent states into a network-transmittable form is performed exactly once per tick for each modified agent. To minimize the number of network packets, RTF aggregates multiple agent updates to a single packet depending on the maximum

580

O. Scharf et al.

transfer unit of the underlying network. RTF decouples de-/ serialization of state updates and actual transmission over the network: while the de-/ serialization of agent states is synchronized with the simulation process in order to prevent concurrency issues, the actual network transmission is handled by RTF asynchronously. Hence, the transmission usually does not add up to the processing time of the application. The amount of transmitted messages could be further reduced using multicast communication which, however, is not always available in distributed systems operated over the Internet. 3.3

Visualization

We implement the crowd simulation as a real-time system with an intuitive 3D representation of the current simulation state which facilitates the evaluation of the simulation state and results by the user. The users can interactively change a running or a paused simulation and manipulate simulation parameters. The parameters currently span 18 different agent-specific attributes, ranging from simple radius and maximum velocity to more advanced right preference angle and attractor weight modifiers. The fine-grain control over every single simulated entity is one of the main advantages of agent-based simulations, and it is made readily available in the user interface of our system.

Fig. 6. Example of changing the simulation parameters at runtime

In our implementation, we used the following free software: OGRE (ObjectOriented Graphics Rendering Engine, [10]) for three-dimensional visualization, CEGUI (Crazy Eddie’s GUI, [4]) as graphical user interface, and OIS (Object Oriented Input System, [11]) for handling mouse and keyboard. Also, we employ Art of Illusion 3D-modeling software [1] for scenario-ready representations of environmental surroundings, with arbitrary orientation/quantities of walls.

4


In order to assess the performance of our simulation system, we conducted a series of tests in a high-load setup which emphasized those elements of a simulation


581

scenario that lead to bottlenecks in the system’s performance. We studied a complex indoor environment with many rooms and one large, unobstructed area, which is much more challenging than the simpler scenarios studied previously. While agents hidden from another agent’s sight can be disregarded in many calculations, open space takes away this potential performance gain. Also, our testing scenario’s setup ensures permanent agent movement because this induces additional computational workload. E.g., a scenario with 400 stationary agents might require less computing power than a scenario with 200 moving agents. Measurements were conducted on a local network of common desktop PCs (servers) at the University of Muenster, with identical hardware characteristics: CPU: Intel(R) Core(TM) 2 Duo CPU E6750 @ 2.66 GHz; Memory: 4 GB; Network connection: 100 Mbit / half duplex. The maximum number of such simultaneously used servers in our experiments was 12. The measured value in the experiments is the rate of simulation frames per second (fps) successfully calculated on a server. By always choosing the value of the weakest system-wide server as the indicator for overall performance, we again followed the high-load approach. Measurements were done in the following manner: First, the server environment was prepared, comprising 1, 2, 4, 6, 8, 10, or 12 servers, with a fair assignment of agents to servers. Then, the test scenario was populated with 20 agents. After 1 minute runtime, the server simulation speed was measured. Subsequently, all agents were removed, the simulation then again was populated with 40 to 400 agents, with step 20 agents, and measurements were taken again after 1 minute.

Fig. 7. Measurements of the simulation speed on 1 to 10 servers

582

O. Scharf et al.

Our series of tests with the specifically designed evacuation scenario for the St. Paulus Cathedral in Muenster (a medieval building of about 5000 sqm with a complex system of doors) produce the results shown in Figure 7. We observe that an increase in the number of servers allows for the simulation of more agents, or, at a fixed number of agents, increases the rate of simulation in fps. A value of 10 fps is an empirically found threshold to ensure correct calculations in our implementation: rate

Euro-Par 2010 - Parallel Processing: 16th International Euro-Par Conference, Ischia, Italy, August 31 - September 3, 2010, Proceedings, Part I ... Computer Science and General Issues)

Euro-Par 2010 - Parallel Processing: 16th International Euro-Par Conference, Ischia, Italy, August 31 - September 3, 2010, Proceedings, Part II ... Computer Science and General Issues)

CONCUR 2010 - Concurrency Theory: 21th International Conference, CONCUR 2010, Paris, France, August 31-September 3, 2010, Proceedings (Lecture Notes ... Computer Science and General Issues)

Fun with Algorithms: 5th International Conference, FUN 2010, Ischia, Italy, June 2-4, 2010, Proceedings (Lecture Notes in Computer Science Theoretical Computer Science and General Issues)

Algorithms and Architectures for Parallel Processing: 10th International Conference, ICA3PP 2010, Busan, Korea, May 21-23, 2010. Proceedings, Part I ... Computer Science and General Issues)

Artificial Neural Networks - ICANN 2010: 20th International Conference, Thessaloniki, Greece, September 15-18, 2010, Proceedings, Part I (Lecture ... Computer Science and General Issues)

Parallel Problem Solving from Nature, PPSN XI: 11th International Conference, Krakov, Poland, September 11-15, 2010, Proceedings, Part I (Lecture Notes ... Computer Science and General Issues)

Algorithms and Architectures for Parallel Processing: 10th International Conference, ICA3PP 2010, Busan, Korea, May 21-23, 2010. Workshops, Part II ... Computer Science and General Issues)

Network and Parallel Computing: IFIP International Conference, NPC 2010, Zhengzhou, China, September 13-15, 2010, Proceedings (Lecture Notes in ... Computer Science and General Issues)

Mathematical Foundations of Computer Science 2010: 35th International Symposium, MFCS 2010, Brno, Czech Republic, August 23-27, 2010, Proceedings ... Computer Science and General Issues)

Artificial Neural Networks - ICANN 2010: 20th International Conference, Thessaloniki, Greece, September 15-18, 2010, Proceedings, Part III (Lecture ... Computer Science and General Issues)

Parallel Problem Solving from Nature, PPSN XI: 11th International Conference, Krakov, Poland, September 11-15, 2010, Proceedings, Part II (Lecture Notes ... Computer Science and General Issues)

Advances in Swarm Intelligence: First International Conference, ICSI 2010, Beijing, China, June 12-15, 2010, Proceedings, Part I (Lecture Notes in ... Computer Science and General Issues)

Entertainment Computing - ICEC 2010: 9th International Conference, ICEC 2010, Seoul, Korea, September 8-11, 2010. Proceedings

Computing and Combinatorics: 16th Annual International Conference, COCOON 2010, Nha Trang, Vietnam, July 19-21, 2010 Proceedings (Lecture Notes in ... Computer Science and General Issues)

Euro-Par '99 Parallel Processing: 5th International Euro-Par Conference, Toulouse, France, August 31 - September 3, 1999, Proceedings: Fifth

Popular Science September 2010

Reachability Problems: 4th International Workshop, RP 2010, Brno, Czech Republic, August 28-29, 2010. Proceedings (Lecture Notes in Computer Science Theoretical Computer Science and General Issues)

Algorithms and Complexity: 7th International Conference, CIAC 2010, Rome, Italy, May 26-28, 2010, Proceedings (Lecture Notes in Computer Science)

Algorithms -- ESA 2010, Part II: 18th Annual European Symposium, Liverpool, UK, September 6-8, 2010, Proceedings (Lecture Notes in Computer Science Theoretical Computer Science and General Issues)

Wireless Algorithms, Systems, and Applications: 5th International Conference, WASA 2010, Beijing, China, August 15-17, 2010. Proceedings (Lecture ... Computer Science and General Issues)

Computer Science Logic: 24th International Workshop, CSL 2010, 19th Annual Conference of the EACSL, Brno, Czech Republic, August 23-27, 2010, ... Computer Science and General Issues)

High Performance Embedded Architectures and Compilers: 5th International Conference, HiPEAC 2010, Pisa, Italy, January 25-27, 2010, Proceedings ... Computer Science and General Issues)

Latent Variable Analysis and Signal Separation: 9th International Conference, LVA ICA 2010, St. Malo, France, September 27-30, 2010, Proceedings ... Computer Science and General Issues)

Sequences and Their Applications - SETA 2010: 6th International Conference, Paris, France, September 13-17, 2010. Proceedings (Lecture Notes in ... Computer Science and General Issues)

Frontiers in Algorithmics: 4th International Workshop, FAW 2010, Wuhan, China, August 11-13, 2010, Proceedings (Lecture Notes in Computer Science Theoretical Computer Science and General Issues)

Computer Vision - ACCV 2010, Part I

Advances in Geometric Modeling and Processing: 6th International Conference, GMP 2010, Castro Urdiales, Spain, June 16-18, 2010, Proceedings (Lecture ... Computer Science and General Issues)

Formal Modeling and Analysis of Timed Systems: 8th International Conference, FORMATS 2010, Klosterneuburg, Austria, September 8-10, 2010, Proceedings ... Computer Science and General Issues)

Parallel Processing and Applied Mathematics, Part II: 8th International Conference, PPAM 2009, Wroclaw, Poland, September 13-16, 2009, Proceedings ... Computer Science and General Issues)

Theory and Applications of Satisfiability Testing - SAT 2010: 13th International Conference, SAT 2010, Edinburgh, UK, July 11-14, 2010, Proceedings ... Computer Science and General Issues)

Euro-Par 2010 - Parallel Processing: 16th International Euro-Par Conference, Ischia, Italy, August 31 - September 3, 2010, Proceedings, Part I ... Computer Science and General Issues)

Euro-Par 2010 - Parallel Processing: 16th International Euro-Par Conference, Ischia, Italy, August 31 - September 3, 2010, Proceedings, Part II ... Computer Science and General Issues)

CONCUR 2010 - Concurrency Theory: 21th International Conference, CONCUR 2010, Paris, France, August 31-September 3, 2010, Proceedings (Lecture Notes ... Computer Science and General Issues)

Fun with Algorithms: 5th International Conference, FUN 2010, Ischia, Italy, June 2-4, 2010, Proceedings (Lecture Notes in Computer Science Theoretical Computer Science and General Issues)

Algorithms and Architectures for Parallel Processing: 10th International Conference, ICA3PP 2010, Busan, Korea, May 21-23, 2010. Proceedings, Part I ... Computer Science and General Issues)

Artificial Neural Networks - ICANN 2010: 20th International Conference, Thessaloniki, Greece, September 15-18, 2010, Proceedings, Part I (Lecture ... Computer Science and General Issues)

Parallel Problem Solving from Nature, PPSN XI: 11th International Conference, Krakov, Poland, September 11-15, 2010, Proceedings, Part I (Lecture Notes ... Computer Science and General Issues)

Algorithms and Architectures for Parallel Processing: 10th International Conference, ICA3PP 2010, Busan, Korea, May 21-23, 2010. Workshops, Part II ... Computer Science and General Issues)

Network and Parallel Computing: IFIP International Conference, NPC 2010, Zhengzhou, China, September 13-15, 2010, Proceedings (Lecture Notes in ... Computer Science and General Issues)

Mathematical Foundations of Computer Science 2010: 35th International Symposium, MFCS 2010, Brno, Czech Republic, August 23-27, 2010, Proceedings ... Computer Science and General Issues)

Artificial Neural Networks - ICANN 2010: 20th International Conference, Thessaloniki, Greece, September 15-18, 2010, Proceedings, Part III (Lecture ... Computer Science and General Issues)

Parallel Problem Solving from Nature, PPSN XI: 11th International Conference, Krakov, Poland, September 11-15, 2010, Proceedings, Part II (Lecture Notes ... Computer Science and General Issues)

Advances in Swarm Intelligence: First International Conference, ICSI 2010, Beijing, China, June 12-15, 2010, Proceedings, Part I (Lecture Notes in ... Computer Science and General Issues)

Entertainment Computing - ICEC 2010: 9th International Conference, ICEC 2010, Seoul, Korea, September 8-11, 2010. Proceedings

Computing and Combinatorics: 16th Annual International Conference, COCOON 2010, Nha Trang, Vietnam, July 19-21, 2010 Proceedings (Lecture Notes in ... Computer Science and General Issues)

Euro-Par '99 Parallel Processing: 5th International Euro-Par Conference, Toulouse, France, August 31 - September 3, 1999, Proceedings: Fifth

Popular Science September 2010

Reachability Problems: 4th International Workshop, RP 2010, Brno, Czech Republic, August 28-29, 2010. Proceedings (Lecture Notes in Computer Science Theoretical Computer Science and General Issues)

Algorithms and Complexity: 7th International Conference, CIAC 2010, Rome, Italy, May 26-28, 2010, Proceedings (Lecture Notes in Computer Science)

Algorithms -- ESA 2010, Part II: 18th Annual European Symposium, Liverpool, UK, September 6-8, 2010, Proceedings (Lecture Notes in Computer Science Theoretical Computer Science and General Issues)

Wireless Algorithms, Systems, and Applications: 5th International Conference, WASA 2010, Beijing, China, August 15-17, 2010. Proceedings (Lecture ... Computer Science and General Issues)

Computer Science Logic: 24th International Workshop, CSL 2010, 19th Annual Conference of the EACSL, Brno, Czech Republic, August 23-27, 2010, ... Computer Science and General Issues)

High Performance Embedded Architectures and Compilers: 5th International Conference, HiPEAC 2010, Pisa, Italy, January 25-27, 2010, Proceedings ... Computer Science and General Issues)

Latent Variable Analysis and Signal Separation: 9th International Conference, LVA ICA 2010, St. Malo, France, September 27-30, 2010, Proceedings ... Computer Science and General Issues)

Sequences and Their Applications - SETA 2010: 6th International Conference, Paris, France, September 13-17, 2010. Proceedings (Lecture Notes in ... Computer Science and General Issues)

Frontiers in Algorithmics: 4th International Workshop, FAW 2010, Wuhan, China, August 11-13, 2010, Proceedings (Lecture Notes in Computer Science Theoretical Computer Science and General Issues)

Computer Vision - ACCV 2010, Part I

Advances in Geometric Modeling and Processing: 6th International Conference, GMP 2010, Castro Urdiales, Spain, June 16-18, 2010, Proceedings (Lecture ... Computer Science and General Issues)

Formal Modeling and Analysis of Timed Systems: 8th International Conference, FORMATS 2010, Klosterneuburg, Austria, September 8-10, 2010, Proceedings ... Computer Science and General Issues)

Parallel Processing and Applied Mathematics, Part II: 8th International Conference, PPAM 2009, Wroclaw, Poland, September 13-16, 2009, Proceedings ... Computer Science and General Issues)

Theory and Applications of Satisfiability Testing - SAT 2010: 13th International Conference, SAT 2010, Edinburgh, UK, July 11-14, 2010, Proceedings ... Computer Science and General Issues)

Recommend Documents