Lecture Notes in Computer Science Edited by G. Goos, J. Hartmanis, and J. van Leeuwen
2790
3
Berlin Heidelberg New York Hong Kong London Milan Paris Tokyo
Harald Kosch László Böszörményi Hermann Hellwagner (Eds.)
Euro-Par 2003 Parallel Processing 9th International Euro-Par Conference Klagenfurt, Austria, August 26-29, 2003 Proceedings
13
Series Editors Gerhard Goos, Karlsruhe University, Germany Juris Hartmanis, Cornell University, NY, USA Jan van Leeuwen, Utrecht University, The Netherlands Volume Editors Harald Kosch László Böszörményi Hermann Hellwagner University Klagenfurt, Institute for Information Technology Universitätsstr. 65-67, 9020 Klagenfurt, Austria E-mail: {harald.kosch, laszlo, hermann.hellwagner}@itec.uni-klu.ac.at
Cataloging-in-Publication Data applied for A catalog record for this book is available from the Library of Congress Bibliographic information published by Die Deutsche Bibliothek Die Deutsche Bibliothek lists this publication in the Deutsche Nationalbibliographie; detailed bibliographic data is available in the Internet at .
CR Subject Classification (1998): C.1-4, D.1-4, F.1-3, G.1-2, H.2 ISSN 0302-9743 ISBN 3-540-40788-X Springer-Verlag Berlin Heidelberg New York This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, re-use of illustrations, recitation, broadcasting, reproduction on microfilms or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer-Verlag. Violations are liable for prosecution under the German Copyright Law. Springer-Verlag Berlin Heidelberg New York a member of BertelsmannSpringer Science+Business Media GmbH http://www.springer.de © Springer-Verlag Berlin Heidelberg 2003 Printed in Germany Typesetting: Camera-ready by author, data conversion by PTP-Berlin GmbH Printed on acid-free paper SPIN: 10931837 06/3142 543210
Preface
Euro-Par Conference Series The European Conference on Parallel Computing (Euro-Par) is an international conference series dedicated to the promotion and advancement of all aspects of parallel and distributed computing. The major themes fall into the categories of hardware, software, algorithms, and applications. This year, new and interesting topics were introduced, like Peer-to-Peer Computing, Distributed Multimedia Systems, and Mobile and Ubiquitous Computing. For the first time, we organized a Demo Session showing many challenging applications. The general objective of Euro-Par is to provide a forum promoting the development of parallel and distributed computing both as an industrial technique and an academic discipline, extending the frontiers of both the state of the art and the state of the practice. The industrial importance of parallel and distributed computing is supported this year by a special Industrial Session as well as a vendors’ exhibition. This is particularly important as currently parallel and distributed computing is evolving into a globally important technology; the buzzword Grid Computing clearly expresses this move. In addition, the trend to a mobile world is clearly visible in this year’s Euro-Par. The main audience for and participants at Euro-Par are researchers in academic departments, industrial organizations, and government laboratories. EuroPar aims to become the primary choice of such professionals for the presentation of new results in their specific areas. Euro-Par has its own Internet domain with a permanent Web site where the history of the conference series is described: http://www.euro-par.org. The Euro-Par conference series is sponsored by the Association for Computer Machinery (ACM) and the International Federation for Information Processing (IFIP).
Euro-Par 2003 at Klagenfurt, Austria Euro-Par 2003 was organized by the Institute of Information Technology, University of Klagenfurt, Austria. The conference location was the University of Klagenfurt which provided a convenient and stimulating environment for the presentation and discussion of recent research results. A number of tutorials and invited talks extended the regular scientific program. Euro-Par 2003 invited five tutorials: Project JXTA: An Open P2P Platform Architecture (Bernard Traversat, Sun Microsystems); Grid Computing with Jini (Mark Baker, University of Portsmouth, and Zoltan Juhasz, University of Veszprem); Pervasive Computing (Alois Ferscha, University of Linz); Carrier Grade Linux Platforms (Ibrahim Haddad, Ericsson Research); and A Family of
VI
Preface
Multimedia Representation Standards: MPEG-4/7/21 (Fernando Pereira, University of Technology Lisbon, and Hermann Hellwagner, University of Klagenfurt). Invited talks were given by C.A.R. Hoare (Microsoft Research and Oxford University) on The Verifying Compiler ; Jim Miller (Microsoft Research) on Lessons from .NET ; Stefan Dessloch (Kaiserslautern University of Technology) on Databases, Web Services, and Grid Computing; and Henri E. Bal (Vrije Universiteit, Amsterdam) on Ibis: A Java-Based Grid Programming Environment. The first two invited speeches were in common with the co-located Fifth Joint Modular Languages Conference (JMLC 2003), the main track of which took place prior to Euro-Par 2003 at the same venue. The co-location of both conferences motivated us to organize a special “event” in the conference week: a memorial panel and an exhibition in honor of the recently deceased great computer scientists Ole-Johan Dahl, Edsger W. Dijkstra, and Kristen Nygaard. The virtual part of the exhibition has been made available for everybody via the Euro-Par 2003 Web site: http://europar-itec.uni-klu.ac.at/.
Euro-Par 2003 Statistics The format of Euro-Par 2003 followed that of the previous editions of the conference and consisted of a number of topics, each of them monitored by a committee of four members. In this year’s conference, there were 19 topics, four of which were included for the first time: Mobile and Ubiquitous Computing (Topic 15), Distributed Systems and Distributed Multimedia (Topic 16), Peer-to-Peer Computing (Topic 18), and a Demo Session (Topic 19) for the presentation of applications. The call for papers attracted 338 submissions, of which 159 were accepted. 103 were selected as regular papers and 52 as research notes. It is worth mentioning that four of the accepted papers were considered to be distinguished papers by the program committee. In total, 1233 review reports were collected, an average of 3.72 per paper. Submissions were received from 43 countries (based on the corresponding author’s country), 29 of which were represented at the conference. The principal contributors by country were USA (25 accepted papers), Germany and Spain (each 21 accepted papers), and France (15 accepted papers).
Acknowledgments A number of institutions and many individuals, in widely different respects, contributed to Euro-Par 2003. We thank for their generous support of the University of Klagenfurt; the Carinthian Economic Fund (KWF); the Carinthian International Campus for Science and Technology (Lakeside Park); the City of Klagenfurt; the Austrian Ministry of Education, Science and Culture (bm:bwk); the Austrian Ministry of Transportation, Innovation and Technology (bmvit); and
Preface
VII
the Austrian Computer Society (OCG). The sponsor companies, Microsoft Research, Hewlett-Packard, Quant-X, Uniquare, IBM, ParTec, Sun Microsystems and the Verein der Freunde der Informatik@University of Klagenfurt provided the financial background required for the organization of a major conference. Finally, we are grateful to Springer-Verlag for publishing this proceedings. We owe special thanks to all the authors for their contributions, members of the topics committee (more than 70 persons), and the numerous reviewers for their excellent work, ensuring the high quality of the conference. We are especially grateful to Christian Lengauer, the chair of the Euro-Par steering committee, who gave us the benefit of his experience in the 18 months leading up to the conference. Last, but not least, we are deeply indebted to the local organization team for their enthusiastic work, especially Martina Steinbacher, Mario D¨ oller, Mulugeta Libsie, Angelika Rossak and the technical staff of our institute. We hope that all participants had a very enjoyable experience here in Klagenfurt, Austria, at Euro-Par 2003! Klagenfurt, June 2003
Harald Kosch L´aszl´o B¨osz¨orm´enyi Hermann Hellwagner
Euro-Par Steering Committee Chair Christian Lengauer University of Passau, Germany Vice Chair Luc Boug´e ENS Cachan, France European Representatives Marco Danelutto University of Pisa, Italy Michel Dayd´e INP Toulouse, France Rainer Feldmann University of Paderborn, Germany Christos Kaklamanis Computer Technology Institute, Greece Paul Kelly Imperial College, London, UK Thomas Ludwig University of Heidelberg, Germany Luc Moreau University of Southampton, UK Rizos Sakellariou University of Manchester, UK Henk Sips Technical University, Delft, The Netherlands Non-European Representatives Jack Dongarra University of Tennessee at Knoxville, USA Shinji Tomita Kyoto University, Japan Honorary Members Ron Perrott Queen’s University Belfast, UK Karl Dieter Reinartz University of Erlangen-Nuremberg, Germany
Euro-Par 2003 Local Organization Euro-Par 2003 was organized by the University of Klagenfurt. Conference Chair Harald Kosch L´ aszl´o B¨osz¨orm´enyi Hermann Hellwagner Committee Martina Steinbacher Angelika Rossak Peter Schojer
Mario D¨ oller Andreas Griesser Remigiusz G´ orecki in Topic 8
Mulugeta Libsie Ronald Sowa
Organization
Euro-Par 2003 Programme Committee Topic 1: Support Tools and Environments Global Chair Helmar Burkhart Local Chair Thomas Ludwig Vice Chairs Rudolf Eigenmann Tom`as Margalef
Institut f¨ ur Informatik, University of Basel, Switzerland Institut f¨ ur Informatik, Ruprecht-KarlsUniversit¨ at, Heidelberg, Germany School of Electrical and Computer Engineering, Purdue University, USA Computer Science Department, Universitat Aut´ onoma de Barcelona, Spain
Topic 2: Performance Evaluation and Prediction Global Chair Jeff Hollingsworth Local Chair Thomas Fahringer Vice Chairs Allen D. Malony Jes´ us Labarta
Computer Science Department, University of Maryland, USA Institute for Software Science, University of Vienna, Austria Department of Computer and Information Science, University of Oregon, USA European Center for Parallelism of Barcelona, Technical University of Catalonia, Spain
Topic 3: Scheduling and Load Balancing Global Chair Yves Robert Local Chair Dieter Kranzlm¨ uller Vice Chairs A.J.C. van Gemund Henri Casanova
Lab. de l’Informatique du Parall´elisme, ENS Lyon, France GUP Linz, Johannes Kepler University, Linz, Austria Delft University of Technology, The Netherlands San Diego Supercomputing Center, USA
IX
X
Organization
Topic 4: Compilers for High Performance Global Chair Michael Gerndt Local Chair Markus Schordan Vice Chairs Chau-Wen Tseng Michael O’Boyle
Institut f¨ ur Informatik, Technische Universit¨at M¨ unchen, Germany Lawrence Livermore National Laboratory, Livermore, USA University of Maryland, College Park, USA University of Edinburgh, UK
Topic 5: Parallel and Distributed Databases, Data Mining and Knowledge Discovery Global Chair Bernhard Mitschang Local Chair Domenico Talia Vice Chairs David Skillicorn Philippe Bonnet
Institute of Parallel and Distributed Systems, Universit¨at Stuttgart, Germany Dipartimento di Elettronica Informatica e Sistemistica, University of Calabria, Italy Queen’s University, Kingston, Canada Datalogisk Institut, Københavns Universitet, Denmark
Topic 6: Grid Computing and Middleware Systems Global Chair Henri Bal Local Chair Peter Kacsuk Vice Chairs Domenico LaForenza
Thierry Priol
Department of Mathematics and Computer Science, Vrije Universiteit, The Netherlands Computer and Automation Research Institute, Hungarian Academy of Sciences, Budapest, Hungary Information Science and Technologies Institute, Italian National Research Council (CNR), Pisa, Italy INRIA Rennes Research Unit, France
Organization
Topic 7: Applications on High-Performance Computers Global Chair Jacek Kitowski Local Chair Peter Luksch Vice Chairs Boleslaw K. Szymanski Andrzej M. Goscinski
Institute of Computer Science and ACC CYFRONET UMM, University of Mining and Metallurgy, Cracow, Poland Institut f¨ ur Informatik, Technische Universit¨ at M¨ unchen, Germany Department of Computer Science, Rensselaer Polytechnic Institute, USA School of Information Technology, Deakin University, Australia
Topic 8: Parallel Computer Architecture and Instruction Level Parallelism Global Chair Stamatis Vassiliadis Local Chair Arndt Bode Vice Chairs Nikitas J. Dimopoulos Jean-Fran¸cois Collard
Computer Engineering Laboratory, Delft University of Technology, The Netherlands Institut f¨ ur Informatik, Technische Universit¨ at M¨ unchen, Germany Electrical and Computer Engineering, University of Victoria, Canada HP Labs 3U, Hewlett-Packard, US
Topic 9: Distributed Algorithms Global Chair Jayadev Misra Local Chair Laurent Lef`evre Vice Chairs Wolfgang Reisig Michael Sch¨ ottner
Department of Computer Sciences, University of Texas at Austin, USA ´ RESO/LIP, Ecole Normale Sup´erieure de Lyon, France Institut f¨ ur Informatik, HumboldtUniversit¨ a¨at zu Berlin, Germany Abteilung Verteilte Systeme, Universit¨ at Ulm, Germany
XI
XII
Organization
Topic 10: Parallel Programming: Models, Methods and Programming Languages Global Chair Jos´e C. Cunha Local Chair Christoph Herrmann Vice Chairs Marco Danelutto Peter H. Welch
New University of Lisbon, Portugal Universit¨ at Passau, Germany University of Pisa, Italy University of Kent, UK
Topic 11: Numerical Algorithms and Scientific Engineering Problems Global Chair Iain Duff Local Chair Peter Zinterhof Vice Chairs Henk van der Vorst Luc Giraud
Computational Science and Engineering Department, Rutherford Appleton Laboratory, Oxfordshire, UK Department of Scientific Computing, Salzburg University, Austria Mathematical Institute, Utrecht University, The Netherlands CERFACS, Toulouse, France
Topic 12: Architectures and Algorithms for Multimedia Applications Global Chair Ishfaq Ahmad Local Chair Andreas Uhl Vice Chairs Pieter Jonker Bertil Schmidt
Computer Science Department, The Hong Kong University of Science and Technology Department of Scientific Computing, Salzburg University, Austria Department of Applied Physics, Delft University of Technology, The Netherlands School of Computer Engineering, Nanyang Technological University, Singapore
Organization
XIII
Topic 13: Theory and Algorithms for Parallel Computation Global Chair Christos Kaklamanis
Local Chair Michael Kaufmann Vice Chairs Danny Krizanc Pierre Fraigniaud
Computer Technology Institute and Department of Computer Engineering and Informatics, University of Patras, Greece Wilhelm-Schickard-Institut f¨ ur Informatik, Universit¨ at T¨ ubingen, Germany Computer Science Group, Mathematics Department, Wesleyan University, USA Laboratoire de Recherche en Informatique, Universit´e Paris-Sud, France
XIV
Organization
Topic 14: Routing and Communication in Interconnection Networks Global Chair Jos´e Duato Local Chair Hermann Hellwagner Vice Chairs Olav Lysne Timothy Pinkston
Technical University of Valencia, Spain Institute of Information Technology, University of Klagenfurt, Austria Simula Research Lab and University of Oslo, Norway University of Southern California, USA
Topic 15: Mobile and Ubiquitous Computing Global Chair Max M¨ uhlh¨ auser Local Chair Alois Ferscha Vice Chairs Azzedine Boukerche Karin Hummel
FG Telekooperation, TU Darmstadt, Germany Institut f¨ ur Praktische Informatik, Gruppe Software, Johannes Kepler Universit¨ at, Linz, Austria University of Ottawa, Canada Institute for Computer Science and Business Informatics, University of Vienna, Austria
Topic 16: Distributed Systems and Distributed Multimedia Global Chair Fernando Pereira Local Chair L´ aszl´o B¨osz¨orm´enyi
Electrical and Computers Department, Instituto Superior T´ecnico, Lisboa, Portugal Institute of Information Technology, University of Klagenfurt, Austria
Vice Chairs Abdulmotaleb El Saddik School of Information Technology and Engineering (SITE), University of Ottawa, Canada Roy Friedman Department of Computer Science, Technion – Israel Institute of Technology, Haifa, Israel
Organization
Topic 17: High-Performance Object-Oriented and Middleware Systems Global Chair Geoffrey Fox
Community Grids Laboratory, Indiana University, USA
Local Chair Michael Philippsen Institut f¨ ur Informatik, Universit¨ at Erlangen-N¨ urnberg, Germany Vice Chairs Mark Bull Edinburgh Parallel Computing Centre (EPCC), University of Edinburgh, UK Andrew Wendelborn Department of Computer Science, University of Adelaide, Australia
Topic 18: Peer-to-Peer Computing Global Chair Luc Boug´e Local Chair Franck Cappello Vice Chairs Bernard Traversat Omer Rana
IRISA, ENS Cachan, Brittany Extension, Rennes, France CNRS, LRI-Universit´e Paris-Sud, France Project JXTA, Sun Microsystems, Santa Clara, USA Department of Computer Science, Cardiff University, UK
Topic 19: Demonstrations of Parallel and Distributed Computing Global Chair Ron Perrott
School of Computer Science, Queen’s University Belfast, UK
Local Chair Michael Kropfberger Institute of Information Technology, University of Klagenfurt, Austria Vice Chairs Henk Sips Faculty of Information Technology and Systems, Technical University of Delft, The Netherlands Jarek Nabrzyski Poznan Supercomputing and Networking Center, Poznan, Poland
XV
XVI
Organization
Euro-Par 2003 Referees (not including members of the programme or organization committees) Afsahi, Ahmad Alda, Witold Aldinucci, M. Alexandru, Jugravu Allcock, Bill Alt, Martin Amodio, Pierluigi Antochi, Iosif Antoniu, Gabriel Armstrong, Brian Ashby, Tom Attiya, Hagit Aumage, Olivier Austaller, Gerhard Balatan, Zoltan Badia, Rosa M. Bahi, Jacques Bajard, Jean-Claude Bancz´ ur, Andr´ as Baniasadi, Amirali Baraglia, Ranieri Barthou, Denis Basumallik, Ayon Baude, Francoise Beaumont, Olivier Beck, Micah Bellosa, Frank Birnbaum, Adam Bischof, Holger Bivens, Alan Boavida, Fernando Bodin, Francois Boudet, Vincent Braun, Elmar Breimer, Eric Breton, Vincent Bretschneider, Timo Rolf Bubak, Marian Buchholz, Peter Buck, Bryan Buyya, Rajkumar
Bystroff, Chris Byun, Tae-Young Caarls, Wouter Cabillic, Gilbert Cafaro, Massimo Cai, Jianfei Cai, Xing Campadello, Stefano Cannataro, Mario Caragiannis, Ioannis Cardinale, Yudith Caromel, Denis Caron, Eddy Carter, Larry Casado, Rafael Catthoor, Francky Chang, Chuan-Hua Chatterjee, Mainak Cheresiz, Dmitry Chiola, Giovanni Chrysos, George Chun, B.N. Chung, I-hsin Cintra, Marcelo Coddington, Paul Cole, Murray Contes, Arnaud Coppola, Massimo Cort´es, Ana Costa, Vitor Santos Cramp, Anthony Crispo, Bruno C´esar, Eduardo Da Costa, Carlos Dail, Holly Dayde, Michel De Castro Dutra, Ines Deelman, Ewa Denis, Alexandre Denneulin, Yves Desprez, Frederic
Organization
Dhaenens, Clarisse Di Cosmo, Roberto Di Serafino, Daniela Dias, Artur Miguel Diessel, Oliver Dimakopoulos, Vassilos Do, Tai Dobrucky, Miroslav Dolev, Shlomi Dou, Jialin Drach-Temam, Nathalie Ducourthial, Bertrand Durr, C. Dutot, Pierre-Francois Dzwinel, Witold Eijkhout, Victor El Khatib, Khalil Ekaterinides, Yannis Emmanuel, S. Espinosa, Antonio Faber, Peter Fabrega, Josep Fagni, Tiziano Falkner, Katrina E. Farcy, Alexandre Feng, W. Ferragina, Paola Ferrante, Jeanne Fink, Torsten Fisher, Steve Fleury, Eric Folino, Gianluigi Ford, Rupert Fowler, Rob Franco, Daniel Franke, Bjoern Frenz, Stefan Frigo, Matteo Frohner, Akos Funika, Wlodzimierz Furfaro, Filippo Fursin, Grigori F¨ urlinger, Karl Gansterer, Wilfried Garrido, Antonio
Gautama, Hasyim Gaydadjiev, G.N. Geist, Al Gelas, Jean-Patrick Getov, Vladimir Geuzebroek, Jeroen Gibbins, Nick Gjessing, Stein Glossner, John Gombas, Gabor Gorlatch, Sergei Goyeneche, Ariel Gratton, Serge Guermouche, Abdou Gupta, Amitava Haase, Gundolf Hammond, Kevin Hartl, Andreas Haumacher, Bernhard Hauswirth, Manfred Heinemann, Andreas Heinrich, Mark A. Hermenegildo, Manuel V. Hern´ andez, Porfidio Heymann, Elisa Hlavacs, Helmut Hluchy, Ladislav Hopkins, T.R. Horn, Geir Hoschek, Wolfgang Hotop, Ewald Houda, Lamehamedi Hu, Zhenjiang Hutchison, David Hyon, Emmanuel Iqbal, M. Ashraf Isaila, Florin Jegou, Yvon Jeitner, J¨ urgen Jin, Hai Johnson, Troy A. Jorba, Josep Jouhaud, Jen-Christophe Jouppi, Norman P. Ju, Roy
XVII
XVIII Organization
Juhasz, Zoltan Juurlink, Ben Kaeli, David Kagstrom, Bo Kalantery, Nasser Karl, Wolfgang Karp, Alan Kat, Ronen Keahey, Kate Kelly, Paul Kereku, Edmond Kesavan, Ram Khunjush, Farshad Kielmann, Thilo Kindermann, Stephan Kleinjung, Jens Kohn, Scott Kondo, Derrick Kotsis, Gabriele Kowarschik, Markus Krishnamurthy, Arvind Kuchen, Herbert Kumar, Sanjeev Kunszt, Peter Kuzmanov, Georgi L’Excellent, Jean-Yves Lagendijk, R. Langer, Ulrich Lanteri, Stephane Lauff, Markus Lavenier, Dominique Layuan, Li Lee, Jack Lisi, Francesca Liu, Jane W.S. Lopez, Pedro Lourenco, Joao Luque, Emilio Luszczek, Piotr Mairandres, Martin Maman, Nathan Manco, Giuseppe Marcos, Aderito Fernandes Markatos, Evangelos Marques, Osni
Marques, Rui Martinaitis, Paul Mastroianni, Carlo Matyska, Ludek Mayrhofer, Rene Mazzia, Francesca McCance, Gavin Medeiros, Pedro Meier, Harald Merzky, Andre Michaelson, Greg Midkiff, Sam Min, Seung Jai Miron, Pavlus Molnos, Anca Monteiro, Edmundo Moreau, Luc Moro, Gianluca Moscu, Elena Moshovos, Andreas Moure, Juan Carlos Muller, Jens Muthukumar, Kalyan Namyst, Raymond Nandy, Sagnik Napolitano, Jim Nawarecki, Edward Newhall, Tia Nieminen, Risto Nikoletseas, Sotiris Nolte, Tina Notare, Mirela Sechi O’Donnell, John Ohsumi, Toshiro Orban, Dominique Orduna, Juan Manuel Orlando, Salvatore Ortega, Julio Ould-Khaoua, Mohamed Overeinder, Benno J. Paar, Alexander Pallickara, Shrideep Palmerini, Paolo Pan, Zhelong Park, Yong Woon
Organization
Peinl, Peter Peng, Jufeng Perego, Raffaele Perez, Christian Petitet, Antoine Petrini, Fabrizio Pham, Congduc Pichler, Mario Pierce, Evelyn Pllana, Sabri Podlipnig, Stefan Poetzsch-Heffter, Arnd Pommer, Andreas Poplavko, Peter Pralet, Stepahe Pramanick, Ira Prodan, Radu Pugliese, Andrea Puliafito, Antonio Quinson, Martin Radulescu, Andrei Rakhmatov, Daler N. Rantzau, Ralf Rathmayer, Sabine Regin, Jean-Charles Reinemo, Sven-Arne Renambot, Luc Resch, Michael Ripoll, Ana Roe, Paul Ruiz, Daniel Saffre, Fabrice Safwat, Ahmed Saha, Debashis Sanders, Beverly Santos, Luis Paulo Sartori, Claudio Sasaki, Galen Schillo, Michael Schimmler, Manfred Schintke, Florian Schlansker, Michael Schojer, Peter Schreiber, Rob Schulz, Martin
Schuster, Assaf Schwarz, Holger Seitz, Christian Senar, Miquel Angel Sens, Pierre Seragiotto, Clovis, Jr. Sethumadhavan, Simha Shankar, Udaya A. Siciliano, Bruno Silva, Luis Moura Silvestri, Fabrizio Sima, Mihai Simpson, Steven Sion, Radu Skeie, Tor Sommeijer, Ben Sorensen, Dan Spriestersbach, Axel Srinivasan, Srikanth T. Stamatakis, Alexandros Stathis, Pyrrhos Stefan, Peter Stiles, Gardiner S. Stricker, Thomas M. Su, Alan Sulistio, Anthony Suppi, Remo Suter, Frederic Szeberenyi, Imre S´erot, Jocelyn Tao, Jie Taylor, Ian Tchernykh, Andrei Teich, J¨ urgen Temam, Olivier Teresco, Jim Terstyanszky, Gabor Theiss, Ingebjorg Thelin Thottethodi, Mithuna Todorova, Petia Tolia, Sovrin Tolksdorf, Robert Tonellotto, Nicola Tran, Viet Trinitis, Carsten
XIX
XX
Organization
Trobec, Roman Trunfio, Paolo Truong, Hong-Linh Tudruj, Marek Turner, S.J. Tusch, Roland Ueberhuber, Christoph Unger, Shelley Utard, Gil Vajtersic, Marian Van Gijzen, Martin Van der Vorst, Henk Varela, Carlos Varga, Laszlo Z. Varshney, Upkar Veldema, Ronald Vivien, Frederic Vogels, Werner Vogl, Simon Volker, Christian Volkert, Jens Von Laszewski, Gregor Walter, Max
Wang, Dajin Wasniewski, Jerzy Weidendorfer, Josef Welzl, Michael Wism¨ uller, Roland Wong, Stephan Woodcock, Jim Wyrzykowski, Roman Xiao, Li Yan, Ken Qing Yang, Yang Yeo, Chai Kiat Yi, Qing Yoo, Chuck Yuksel, Murat Zambonelli, Franco Zhang, Ming Zheng, Yili Zhou, Xiaobo Zoccolo, C. Zottl, Joachim
Table of Contents
Invited Talks The Verifying Compiler: A Grand Challenge for Computing Research . . . . C.A.R. Hoare
1
Evolving a Multi-language Object-Oriented Framework: Lessons from .NET . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jim Miller
2
Databases, Web Services, and Grid Computing – Standards and Directions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Stefan Dessloch
3
Ibis: A Java-Based Grid Programming Environment . . . . . . . . . . . . . . . . . . . Henri E. Bal
4
Topic 1: Support Tools and Environments Topic Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Topic Chairs
5
A Hardware Counters Based Tool for System Monitoring . . . . . . . . . . . . . . . Tiago C. Ferreto, Luiz DeRose, C´esar A.F. De Rose
7
ParaProf: A Portable, Extensible, and Scalable Tool for Parallel Performance Profile Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Robert Bell, Allen D. Malony, Sameer Shende
17
On Utilizing Experiment Data Repository for Performance Analysis of Parallel Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hong-Linh Truong, Thomas Fahringer
27
Flexible Performance Debugging of Parallel and Distributed Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jacques Chassin de Kergommeaux, Cyril Guilloud, B. de Oliveira Stein
38
EventSpace – Exposing and Observing Communication Behavior of Parallel Cluster Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Lars Ailo Bongo, Otto J. Anshus, John Markus Bjørndalen
47
A Race Detection Mechanism Embedded in a Conceptual Model for the Debugging of Message-Passing Distributed Programs . . . . . . . . . . . . . . . . . . Ana Paula Cl´ audio, Jo˜ ao Duarte Cunha
57
XXII
Table of Contents
DIOS++: A Framework for Rule-Based Autonomic Management of Distributed Scientific Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hua Liu, Manish Parashar
66
DeWiz – A Modular Tool Architecture for Parallel Program Analysis . . . . Dieter Kranzlm¨ uller, Michael Scarpa, Jens Volkert
74
Why Not Use a Pattern-Based Parallel Programming System? . . . . . . . . . . John Anvik, Jonathan Schaeffer, Duane Szafron, Kai Tan
81
Topic 2: Performance Evaluation and Prediction Topic Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Topic Chairs
87
Symbolic Performance Prediction of Speculative Parallel Programs . . . . . . Hasyim Gautama, Arjan J.C. van Gemund
88
A Reconfigurable Monitoring System for Large-Scale Network Computing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Rajesh Subramanyan, Jos´e Miguel-Alonso, Jos´e A.B Fortes
98
Obtaining Hardware Performance Metrics for the BlueGene/L Supercomputer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109 Pedro Mindlin, Jos´e R. Brunheroto, Luiz DeRose, Jos´e E. Moreira Presentation and Analysis of Grid Performance Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 Norbert Podhorszki, Peter Kacsuk Distributed Application Monitoring for Clustered SMP Architectures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127 Karl F¨ urlinger, Michael Gerndt An Emulation System for Predicting Master/Slave Program Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 Yasuharu Mizutani, Fumihiko Ino, Kenichi Hagihara POETRIES: Performance Oriented Environment for Transparent Resource-Management, Implementing End-User Parallel/Distributed Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141 Eduardo Cesar, J.G. Mesa, Joan Sorribes, Emilio Luque
Topic 3: Scheduling and Load Balancing Topic Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147 Topic Chairs
Table of Contents
XXIII
Static Load-Balancing Techniques for Iterative Computations on Heterogeneous Clusters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148 H´el`ene Renard, Yves Robert, Fr´ed´eric Vivien Impact of Job Allocation Strategies on Communication-Driven Coscheduling in Clusters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160 Gyu Sang Choi, Saurabh Agarwal, Jin-Ha Kim, Anydy B. Yoo, Chita R. Das Trading Cycles for Information: Using Replication to Schedule Bag-of-Tasks Applications on Computational Grids . . . . . . . . . . . . . . . . . . . . 169 Daniel Paranhos da Silva, Walfredo Cirne, Francisco Vilar Brasileiro Dynamic Load Partitioning Strategies for Managing Data of Space and Time Heterogeneity in Parallel SAMR Applications . . . . . . . . . . . . . . . 181 Xiaolin Li, Manish Parashar An Experimental Investigation into the Rank Function of the Heterogeneous Earliest Finish Time Scheduling Algorithm . . . . . . . . . . 189 Henan Zhao, Rizos Sakellariou Performance-Based Dynamic Scheduling of Hybrid Real-Time Applications on a Cluster of Heterogeneous Workstations . . . . . . . . . . . . . . 195 Ligang He, Stephen A. Jarvis, Daniel P. Spooner, Graham R. Nudd Recursive Refinement of Lower Bounds in the Multiprocessor Scheduling Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201 Satoshi Fujita, Masayuki Masukawa, Shigeaki Tagashira Efficient Dynamic Load Balancing Strategies for Parallel Active Set Optimization Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206 I. Pardines, Francisco F. Rivera Cooperating Coscheduling in a Non-dedicated Cluster . . . . . . . . . . . . . . . . . 212 Francesc Gin´e, Francesc Solsona, Porfidio Hern´ andez, Emilio Luque Predicting the Best Mapping for Efficient Exploitation of Task and Data Parallelism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218 Fernando Guirado, Ana Ripoll, Concepci´ o Roig, Xiao Yuan, Emilio Luque Dynamic Load Balancing for I/O- and Memory-Intensive Workload in Clusters Using a Feedback Control Mechanism . . . . . . . . . . . . . . . . . . . . . . . . 224 Xiao Qin, Hong Jiang, Yifeng Zhu, David R. Swanson
XXIV
Table of Contents
An Experimental Study of k-Splittable Scheduling for DNS-Based Traffic Allocation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230 Amit Agarwal, Tarun Agarwal, Sumit Chopra , Anja Feldmann, Nils Kammenhuber, Piotr Krysta, Berthold V¨ ocking Scheduling Strategies of Divisible Loads in DIN Networks . . . . . . . . . . . . . . 236 Ligang Dong, Lek Heng Ngoh, Joo Geok Tan
Topic 4: Compilers for High Performance Topic Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241 Topic Chairs Partial Redundancy Elimination with Predication Techniques . . . . . . . . . . . 242 Bernhard Scholz, Eduard Mehofer, Nigel Horspool SIMD Vectorization of Straight Line FFT Code . . . . . . . . . . . . . . . . . . . . . . . 251 Stefan Kral, Franz Franchetti, Juergen Lorenz, Christoph W. Ueberhuber Branch Elimination via Multi-variable Condition Merging . . . . . . . . . . . . . . 261 William Kreahling, David Whalley, Mark Bailey, Xin Yuan, Gang-Ryung Uh, Robert van Engelen Exploiting On-Chip Data Transfers for Improving Performance of Chip-Scale Multiprocessors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271 G. Chen, M. Kandemir, I. Kolcu, A. Choudhary An Energy-Oriented Evaluation of Communication Optimizations for Microsensor Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279 I. Kadayif, M. Kandemir, A. Choudhary, M. Karakoy Increasing the Parallelism of Irregular Loops with Dependences . . . . . . . . . 287 David E. Singh, Mar´ıa J. Mart´ın, Francisco F. Rivera Finding Free Schedules for Non-uniform Loops . . . . . . . . . . . . . . . . . . . . . . . . 297 Volodymyr Beletskyy, Krzysztof Siedlecki Replicated Placements in the Polyhedron Model . . . . . . . . . . . . . . . . . . . . . . 303 Peter Faber, Martin Griebl, Christian Lengauer
Topic 5: Parallel and Distributed Databases, Data Mining, and Knowledge Discovery Topic Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 309 Topic Chairs A Parallel Algorithm for Incremental Compact Clustering . . . . . . . . . . . . . . 310 Reynaldo Gil-Garc´ıa, Jos´e M. Bad´ıa-Contelles, Aurora Pons-Porrata
Table of Contents
XXV
Preventive Multi-master Replication in a Cluster of Autonomous Databases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 318 ¨ Esther Pacitti, M. Tamer Ozsu, C´edric Coulon Pushing Down Bit Filters in the Pipelined Execution of Large Queries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 328 Josep Aguilar-Saborit, Victor Munt´es-Mulero, Josep-L. Larriba-Pey Suffix Arrays in Parallel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 338 Mauricio Mar´ın, Gonzalo Navarro Revisiting Join Site Selection in Distributed Database Systems . . . . . . . . . 342 Haiwei Ye, Brigitte Kerherv´e, Gregor v. Bochmann SCINTRA: A Model for Quantifying Inconsistencies in Grid-Organized Sensor Database Systems . . . . . . . . . . . . . . . . . . . . . . . . . . 348 Lutz Schlesinger, Wolfgang Lehner
Topic 6: Grid Computing and Middleware Systems Topic Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 356 Topic Chairs Implementation of a Grid Computation Toolkit for Design Optimisation with Matlab and Condor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 357 Gang Xue, Matthew J. Fairman, Graeme E. Pound, Simon J. Cox Grid Resource Selection for Opportunistic Job Migration . . . . . . . . . . . . . . . 366 Rub´en S. Montero, Eduardo Huedo, Ignacio M. Llorente Semantic Access Control for Medical Applications in Grid Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 374 Ludwig Seitz, Jean-Marc Pierson, Lionel Brunie Automated Negotiation for Grid Notification Services . . . . . . . . . . . . . . . . . . 384 Richard Lawley, Keith Decker, Michael Luck, Terry Payne, Luc Moreau GrADSolve – RPC for High Performance Computing on the Grid . . . . . . . 394 Sathish Vadhiyar, Jack Dongarra, Asim YarKhan Resource and Job Monitoring in the Grid . . . . . . . . . . . . . . . . . . . . . . . . . . . . 404 Zolt´ an Balaton, G´ abor Gomb´ as Delivering Data Management for Engineers on the Grid . . . . . . . . . . . . . . . . 412 Jasmin Wason, Marc Molinari, Zhuoan Jiao, Simon J. Cox A Resource Accounting and Charging System in Condor Environment . . . 417 Csongor Somogyi, Zolt´ an L´ aszl´ o, Imre Szeber´enyi
XXVI
Table of Contents
Secure Web Services with Globus GSI and gSOAP . . . . . . . . . . . . . . . . . . . . 421 Giovanni Aloisio, Massimo Cafaro, Daniele Lezzi, Robert Van Engelen Future-Based RMI: Optimizing Compositions of Remote Method Calls on the Grid . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 427 Martin Alt, Sergei Gorlatch
Topic 7: Applications on High-Performance Computers Topic Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 431 Topic Chairs CAD Grid: Corporate-Wide Resource Sharing for Parameter Studies . . . . 433 Ed Wheelhouse, Carsten Trinitis, Martin Schulz Cache Performance Optimizations for Parallel Lattice Boltzmann Codes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 441 Jens Wilke, Thomas Pohl, Markus Kowarschik, Ulrich R¨ ude Effectiveness of Parallelizing the ILOG-CPLEX Mixed Integer Optimizer in the PUBB2 Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 451 Yuji Shinano, Tetsuya Fujie, Yuusuke Kounoike Improving Performance of Hypermatrix Cholesky Factorization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 461 Jos´e R. Herrero, Juan J. Navarro Parallel Agent-Based Simulation on a Cluster of Workstations . . . . . . . . . . 470 Konstantin Popov, Vladimir Vlassov, Mahmoud Rafea, Fredrik Holmgren, Per Brand, Seif Haridi Low Level Parallelization of Nonlinear Diffusion Filtering Algorithms for Cluster Computing Environments . . . . . . . . . . . . . . . . . . . . . 481 David Slogsnat, Markus Fischer, Andr´es Bruhn, Joachim Weickert, Ulrich Br¨ uning Implementation of Adaptive Control Algorithms in Robot Manipulators Using Parallel Computing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 491 Juan C. Fern´ andez, Vicente Hern´ andez, Lourdes Pe˜ nalver Interactive Ray Tracing on Commodity PC Clusters . . . . . . . . . . . . . . . . . . . 499 Ingo Wald, Carsten Benthin, Andreas Dietrich, Philipp Slusallek Toward Automatic Management of Embarrassingly Parallel Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 509 Inˆes Dutra, David Page, Vitor Santos Costa, Jude Shavlik, Michael Waddell
Table of Contents
XXVII
Comparing Two Long Biological Sequences Using a DSM System . . . . . . . 517 Renata Cristina F. Melo, Maria Em´ılia Telles Walter, Alba Cristina Magalhaes Alves Melo, Rodolfo Batista, Marcelo Nardelli, Thelmo Martins, Tiago Fonseca Two Dimensional Airfoil Optimisation Using CFD in a Grid Computing Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 525 Wenbin Song, Andy Keane, Hakki Eres, Graeme Pound, Simon Cox Applied Grid Computing: Optimisation of Photonic Devices . . . . . . . . . . . . 533 Duan H. Beckett, Ben Hiett, Ken S. Thomas, Simon J. Cox Parallel Linear System Solution and Its Application to Railway Power Network Simulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 537 Muhammet F. Ercan, Yu-fai Fung, Tin-kin Ho, Wai-leung Cheung
Topic 8: Parallel Computer Architecture and Instruction-Level Parallelism Topic Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 541 Topic Chairs An Overview of the Blue Gene/L System Software Organization . . . . . . . . 543 George Alm´ asi, Ralph Bellofatto, Jos´e Brunheroto, C˘ alin Ca¸scaval, Jos´e G. Casta˜ nos, Luis Ceze, Paul Crumley, C. Christopher Erway, Joseph Gagliano, Derek Lieber, Xavier Martorell, Jos´e E. Moreira, Alda Sanomiya, Karin Strauss Trace Substitution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 556 Hans Vandierendonck, Hans Logie, Koen De Bosschere Optimizing a Decoupled Front-End Architecture: The Indexed Fetch Target Buffer (iFTB) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 566 Juan C. Moure, Dolores I. Rexachs, Emilio Luque Clustered Microarchitecture Simultaneous Multithreading . . . . . . . . . . . . . . 576 Seong-Won Lee, Jean-Luc Gaudiot Counteracting Bank Misprediction in Sliced First-Level Caches . . . . . . . . . 586 Enrique F. Torres, P. Iba˜ nez, V. Vi˜ nals, J.M. Llaber´ıa An Enhanced Trace Scheduler for SPARC Processors . . . . . . . . . . . . . . . . . . 597 Spiros Kalogeropulos Compiler-Assisted Thread Level Control Speculation . . . . . . . . . . . . . . . . . . 603 Hideyuki Miura, Luong Dinh Hung, Chitaka Iwama, Daisuke Tashiro, Niko Demus Barli, Shuichi Sakai, Hidehiko Tanaka
XXVIII
Table of Contents
Compression in Data Caches with Compressible Field Isolation for Recursive Data Structures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 609 Masamichi Takagi, Kei Hiraki Value Compression to Reduce Power in Data Caches . . . . . . . . . . . . . . . . . . 616 Carles Aliagas, Carlos Molina, Montse Garcia, Antonio Gonzalez, Jordi Tubella
Topic 9: Distributed Algorithms Topic Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 623 Topic Chairs Multiresolution Watershed Segmentation on a Beowulf Network . . . . . . . . . 624 Syarrraieni Ishar, Michel Bister i RBP – A Fault Tolerant Total Order Broadcast for Large Scale Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 632 Luiz Angelo Barchet-Estefanel Computational Models for Web- and Grid-Based Computation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 640 Joaquim Gabarr´ o, Alan Stewart, Maurice Clint, Eamonn Boyle, Isabel Vallejo CAS-Based Lock-Free Algorithm for Shared Deques . . . . . . . . . . . . . . . . . . . 651 Maged M. Michael Energy Efficient Algorithm for Disconnected Write Operations in Mobile Web Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 661 Jong-Mu Choi, Jin-Seok Choi, Jai-Hoon Kim, Young-Bae Ko Distributed Scheduling of Mobile Priority Requests . . . . . . . . . . . . . . . . . . . . 669 Ahmed Housni, Michel Lacroix, Michel Trehel Parallel Distributed Algorithms of the β-Model of the Small World Graphs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 675 Mahmoud Rafea, Konstantin Popov, Per Brand, Fredrik Holmgren, Seif Haridi
Topic 10: Parallel Programming: Models, Methods, and Programming Languages Topic Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 681 Topic Chairs Cost Optimality and Predictability of Parallel Programming with Skeletons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 682 Holger Bischof, Sergei Gorlatch, Emanuel Kitzelmann
Table of Contents
XXIX
A Methodology for Order-Sensitive Execution of Non-deterministic Languages on Beowulf Platforms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 694 K. Villaverde, E. Pontelli, H-F. Guo, G. Gupta From Complexity Analysis to Performance Analysis . . . . . . . . . . . . . . . . . . . 704 Vicente Blanco, Jes´ us A. Gonz´ alez, Coromoto Le´ on, Casiano Rodr´ıguez, Germ´ an Rodr´ıguez The Implementation of ASSIST, an Environment for Parallel and Distributed Programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 712 Marco Aldinucci, Sonia Campa, Pierpaolo Ciullo, Massimo Coppola, Silvia Magini, Paolo Pesciullesi, Laura Potiti, Roberto Ravazzolo, Massimo Torquati, Marco Vanneschi, Corrado Zoccolo The Design of an API for Strict Multithreading in C++ . . . . . . . . . . . . . . . 722 Wolfgang Blochinger, Wolfgang K¨ uchlin High-Level Process Control in Eden . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 732 Jost Berthold, Ulrike Klusik, Rita Loogen, Steffen Priebe, Nils Weskamp Using Skeletons in a Java-Based Grid System . . . . . . . . . . . . . . . . . . . . . . . . . 742 Martin Alt, Sergei Gorlatch Prototyping Application Models in Concurrent ML . . . . . . . . . . . . . . . . . . . . 750 David Johnston, Martin Fleury, Andy Downton THROOM – Supporting POSIX Multithreaded Binaries on a Cluster . . . . 760 Henrik L¨ of, Zoran Radovi´c, Erik Hagersten An Inter-entry Invocation Selection Mechanism for Concurrent Programming Languages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 770 Aaron W. Keen, Ronald A. Olsson Parallel Juxtaposition for Bulk Synchronous Parallel ML . . . . . . . . . . . . . . . 781 Fr´ed´eric Loulergue Parallelization with Tree Skeletons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 789 Kiminori Matsuzaki, Zhenjiang Hu, Masato Takeichi
Topic 11: Numerical Algorithms and Scientific Engineering Problems Topic Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 799 Topic Chairs Parallel ScaLAPACK-Style Algorithms for Solving Continuous-Time Sylvester Matrix Equations . . . . . . . . . . . . . . . . . . . . . . . . . 800 Robert Granat, Bo K˚ agstr¨ om, Peter Poromaa
XXX
Table of Contents
RECSY – A High Performance Library for Sylvester-Type Matrix Equations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 810 Isak Jonsson, Bo K˚ agstr¨ om Two Level Parallelism in a Stream-Function Model for Global Ocean Circulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 820 Martin van Gijzen Scalable Parallel RK Solvers for ODEs Derived by the Method of Lines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 830 Matthias Korch, Thomas Rauber Hierarchical Hybrid Grids as Basis for Parallel Numerical Solution of PDE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 840 Frank H¨ ulsemann, Benjamin Bergen, Ulrich R¨ ude Overlapping Computation/Communication in the Parallel One-Sided Jacobi Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 844 El Mostafa Daoudi, Abdelhak Lakhouaja, Halima Outada
Topic 12: Architectures and Algorithms for Multimedia Applications Topic Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 850 Topic Chairs Distributed Multimedia Streaming over Peer-to-Peer Networks . . . . . . . . . . 851 Jin B. Kwon, Heon Y. Yeom Exploiting Traffic Balancing and Multicast Efficiency in Distributed Video-on-Demand Architectures . . . . . . . . . . . . . . . . . . . . . . . . . . 859 Fernando Cores, Ana Ripoll, Bahjat Qazzaz, Remo Suppi, Xiaoyuan Yang, Porfidio Hernandez, Emilio Luque On Transmission Scheduling in a Server-Less Video-on-Demand System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 870 C.Y. Chan, Jack Y.B. Lee A Proxy-Based Dynamic Multicasting Policy Using Stream’s Access Pattern . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 880 Yong Woon Park, Si Woong Jang
Topic 13: Theory and Algorithms for Parallel Computation Topic Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 884 Topic Chairs
Table of Contents
XXXI
Improving Communication Sensitive Parallel Radix Sort for Unbalanced Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 885 Martin Schmollinger Minimizing Global Communication in Parallel List Ranking . . . . . . . . . . . . 894 Jop F. Sibeyn Construction of Efficient Communication Sub-structures: Non-approximability Results and Polynomial Sub-cases . . . . . . . . . . . . . . . . 903 Christian Laforest c-Perfect Hashing Schemes for Binary Trees, with Applications to Parallel Memories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 911 Gennaro Cordasco, Alberto Negro, Vittorio Scarano, Arnold L. Rosenberg A Model of Pipelined Mutual Exclusion on Cache-Coherent Multiprocessors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 917 Masaru Takesue Efficient Parallel Multiplication Algorithm for Large Integers . . . . . . . . . . . 923 Viktor Bunimov, Manfred Schimmler
Topic 14: Routing and Communication in Interconnection Networks Topic Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 929 Topic Chairs Dynamic Streams for Efficient Communications between Migrating Processes in a Cluster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 930 Pascal Gallard, Christine Morin FOBS: A Lightweight Communication Protocol for Grid Computing . . . . . 938 Phillip M. Dickens Low-Fragmentation Mapping Strategies for Linear Forwarding Tables in InfiniBandTM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 947 P. L´ opez, J. Flich, A. Robles A Robust Mechanism for Congestion Control: INC . . . . . . . . . . . . . . . . . . . . 958 Elvira Baydal, P. L´ opez RoCL: A Resource Oriented Communication Library . . . . . . . . . . . . . . . . . . 969 Albano Alves, Ant´ onio Pina, Jos´e Exposto, Jos´e Rufino A QoS Multicast Routing Protocol for Dynamic Group Topology . . . . . . . 980 Li Layuan, Li Chunlin
XXXII
Table of Contents
A Study of Network Capacity under Deflection Routing Schemes . . . . . . . . 989 Josep F` abrega, Xavier Mu˜ noz Implementation and Performance Evaluation of M-VIA on AceNIC Gigabit Ethernet Card . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 995 In-Su Yoon, Sang-Hwa Chung, Ben Lee, Hyuk-Chul Kwon
Topic 15: Mobile and Ubiquitous Computings Topic Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1001 Topic Chairs A Comparative Study of Protocols for Efficient Data Propagation in Smart Dust Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1003 I. Chatzigiannakis, T. Dimitriou, M. Mavronicolas, S. Nikoletseas, P. Spirakis Network Based Mobile Station Positioning in Metropolitan Area . . . . . . . . 1017 Karl R.P.H. Leung, Joseph Kee-Yin Ng, Tim K.T. Chan, Kenneth M.K. Chu, Chun Hung Li Programming Coordinated Motion Patterns with the TOTA Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1027 Marco Mamei, Franco Zambonelli, Letizia Leonardi iClouds – Peer-to-Peer Information Sharing in Mobile Environments . . . . . 1038 Andreas Heinemann, Jussi Kangasharju, Fernando Lyardet, Max M¨ uhlh¨ auser Support for Personal and Service Mobility in Ubiquitous Computing Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1046 K. El-Khatib, N. Hadibi, Gregor v. Bochmann Dynamic Layouts for Wireless ATM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1056 Michele Flammini, Giorgio Gambosi, Alessandro Gasparini, Alfredo Navarra Modeling Context-Aware Behavior by Interpreted ECA Rules . . . . . . . . . . 1064 Wolfgang Beer, Volker Christian, Alois Ferscha, Lars Mehrmann A Coordination Model for ad hoc Mobile Systems . . . . . . . . . . . . . . . . . . . . . 1074 Marco Tulio Valente, Fernando Magno Pereira, Roberto da Silva Bigonha, Mariza Andrade da Silva Bigonha Making Existing Interactive Applications Context-Aware . . . . . . . . . . . . . . . 1082 Tatsuo Nakajima, Atsushi Hasegawa, Tomoyoshi Akutagawa, Akihiro Ibe, Kouji Yamamoto
Table of Contents
XXXIII
Benefits and Requirements of Using Multi-agent Systems on Smart Devices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1091 Cosmin Carabelea, Olivier Boissier, Fano Ramparany Performance Evaluation of Two Congestion Control Mechanisms with On-Demand Distance Vector (AODV) Routing Protocol for Mobile and Wireless Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1099 Azzedine Boukerche Towards an Approach for Mobile Profile Based Distributed Clustering . . . 1109 Christian Seitz, Michael Berger Simulating Demand-Driven Server and Service Location in Third Generation Mobile Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1118 Geraldo Robson Mateus, Olga Goussevskaia, Antonio A.F. Loureiro Designing Mobile Games for a Challenging Experience of the Urban Heritage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1129 Francesco Bellotti, Riccardo Berta, Alessandro De Gloria, Edmondo Ferretti, Massimiliano Margarone QoS Provision in IP Based Mobile Networks . . . . . . . . . . . . . . . . . . . . . . . . . . 1137 ´ ad Husz´ Vilmos Simon, Arp´ ak, S´ andor Szab´ o, S´ andor Imre Design of a Management System for Wireless Home Area Networking . . . . 1141 Tapio Rantanen, Janne Siki¨ o, Marko H¨ annik¨ ainen, Timo Vanhatupa, Olavi Karasti, Timo H¨ am¨ al¨ ainen1 Short Message Service in a Grid-Enabled Computing Environment . . . . . . 1148 Fenglian Xu, Hakki Eres, Simon Cox Service Migration Mechanism Using Mobile Sensor Network . . . . . . . . . . . . 1153 Kyungsoo Lim, Woojin Park, Sinam Woo, Sunshin An
Topic 16: Distributed Systems and Distributed Multimedia Topic Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1159 Topic Chairs Nswap: A Network Swapping Module for Linux Clusters . . . . . . . . . . . . . . . 1160 Tia Newhall, Sean Finney, Kuzman Ganchev, Michael Spiegel Low Overhead Agent Replication for the Reliable Mobile Agent System . . 1170 Taesoon Park, Ilsoo Byun A Transparent Software Distributed Shared Memory . . . . . . . . . . . . . . . . . . 1180 Emil-Dan Kohn, Assaf Schuster
XXXIV
Table of Contents
On the Characterization of Distributed Virtual Environment Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1190 Pedro Morillo, Juan M. Ordu˜ na, M. Fern´ andez, J. Duato A Proxy Placement Algorithm for the Adaptive Multimedia Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1199 Bal´ azs Goldschmidt, Zolt´ an L´ aszl´ o A New Distributed JVM for Cluster Computing . . . . . . . . . . . . . . . . . . . . . . 1207 Marcelo Lobosco, Anderson Silva, Orlando Loques, Claudio L. de Amorim An Extension of BSDL for Multimedia Bitstream Syntax Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1216 Sylvain Devillers Fast Construction, Easy Configuration, and Flexible Management of a Cluster System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1224 Ha Yoon Song, Han-gyoo Kim, Kee Cheol Lee
Topic 17: Peer-to-Peer Computing Topic Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1229 Topic Chairs Hierarchical Peer-to-Peer Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1230 L. Garc´es-Erice, E.W. Biersack, P.A. Felber, K.W. Ross, G. Urvoy-Keller Enabling Peer-to-Peer Interactions for Scientific Applications on the Grid . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1240 Vincent Matossian, Manish Parashar A Spontaneous Overlay Search Tree . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1248 Hung-Chang Hsiao, Chuan-Mao Lin, Chung-Ta King Fault Tolerant Peer-to-Peer Dissemination Network . . . . . . . . . . . . . . . . . . . 1257 Konstantinos G. Zerfiridis, Helen D. Karatza Exploring the Catallactic Coordination Approach for Peer-to-Peer Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1265 Oscar Ardaiz, Pau Artigas, Torsten Eymann, Felix Freitag, Roc Messeguer, Leandro Navarro, and Michael Reinicke Incentives for Combatting Freeriding on P2P Networks . . . . . . . . . . . . . . . . 1273 Sepandar D. Kamvar, Mario T. Schlosser, Hector Garcia-Molina
Table of Contents
XXXV
Topic 18: Demonstrations of Parallel and Distributed Computing Topic Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1280 Topic Chairs Demonstration of P-GRADE Job-Mode for the Grid . . . . . . . . . . . . . . . . . . . 1281 P. Kacsuk, R. Lovas, J. Kov´ acs, F. Szalai, G. Gomb´ as, ´ Horv´ N. Podhorszki, A. ath, A. Hor´ anyi, I. Szeber´enyi, T. Delaitre, G. Tersty´ anszky, A. Gourgoulis Coupling Parallel Simulation and Multi-display Visualization on a PC Cluster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1287 J´er´emie Allard, Bruno Raffin, Florence Zara Kerrighed: A Single System Image Cluster Operating System for High Performance Computing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1291 Christine Morin, Renaud Lottiaux Geoffroy Vall´ee, Pascal Gallard, Ga¨el Utard, R. Badrinath, Louis Rilling ASSIST Demo: A High Level, High Performance, Portable, Structured Parallel Programming Environment at Work . . . . . . . . . . . . . . . 1295 M. Aldinucci, S. Campa, P. Ciullo, M. Coppola, M. Danelutto, P. Pesciullesi, R. Ravazzolo, M. Torquati, M. Vanneschi, C. Zoccolo KOJAK – A Tool Set for Automatic Performance Analysis of Parallel Programs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1301 Bernd Mohr, Felix Wolf Visual System for Developing of Parallel Programs . . . . . . . . . . . . . . . . . . . . 1305 O.G. Monakhov
Late Paper Peer-to-Peer Communication through the Design and Implementation of Xiangqi . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1309 Abdulmotaleb El Saddik, Andre Dufour
Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1315
The Verifying Compiler: A Grand Challenge for Computing Research C.A.R. Hoare Microsoft Research Ltd., 7 JJ Thomson Ave, Cambridge CB3 0FB, UK
[email protected] Abstract. I propose a set of criteria which distinguish a grand challenge in science or engineering from the many other kinds of short-term or long-term research problems that engage the interest of scientists and engineers. The primary purpose of the formulation and promulgation of a grand challenge is to contribute to the advancement of some branch of science or engineering. A grand challenge represents a commitment by a significant section of the research community to work together towards a common goal, agreed to be valuable and achievable by a team effort within a predicted timescale. The challenge is formulated by the researchers themselves as a focus for the research that they wish to pursue in any case, and which they believe can be pursued more effectively by advance planning and co-ordination. Unlike other common kinds of research initiative, a grand challenge should not be triggered by hope of short-term economic, commercial, medical, military or social benefits; and its initiation should not wait for political promotion or for prior allocation of special funding. The goals of the challenge should be purely scientific goals of the advancement of skill and of knowledge. It should appeal not only to the curiosity of scientists and to the ambition of engineers; ideally it should appeal also to the imagination of the general public; thereby it may enlarge the general understanding and appreciation of science, and attract new entrants to a rewarding career in scientific research. As an example drawn from Computer Science, I revive an old challenge: the construction and application of a verifying compiler that guarantees correctness of a program before running it. A verifying compiler uses automated mathematical and logical reasoning methods to check the correctness of the programs that it compiles. The criterion of correctness is specified by types, assertions, and other redundant annotations that are associated with the code of the program, often inferred automatically, and increasingly often supplied by the original programmer. The compiler will work in combination with other program development and testing tools, to achieve any desired degree of confidence in the structural soundness of the system and the total correctness of its more critical components. The only limit to its use will be set by an evaluation of the cost and benefits of accurate and complete formalization of the criterion of correctness for the software.
H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, p. 1, 2003. c Springer-Verlag Berlin Heidelberg 2003
Evolving a Multi-language Object-Oriented Framework: Lessons from .NET Jim Miller Microsoft Corporation,
[email protected] Abstract. In 2001 Microsoft shipped the first public version of its Common Language Runtime (CLR) and the associated object-oriented .NET Framework. This Framework was designed for use by multiple languages through adherence to a Common Language Specification (CLS). The CLR, the CLS, and the basic level of the .NET Framework are all part of International Standard ISO/IEC 23271. Over 20 programming languages have been implemented on top of the CLR, all providing access to the same .Net Framework, and over 20,000,000 copies have been downloaded since its initial release. As a commercial software vendor, Microsoft is deeply concerned with evolving this system. Innovation is required to address new needs, new ideas, and new applications. But backwards compatibility is equally important to give existing customers the confidence that they can build on a stable base even as it evolves over time. This is a hard problem in general, it is made harder by the common use of virtual methods and public state, and harder still by a desire to make the programming model simple. This talk will describe the architectural ramifications of combining easeof-use with system evolution and modularity. These ramifications extend widely throughout the system infrastructure, ranging from the underlying binding mechanism of the virtual machine, through program language syntax extensions, and into the programming environment.
H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, p. 2, 2003. c Springer-Verlag Berlin Heidelberg 2003
Databases, Web Services, and Grid Computing – Standards and Directions Stefan Dessloch University of Kaiserslautern Department of Computer Science Heterogenous Information Systems Group D-67653 Kaiserslautern, Germany
[email protected] Abstract. Over the last two years, web services have emerged as a key technology for distributed object computing, promising significant advantages for overcoming interoperability and heterogeneity problems in large scale, distributed environments. Major vendors have started to incorporate web service technology in their database and middleware products, and companies are starting to exploit the technology in information integration, EAI, and B2B-integration architectures. In the area of Grid computing, which aims at providing a distributed computing architecture and infrastructure for science and engineering, web services have become an important piece of the puzzle by providing so-called Grid Services that help realizing the goal of virtual organizations to coordinate resource sharing and problem solving tasks. The adequate support of data management functionality, or data-oriented services in general within this architectural setting is undoubtedly a key requirement and a number of approaches have been proposed both by research and industry to address the related problems. This talk will give an overview of recent developments in the areas outlined above and discuss important standardization activities as well as trends and directions in industry and research.
H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, p. 3, 2003. c Springer-Verlag Berlin Heidelberg 2003
Ibis: A Java-Based Grid Programming Environment Henri E. Bal Department of Computer Science, Vrije Universiteit, Amsterdam, The Netherlands
[email protected] http://www.cs.vu.nl/ibis/
Ibis [3] is an ongoing research project in which we are building a Java-based Grid programming environment for distributed supercomputing applications. Java’s high portability allows parallel applications to run on a heterogeneous grid without requiring porting or recompilation. A major problem in using Java for high-performance computing, however, is the inferior performance and limited expressiveness of Java’s Remote Method Invocation (RMI). Earlier projects (e.g., Manta [1]) solved the performance problem, but at the cost of using a runtime system written in native code, which gives up Java’s high portability. The philosophy behind Ibis is to try to obtain good performance without using any native code, but allow native solutions as special-case optimizations. For example, a Grid application developed with Ibis can use a pure-Java RMI implementation over TCP/IP that will run "everywhere"; if the application runs on, say, a Myrinet cluster, Ibis can load a more efficient RMI implementation for Myrinet that partially uses native code. The pure-Java implementation of Ibis does several optimizations, using bytecode rewriting. For example, it boosts RMI performance by avoiding the high overhead of runtime type inspection that current RMI implementations have. The special-case implementations do more aggressive optimizations, even allowing zero-copy communication in certain cases. The Ibis programming environment consists of a communication runtime system with a well-defined interface and a range of communication paradigms implemented on top of this interface, including RMI, object replication, group communication, and collective communication, all integrated cleanly into Java. Ibis has also been used to implement Satin [2], which is a Cilk-like wide-area divide-and-conquer system in Java. Experiments have been performanced on two Grid test beds, the Dutch DAS-2 system and the (highly heterogeneous) European GridLab test bed. Our current research on Ibis focuses on fault tolerance and on heterogeneous networks.
References 1. J. Maassen, R. van Nieuwpoort, R. Veldema, H.E. Bal, T. Kielmann, C. Jacobs, and R. Hofman. Efficient Java RMI for Parallel Programming. ACM Trans. on Programming Languages and Systems, 23(6):747–775, November 2001. 2. R. van Nieuwpoort, T. Kielmann, and H.E. Bal. Efficient Load Balancing for Wide-Area Divide-and-Conquer Applications. In ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, pages 34–43, Snowbird, Utah, June 2001. 3. Rob van Nieuwpoort, Jason Maassen, Rutger Hofman, Thilo Kielmann, and Henri Bal. Ibis: an Efficient Java-based Grid Programming Environment. In ACM JavaGrande ISCOPE 2002 Conference, pages 18–27, Seattle, WA, November 2002.
H. Kosch, L. B¨osz¨orm´enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, p. 4, 2003. c Springer-Verlag Berlin Heidelberg 2003
Topic 1 Support Tools and Environments Helmar Burkhart, Rudolf Eigenmann, Tom` as Margalef, and Thomas Ludwig Topic Chairs
It is always an exciting moment to look at the papers that are submitted to Euro-Par’s topic on “Support Tools and Environments”. What types of tools will be presented? What shift of focus in research can we see? What can we conclude for the situation of the user who is always looking for sophisticated tools? This year’s topic will mainly focus on performance analysis and wellelaborated monitoring techniques. It seems as if finally our programs are running but need more tuning in order to be efficient. The research that is conducted in the field of performance analysis is now very advanced. Not only do we find much work with a focus on semi-automatic or even automatic bottleneck detection. Also there is research in the field of comparing multiple experiments with programs and evaluating differences between program runs. This feature does well support the users as code tuning is always an evolutionary process. Current monitoring techniques used as a basis for different types of tools are more generic than ever. They offer configurability with respect to sensors and actuators which cooperate with the program and to events which are triggered and cooperate with the tools. Debuggers now deal with race detection, covering the most problematic error situations in parallel programs. Even with debugging and performance analysis issues being well covered the design and construction of tools remains a challenging task. Users also need tools for e.g., computational steering and load balancing. This is particularly complicated when we consider new architectural structures like the Grid. The distribution of components of tools and programs and the potential failure of connections between them will have to result in more fault tolerant tool architectures. We are convinced that future Euro-Par conferences will reflect this trend. However, independent of the specific research issues presented this year the goal of the topic is still to bring together tool designers, developers, and users and help them in sharing ideas, concepts, and products in this field. This year we have received 19 submissions which is considerably more than in the last year. 7 papers were accepted as full papers for the conference (35%) and 2 papers as short presentation. One paper was accepted as demonstration. H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 5–6, 2003. c Springer-Verlag Berlin Heidelberg 2003
6
H. Burkhart et al.
We would like to thank all authors who submitted a contribution as well as all the reviewers who spent much time to guarantee a sound selection of papers and thus a high quality of the topic. One final comment: We have seen a considerable number of submitted papers that more or less omitted comments how their project is related to earlier or ongoing work. We do not consider this as being proper scientific practice and warmly recommend to put more emphasis on related work sections. We have a 25 years history of tools for parallel computers and an active presence: refer to it.
A Hardware Counters Based Tool for System Monitoring Tiago C. Ferreto1 , Luiz DeRose2 , and C´esar A.F. De Rose1 1
Catholic University of Rio Grande do Sul (PUCRS), Post-Graduate Program on Computer Science, Porto Alegre, Brazil {tferreto,derose}@inf.pucrs.br 2 IBM T.J. Watson Research Center, Yorktown Heights, NY, USA
[email protected] Abstract. In this paper we describe the extensions to the RVision tool to support hardware performance counters monitoring at system level. This monitoring tool is useful for system administrators to detect applications that need tuning. We present a case study using a parallel version of the Swim benchmark from the SPEC suite, running on an Intel Pentium III Linux cluster, where we show a performance improvement of 25%. In addition we present some intrusion measurements showing that our implementation has very low intrusion even with high monitoring frequencies.
1
Introduction
As parallel architectures become more complex it is becoming much more difficult for applications to run at a reasonable fraction of the peak performance of parallel systems. In order to improve program performance, experienced program developers have been using hardware performance counters for application tuning [1]. Likewise, in order to help programmers tune their applications, a variety of utilities and libraries have been created to provide access to the hardware performance counters at user level [2,3,4,5,6, 7,8]. Unfortunately, for most users, it is not always clear when their programs need tuning. Thus, system administrators would like to be able to monitor application performance metrics, such as MFlops/sec and cache hit ratios, to identify programs that could be candidates for optimization. However, there are practically no system-monitoring tools available on cache-based systems that provide such information. An approach being used today by system administrators is to apply “wrappers” to job submission scripts that activate utilities to collect hardware performance information at the application level, in order to generate summary files at the end of the execution. An example of such an approach is the “hpmcollect” interface [9], developed at the Scientific Supercomputing Center at the University of Karlsruhe, which automatically starts an utility [6] to collect hardware performance counters data for all applications submitted for execution, and at the end of the execution, collects and combines the hardware counters output from all parallel tasks, providing a short overview of the total performance and the resource usage of the parallel application.
Work supported by HP-Brazil
H. Kosch, L. B¨osz¨orm´enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 7–16, 2003. c Springer-Verlag Berlin Heidelberg 2003
8
T.C. Ferreto, L. DeRose, and C.A.F. De Rose
In order to provide a more complete solution for system administrators, we extended the RVision tool for cluster monitoring [10] with a monitoring library to collect hardware performance counters information during program execution, and a monitoring client for presentation in the forms of graphs and tables of the hardware events and derived metrics. In this paper we describe the implementation and the main features of this monitoring system, which include the easy of use, the flexibility in selecting events to be monitored, and its very low intrusion. In addition, we demonstrate the value of this system with an example using the SPEC Swim benchmark [11]. The remainder of this paper is organized as follows. We begin in Section 2 describing the utilization of hardware counters for program optimization and explaining the design and implementation of the modules for Hardware Performance Counters monitoring using RVision. In Section 3 we present intrusion measurements. In Section 4 we present an utilization example of the RVision hardware monitoring module. Finally, we summarize our conclusions and directions for future work in Section 5.
2
Hardware Counters Monitoring
Hardware performance counters are special purpose registers available in most modern microprocessors that keep track of programmable hardware events at every cycle. These events represent hardware activities from all functional units of the processor, allowing the low overhead access to hardware performance information, such as counts of instructions, cache misses, and branch misprediction. These registers were originally designed for hardware debugging, but, although the number of counters and type of events available differs significantly between microprocessors, most of them provide a common subset that allows programmers to use these counters to compute derived metrics to correlate the behavior of the application to one or more of the hardware components. With the availability of kernel level APIs to access the hardware counters at a user level, as well as performance tools and libraries that provide event counts and derived metrics, hardware counters have become an invaluable asset for application performance tuning. In contrast, system administrators have not been able to exploit effectively the availability of hardware counters, mainly due to the lack of monitoring systems capable of accessing the counters. The access to derived hardware metrics during program execution would be helpful to system administrators to detect applications that need tuning. For example, the value of a particular metric falling constantly below a pre-defined threshold would be an indication that the program is a candidate for optimization. For our hardware performance counters monitoring approach, we extended RVision, a multi-user monitoring system for GNU/Linux based clusters [10] developed at the Research Center in High Performance Computing (CPAD - PUCRS/HP). It requires the availability of a kernel interface to access the hardware performance counters at a system level, and we consider that programs will have dedicated use of the nodes during execution, which is normally the case on the majority of supercomputing sites. The RVision monitor was originally developed to acquire information such as processor and memory usage, through kernel system calls. Due to RVision’s open architecture, only a new Monitoring Library and a new Monitoring Client, which are described next, were needed to obtain the hardware counters values. The “Monitoring Library” is re-
A Hardware Counters Based Tool for System Monitoring
9
sponsible for the capture of selected hardware events, while the “Monitoring Client” is responsible for presenting this new information, which also includes derived performance metrics. We implemented and tested these components on the “Tropical” Linux Cluster at CPAD, which has 8 dual-processor (Pentium III-1GHz) nodes, switched via a Fast-Ethernet network. Each node has 256 MBytes of main memory. Each processor has two levels of cache: 32 KBytes of Level 1 (with 4-way set associativity) and 256 KBytes of Level 2 (with 8-way set associativity). In both cases the cache line size is 32 bytes. 2.1
Monitoring Library
Currently, Linux does not provide an interface to access the hardware performance counters. Hence, we patched the Linux kernel using the “perfctr” patch and driver, developed by Mikael Petterson [2]. This patch provides a user and a system level interface to access the performance monitoring counters on Intel X86 processors. Since the number of counters and type of events available differs significantly between microprocessors, the RVision Monitoring Library to capture information provided by the hardware counters is architecture dependent. However, our monitoring library can be easily ported to any other platform that provides a kernel level application programming interface to access the hardware counters at system level. The selection of events to be monitored were restricted to the hardware counter events available on the Pentium III processor, which provides two performance monitoring counters capable of counting a total of 77 different events (at most two at a time). In addition, the architecture provides a time stamp counter (tsc), which counts the elapsed machine cycles, as well as the CPU frequency. The complete specification of the performance monitoring counters and the description of all of its events are presented in [12]. From the events provided by the Pentium III architecture, we selected the following set of pair of events (in addition to the tsc) to be used for monitoring: – p6 inst retired and p6 cpu clk unhalted, to count the number of instructions completed and the number of machine cycles used by the program. – p6 flops and p6 cpu clk unhalted, to count the number of floating point instructions and the number of machine cycles used by the program. – p6 data mem refs and p6 dcu lines in, to count the number of level 1 accesses and the number of level 1 cache misses, respectively. – p6 l2 rqsts and p6 l2 lines in, to count the number of level 2 accesses and the number of level 2 cache misses, respectively. – p6 inst decoded and p6 inst retired, to count the number of instructions dispatched and the number of instructions decoded. Depending on the set of counters used, we compute the following derived metrics: MIPS: the average number of instructions per second (in millions), computed as: p6 inst retired/(1000000 ∗ tsc/cpu frequency) Utilization Rate: the ratio of CPU time to wall clock time, computed as: p6 cpu clk unhalted/tsc
10
T.C. Ferreto, L. DeRose, and C.A.F. De Rose
IpC: Instructions per Cycle, computed as: p6 inst retired/p6 cpu clk unhalted MFlops/sec: Millions of floating point operations per second, computed as: p6 flops/(1000000 ∗ tsc/cpu frequency) Level 1 cache hit ratio: Computed as: 100 ∗ (1 − (p6 dcu lines in/p6 data mem refs)) Level 2 cache hit ratio: Computed as: 100 ∗ (1 − (p6 l2 lines in/ p6 l2 rqsts)) Percentage of instructions dispatched that completed: Computed as: 100 ∗ (p6 inst retired/p6 inst decoded) 2.2
Monitoring Client
The monitoring client is responsible for the presentation of tables and graphics with the performance monitoring counters information and the derived metrics. The communication routines of the monitoring client were implemented in C, and the GUI was implemented in Java, with JNI being used to provide the connection between the languages. A snapshot of the monitoring client is presented in Figure 3. It provides a table presenting the hardware counters values and the derived metric values for each node of the cluster, as well as a histogram containing the average of the derived metric for all processors. The histogram scrolls to the left, presenting the most recent information at the rightmost side.
3
Intrusion Measurement
The main concern when a tool is used to monitor some resource is how much the readings are being affected by the tool. Being a parallel application itself, the monitoring tool, when active, is consuming cluster resources such as CPU and network bandwidth. This intrusion should be minimal to guarantee that the monitored data is accurate and that the behavior of the other application running on the cluster is not considerably affected. We measured RVision’s intrusion by defining a monitoring session capturing hardware counters values for all cluster nodes and requesting this data using online monitoring with a regular time interval. Different applications are executed in the cluster with and without monitoring and we compared the execution times. We varied the monitoring time interval from 2 seconds to 200 milliseconds to simulate a worst-case scenario. To evaluate RVision under different workloads we used the following programs from the NAS Parallel Benchmarks [13,14]: Embarassingly Parallel (EP), Integer Sort (IS), LU Decomposition (LU), Conjugate Gradient (CG), and Multigrid (MG). All benchmarks were compiled using class A, defined internally in the NPB, and executed on the 8 nodes of the Tropical Linux Cluster at CPAD [15]. Table 1 presents the intrusion results for all test cases. In each case the test application was executed 10 times for each time interval with monitoring turned on. All benchmarks used in the measurement generated low intrusivity. The worst-case scenario, using 200 milliseconds as time interval, presented intrusion values of less than 1% for all programs with the exception of the IS benchmark. The Integer Sort benchmark
A Hardware Counters Based Tool for System Monitoring
11
Table 1. Intrusion measurements Time Interval (sec) 2.0 1.8 1.6 1.4 1.2 1.0 0.8 0.6 0.4 0.2
EP 0.22% 0.22% 0.23% 0.39% 0.50% 0.54% 0.55% 0.56% 0.66% 0.72%
IS 0.12% 0.15% 0.72% 0.85% 0.86% 0.86% 1.21% 1.61% 1.91% 2.13%
LU 0.09% 0.17% 0.10% 0.11% 0.24% 0.34% 0.38% 0.39% 0.54% 0.71%
CG 0.15% 0.33% 0.44% 0.49% 0.53% 0.53% 0.59% 0.63% 0.64% 0.67%
MG 0.03% 0.10% 0.12% 0.12% 0.17% 0.17% 0.21% 0.29% 0.33% 0.51%
has a high network utilization. Hence, it was more affected by the traffic generated by the on-line monitoring.
4
Example of Use
In order to demonstrate the usefulness of the hardware counters monitoring feature of RVision, we ran a parallel version of the SPEC Swim benchmark with problem size defined by N1=N2=2048, on the Tropical Linux cluster, and monitored both the MFlops/sec rate and the level 1 cache hit ratio during the execution of the program, using 1 second as monitoring interval. Figure 1 presents the system’s average L1 hit ratio when the program started its execution, we observe that the cache hit ratio dropped considerable (most recent information is presented at the rightmost side). Looking at Figure 2, which presents a table with a snapshot with the monitoring values for all processors, we observe that the level 1 hit ratio was in the order of 54% in all processors, indicating that the application needed some sort of program restructuring, due to its poor utilization of the memory subsystem. This poor cache utilization reflects in the MFlops/sec rate, shown in Figure 3, which indicates a sustained performance of around 67 MFlops/sec.
Fig. 1. System’s average L1 hit ratio running the SPEC Swim benchmark
12
T.C. Ferreto, L. DeRose, and C.A.F. De Rose
Fig. 2. System’s L1 hit ratio running the SPEC Swim benchmark
Fig. 3. System’s MFlops/sec rate running the SPEC Swim benchmark
Looking at the source code, we observe that the basic data structure of the application is defined in the “Common block” shown in Figure 4, and used in loops such as the one presented in Figure 5. The use of high powers of 2 as array dimensions (2048 in this case) would result in an excessive cache miss ratio for this application, due to the low associativity of the level 1 cache (4-way). This problem occurs because 9 different arrays
A Hardware Counters Based Tool for System Monitoring
13
are being used in the loop, and since the size of each array is multiple of the cache size, array elements with the same indices will map to the same cache set. Since each set can accommodate only 4 different entries, this loop generates an excessive number of conflict misses. This could be a well-known fact for application tuning specialists and experienced programmers, but not necessarily well understood by scientists with no background in computer architecture. Hence, the availability of monitoring tools that could indicate to system administrators when an application may need tuning is an invaluable asset. PARAMETER (N1=2048, N2=2048) COMMON U(N1,N2), V(N1,N2), P(N1,N2), * UNEW(N1,N2), VNEW(N1,N2), PNEW(N1,N2), UOLD(N1,N2), * VOLD(N1,N2), POLD(N1,N2), CU(N1,N2), CV(N1,N2), * Z(N1,N2), H(N1,N2), PSI(N1,N2) Fig. 4. Definition of the main data structure in the SPEC Swim benchmark DO 300 J=js,je DO 300 I=1,M UOLD(I,J) = U(I,J)+ALPHA*(UNEW(I,J)-2.*U(I,J)+UOLD(I,J)) VOLD(I,J) = V(I,J)+ALPHA*(VNEW(I,J)-2.*V(I,J)+VOLD(I,J)) POLD(I,J) = P(I,J)+ALPHA*(PNEW(I,J)-2.*P(I,J)+POLD(I,J)) U(I,J) = UNEW(I,J) V(I,J) = VNEW(I,J) P(I,J) = PNEW(I,J) 300 CONTINUE Fig. 5. Utilization of the arrays in one of loops in the SPEC Swim benchmark
In this case, the simple solution would be to “pad” the common block in Figure 4 with declaration of dummy vectors between the array declarations, as shown in Figure 6. The use of dummy vectors of 8 elements each will separate the mapping of the elements of the arrays with the same indices by one cache line, which will reduce the conflicts. When executing the modified program, we observe an increase in the Level 1 cache hit ratio to about 78%, as shown in Figure 7 and a sustained performance in the order of 84 MFlops/sec, as shown in Figure 8, which corresponds to an improvement of 25% in performance.
* * * *
PARAMETER (N1=2048, N2=2048) COMMON U(N1,N2), D1(8), V(N1,N2), D2(8), P(N1,N2), UNEW(N1,N2), D3(8), VNEW(N1,N2), D4(8), PNEW(N1,N2), D5(8), UOLD(N1,N2), D6(8), VOLD(N1,N2), D7(8), POLD(N1,N2), D8(8), CU(N1,N2), D9(8), CV(N1,N2), D10(8), Z(N1,N2), D11(8), H(N1,N2), D12(8), PSI(N1,N2) Fig. 6. Padded common block for the SPEC Swim benchmark
14
T.C. Ferreto, L. DeRose, and C.A.F. De Rose
Fig. 7. System’s L1 hit ratio running the padded version of the SPEC Swim benchmark
Other examples of performance problems that can be detected with this monitoring approach, which are not presented here due to space limitations are: Utilization Rate: For a task on a dedicated compute node, this ratio should be close to 1. Lower values would indicate large system activity, which could require some application performance tuning. IpC: Given that the processor has multiple functional units, a well tuned program should have IpC larger than 1. Percentage of instructions dispatched that completed: This metric gives an indication of how well the speculation is working for the program. A low percentage could indicate that the program has loops with few iteration counts that could benefit from loop unrolling. MIPS: This metric can be used for non-floating point intensive codes for monitoring of the sustained performance of the application. L1 and L2 cache hit ratios: Are also useful to detect problems in the memory subsystem caused by capacity misses. These problems might be solved with loop transformations, such as blocking, to increase data locality.
5
Conclusions and Future Work
In this paper we presented a new approach to system monitoring for cluster architectures based on hardware counters. Our main goal is to provide an efficient tool for system
A Hardware Counters Based Tool for System Monitoring
15
Fig. 8. System’s MFlops/sec rate running the padded version of the Swim benchmark
administrators to detect programs that cause bottlenecks in cluster architectures so that throughput can be increased. As expected hardware counters monitoring has a very low intrusion resulting in more precise results. In dedicated systems it is also possible to detect applications that could need tuning, without the instrumentation needed by traditional hardware counters monitoring at application level. To investigate this new concept we expanded our resource monitor RVision to access hardware counters on Intel Pentium III processors and to calculate derived performance metrics for Linux clusters. With this tool we analyzed the results obtained in a case study where we optimized the execution of a parallel version of the Swim benchmark from the SPEC suite, obtaining a performance increase in the order of 25%, on a Linux cluster with 16 processors. We also presented intrusion results for the expanded version of RVision running the NAS benchmark suite. We observed that in most cases intrusion is less than 1% even with high monitoring frequencies like 200 milliseconds. We believe that systems oriented hardware monitoring tools are a very interesting alternative for system and application tuning. Our next steps is to support additional processor architectures like the Intel Pentium IV and the IBM Power4, where it will be possible to work with more derived metrics due to the increased number of hardware counters. More information about RVision and the package for download are available at http://rvision.sourceforge.net.
16
T.C. Ferreto, L. DeRose, and C.A.F. De Rose
References 1. Zagha, M., Larson, B., Turner, S., Itzkowitz, M.: Performance Analysis Using the MIPS R10000 Performance Counters. In: Proceedings of Supercomputing’96. (November 1996) 2. Pettersson, M.: Linux X86 Performance-Monitoring Counters Driver. http://user.it.uu.se/˜mikpe/linux/perfctr/. Computing Science Department; Uppsala University – Sweden. (2002) 3. Browne, S., Dongarra, J., Garner, N., London, K., Mucci, P.: A Scalable Cross-Platform Infrastructure forApplication Performance Tuning Using Hardware Counters. In: Proceedings of Supercomputing’00. (November 2000) 4. Research Centre Juelich GmbH: PCL – The Performance Counter Library: A Common Interface to Access Hardware Performance Counters on Microprocessors. http://www.fzjuelich.de/zam/PCL/. (2002) 5. May, J.M.: MPX: Software for multiplexing hardware performance counters in multithreaded programs. In: Proceedings of 2001 International Parallel and Distributed Processing Symposium. (April 2001) 6. DeRose, L.: The Hardware Performance Monitor Toolkit. In: Proceedings of Euro-Par. (August 2001) 122–131 7. Janssen, C.: The visual Profiler. http://aros.ca.sandia.gov/˜cljanss/perf/vprof/. Sandia National Laboratories. (2002) 8. Buck, B., Hollingsworth, J.K.: Using Hardware Performance Monitors to Isolate Memory Bottlenecks. In: Proceedings of Supercomputing’01. (November 2001) 9. Geers, N.: Automatic Collection of HPM Data in Parallel Applications. http://www.unikarlsruhe.de/˜SP/software/tools/hpm/hpmcollect.de.html. Scientific Supercomputing Center at University of Karlsruhe. (2002) 10. Ferreto, T.C., De Rose, C.A.F., DeRose, L.: RVision: An Open and High Configurable Tool for Cluster Monitoring. In: Proceedings of the Second IEEE/ACM International Symposium on Cluster Computing and the Grid, Berlin, Germany (2002) 75–82 11. Sadourny, R.: The Dynamics of Finite-Difference Model of the Shallow-Water Equations. Journal of Atmospheric Sciences 32 (1975) 12. Intel Corporation: IA-32 Intel Architecture Software Developer’s Manual, Volume 3: System Programming Guide. (2002) 13. Bailey, D., Harris, T., Saphir, W., van der Wijngaart, R., Woo, A.,Yarrow, M.: The NAS Parallel Benchmarks 2.0. Technical Report NAS-95-929, NASA Ames Research Center (1995) 14. Saphir, W., Wijngaart, R.V.D., Woo, A., Yarrow", M.: New implementations and results for the NAS parallel benchmarks 2. In: Proceedings of the 8th SIAM Conference on Parallel Processing for Scientific Computing. (1997) 15. Research Center in High Performance Computing: http://www.cpad.pucrs.br (2002)
ParaProf: A Portable, Extensible, and Scalable Tool for Parallel Performance Profile Analysis Robert Bell, Allen D. Malony, and Sameer Shende University of Oregon, Eugene, OR 97403 USA {bertie,malony,sameer}@cs.uoregon.edu
Abstract. This paper presents the design, implementation, and application of ParaProf, a portable, extensible, and scalable tool for parallel performance profile analysis. ParaProf attempts to offer “best of breed” capabilities to performance analysts – those inherited from a rich history of single processor profilers and those being pioneered in parallel tools research. We present ParaProf as a parallel profile analysis framework that can be retargeted and extended as required. ParaProf’s design and operation is discussed, and its novel support for large-scale parallel analysis demonstrated with a 512-processor application profile generated using the TAU performance system.
1
Introduction
Perhaps the best known method for observing the performance of software and systems is profiling. Profiling techniques designed over thirty years ago [11], such as prof [17] and gprof [5] for Unix, are still apparent in the profiling approaches for modern computer platforms (e.g., vprof [9] and cvperf [20]). While ideological differences exist in profiling instrumentation and data collection, most notably between sample-based versus measurement-based approaches, profiling is the most commonly used tool in the performance analyst’s repertoire. Unfortunately, despite the ubiquitous nature of profiling, profile analysis tools have tended to be system specific, proprietary, and incompatible. Not only does this pose difficulties for cross-platform performance studies, but the lack of reusable profile analysis technology has slowed the development of next-generation tools. With the general availability of hardware counters in modern microprocessors, the complexity of performance profile data will increase, further exacerbating the need for more robust profile analysis tool support. Parallel software and systems introduce more difficult challenges to profile analysis tools. All the concerns for single processor performance analysis are present in parallel profiling, except now the profile data size is amplified by the number of processors involved. Profile analysis scalability is important for large-scale parallel systems where each thread of execution may potentially generate its own profile data set. Parallel execution also introduces new performance properties [1] and problems that require more sophisticated analysis and interpretation, both within a single profiling experiment and across experiments. The identification of parallel inefficiencies and load imbalances requires analysis of all execution threads H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 17–26, 2003. c Springer-Verlag Berlin Heidelberg 2003
18
R. Bell, A.D. Malony, and S. Shende
in a profile, whereas pinpointing reasons for poor scalability, for example, must combine detailed analysis across profiles generated with different numbers of processors. Lastly, the performance problem solving provided by the parallel profile analysis tool should reflect the system architecture as well as the model of parallelism used in the application. This paper presents the design, implementation, and application of ParaProf, a portable, extensible, and scalable tool for parallel performance profile analysis. ParaProf attempts to offer “best of breed” capabilities to performance analysts – those inherited from a rich history of single processor profilers and those being pioneered in parallel tools research. However, ParaProf should be regarded not as a complete solution, but rather a parallel profile analysis framework that can be retargeted and extended as needed. Thus, in this paper we emphasize, in equal measure, the design of ParaProf and its support for customizability, in addition to its application in the context of the TAU performance system. In the sections that follow, we first relate ParaProf to selected profiling tools, sequential and parallel, that highlight important features that ParaProf incorporates, or aspires too. Section §3 describes the ParaProf architecture and section §4 goes into more detail of the operation of core components. While ParaProf is being applied in many applications, for this paper, we focus on its novel support for large-scale profile analysis. Section §5 demonstrates ParaProf’s core capabilities on a highly-parallel application developed with the SAMRAI framework [8], one of several large-scale parallel environments where ParaProf is being deployed. In conclusion, we remark on how we see ParaProf evolving and its integration in a parallel performance diagnosis environment.
2
Related Work
Performance profiling characterizes the execution behavior of an application as a set of summary statistics associating performance data and metrics with program structure and semantics.1 Profiling tools are distinguished by two aspects: what performance information is being analyzed, and how profile results are mapped to the program and presented to the user. Since prof [17] and gprof [5], execution time has been the standard performance data profiled, and the dominant mapping of execution time has been to program source statements. The inclusion of hardware performance counts in profile data (e.g., SGI’s ssrun [20]) has significantly increased the insight on processor and memory system behavior. The performance API (PAPI [2]) provides a common interface to hardware performance counters across microprocessors and is in use by most current profiling tools. The types of program mapping to source statements include statement-, loop-, and routine-level mapping. Callgraph profiling, pioneered in gprof, has also been extended to callpath profiling [6]. All of the above profiling features can be found in various forms in sequential profile analysis tools (e.g., cvperf [20], DynaProf [14], and vprof [9]). The 1
We can also speak of profiling the performance of a system, but to simplify the discussion, we will focus on application performance.
ParaProf: A Portable, Extensible, and Scalable Tool
19
HPCView tool [13] best exemplifies the integration of sequential analysis capabilities. Profile data from multiple sources can be input to HPCView, including performance data sets from hardware counters and different program executions. Internally, an extensible profile data format allows different data to be associated with program sites, each identified by a unique name and source mapping information. HPCView can compute derived performance statistics from mathematical expressions that involve performance data variables. The user interface allows navigation, grouping, and sorting of the profile data to assist in results analysis. However, HPCView is not a parallel profile analysis tool, per se. ParaProf can support many of HPCView’s features, as well as provide scalable parallel profile analysis. Parallel profile analysis tools often target specific parallel programming models. The GuideView [10] tool analyzes performance of multi-threaded OpenMP applications. It has sophisticated analysis functions, including handling of multiple experiments and relating performance metrics to ideal values. VGV [7] extends GuideView to hybrid OpenMP and MPI applications. Unfortunately, neither GuideView nor VGV are open systems (now Intel proprietary), and, thus, are able to accommodate more general parallel profile data. The HPM Toolkit [3] also targets OpenMP and MPI applications, with emphasis on the analysis of hardware performance monitor data and derived statistics, but it only runs on IBM platforms. Aksum [4] handles OpenMP and MPI applications run on Linux clusters and can analyze profiles across multiple experiments. SvPablo [15] processes profiles captured for multiple processes on several platforms, and like HPCView, presents performance results in a source-level view. Expert [19] analyzes more extensive profile information to associate performance properties and problems to different program and execution views. With its general event representation and programmable analysis, ParaProf is able process the profile information for many of the scenarios handled by these tools. Moreover, although none of these tools is specifically focussed on large-scale parallelism, ParaProf’s profile data management and analysis is scalable to thousands of processors.
3
ParaProf Architecture
Software reuse and componentization lies at the heart of much current research in software engineering. Our goal in the ParaProf project is to apply these design principles to performance profile analysis and visualization. Given the commonalities of profile data and semantics, the opportunity is there to develop a framework for analysis and visualization that can be specialized for the parallel profiling problem. To this effect, we have abstracted four key components in the design of ParaProf: the Data Source System (DSS), the Data Management System (DMS), the Event System (ES), and the Visualization System (VS). Each component is independent, and provides well-defined interfaces to other components in the system. The result is high extensibility and flexibility, enabling us to tackle the issues of re-use and scalability. The remainder of this section describes each these components.
Direct from Application
R. Bell, A.D. Malony, and S. Shende
Profile Management Displays
Profile Data Model (node, context, thread)
API
File System Access
Database Access
20
Event System (Java event model)
Fig. 1. ParaProf Architecture.
Current performance profilers provide a range of differing data formats. As done in HPCView [13], external translators have typically been used to merge profile data sets. Since much commonality exists in the profile entities being represented, this is a valid approach, but it requires the adoption of a common format. ParaProf’s DSS addresses this issue in a different manner. DSS consists of two parts. One, DSS can be configured with profile input modules to read profiles from different sources. The existing translators provides a good starting point to implement these modules. An input module can also support interfaces for communication with profiles stored in files, managed by performance databases, or streaming continuously across a network. Two, once the profile is input, DSS converts the profile data to a more efficient internal representation. The DMS provides an abstract representation of performance data to external components. Its supports many advanced capabilities required in a modern performance analysis system, such as derived metrics for relating performance data, cross experiment analysis for analyzing data from disparate experiments, and data reduction for elimination of redundant data, thus allowing large data sources to be tolerated efficiently. The importance of sophisticated data management and its support for exposing data relationships is an increasingly important area of research in performance analysis. The DMS design provides a great degree of flexibility for developing new techniques that can be incorporated to extend its function. The VS components is responsible for graphical profile displays. It is based on the Java2D platform, enabling us to take advantage of a very portable development environment that continues to increase in performance and reliability. Analysis of performance data requires representations from a very fine granularity, perhaps of a single event on a single node, to displays of the performance characteristics of the entire application. ParaProf’s current set of displays range from purely textual based to fully graphical. Significant effort has been put into
ParaProf: A Portable, Extensible, and Scalable Tool
21
making the displays highly interactive and fast to draw. In addition, it is relatively easy to extend the display types to better show data relations. Lastly, in the ES, we have provided a well-defined means by which these components can communicate various state changes, and requests to other components in ParaProf. Many of the display types are hyper-linked enabled, allowing selections to be reflected across currently open windows. Support for runtime performance analysis and application steering, coupled with maintaining connectivity with remote data repositories has required us to focus more attention on the ES, and to treat it as a wholly separate component system.
4
Operation
Let us now examine a typical usage scenario. A ParaProf session begins with a top-level profile management window that lets the user decide what performance experiments they want to analyze. An experiment is generally defined by a particular application and its set of associated performance profiles coming from experiment trials. As shown in Figure 2, the profile manager provides access to different profile sources (e.g., file system, performance database, or online execution) and to different profile experiments. Several profile data sets can be active within ParaProf at the same time. These may be from different experiments, allowing the user to compare performance behavior between applications, or from multiple runs of the same application where each profiled a different performance value. Note that the profile management window is where the user can also specify performance data calculations involving profile data values to derive new performance statistics. We discuss this further below. Once performance profiles have been selected, ParaProf’s view set then offers means to understand the performance data at a variety of levels. The global profile view is used to see profile data for all application events across all threads2 of execution in a single view. Of course, not all of the profile data can be shown, so the user has control over what performance metrics to display through menu options. The global view is interactive in the sense that clicking on various parts of the display provides a means to explore the performance data in more detail through other views. For example, the global view of a 20-processor VTF [18] parallel profile is shown in Figure 3. The two other views show 1) the performance of one event across all threads, gotten from clicking one color segement, and 2) the performance of all events across a single thread, gotten from clicking on the node/context/thead identifier. The full profile data for any thread can be shown in an ASCII view at any time. All of the profile views are scrollable. For large numbers of events or large numbers of threads, scrolling efficency is important and we have optimized this in our Java implementation. We have also provided support for sizing down the 2
We use the term “thread” here in a general sense to denote a thread of execution. ParaProf’s internal profile data is organized based on a parallel execution model of shared-memory computing nodes where contexts reside, each providing a virtual address space shared by multiple threads of execution.
22
R. Bell, A.D. Malony, and S. Shende
Fig. 2. ParaProf Profile Management Window.
display bars so that larger numbers of threads can be visualized. Nevertheless, for applications with high levels of parallelism, the two-dimensional profile displays can become unwieldy when trying to understand performance behavior. We have recently implemented histogram analysis support to determine the distribution of data values across their value range. A histogram display shows the distribution as a set of value bins (the number of bins is user defined) equally spaced between minimum and maximum. The number of profile values falling in a bin determines the height of the bin bar in the display. Concurrent graphical and textual data representation at every stage provides for a seamless translation between levels of data granularity. In addition, the DMS controllers enable easy navigation between data sources thus facilitating understanding of data relationships in the AE. To highlight the importance of this last point, let us consider the following important analysis requirement. Most modern CPUs provide support for tracking a variety of performance metrics. The TAU performance system [16], for example, can simultaneously track CPU cycles, floating point instructions, data cache misses, and so on, using the PAPI library. A consistent problem in performance analysis has been how to examine on a large scale other useful performance metrics that can be derived from measured base values. For example, to determine how efficiently the pipelines of modern super-scalar architectures are being used, our performance system
ParaProf: A Portable, Extensible, and Scalable Tool
23
Fig. 3. ParaProf Display of VTF Profile.
might have access to both CPU cycles, and instruction counts, but gives us no correlation between the two. It is cumbersome to identify performance bottlenecks in CPU pipe utilization across thousands of events or threads if one has to make visual comparisons. To solve such problems, ParaProf’s DMS can apply mathematical operations to the performance metrics gathered, thus obtaining more detailed statistics. These operations can be applied to single executions, to executions in different runs, and even across experiments, thus allowing a complex range of derived metrics to be gathered. When examining data across experiments, the DMS can tolerate disparities in data sources (an event might be present in one source, but not another) by examining the commonalities that do exist, and presenting only those. Data sources are treated as operands in the system, and the DMS allows a user to compose operations to produce a variety of derived statistics. Results from this analysis can then be saved for future reference. To aid in the visual process, the ES passes user activity between windows so that areas of interest highlighted in one window are propagated to all related windows. This greatly reduces the time spent correlating data shown in different displays. Another feature of ParaProf (not show here) is its ability to recognize and create event groupings. Events can be grouped either at runtime by the performance system (supported in TAU), or post-runtime by ParaProf. Displays can then be instructed to show only events that are members of particular groups. This provides another mechanism for reducing visual complexity, and focusing only on points of interest or concern. ParaProf’s DMS demonstrated its ability to simultaneously handle the large quantity of data comfortably. Any data redundancies present in the source data were eliminated as expected, and we showed (via duplication and renaming) that the only practical limits to ParaProf’s operation are the memory limitations of the platform. Even these can be
24
R. Bell, A.D. Malony, and S. Shende
alleviated to a great extent by the use of a central repository of data (such as a database) to which ParaProf can be directed to simply maintain links.
5
ParaProf Application
ParaProf, in its earlier incarnation as jRacy, has been applied in many application performance studies over the last several years. The additions we have made for profile management, multi-experiment support, derivative performance analysis, performance database interaction, and large-scale parallel profiles, plus overall improvements in efficiency, have elevated the tool to its new moniker distinction. Here we would like to focus on ParaProf’s support for profile analysis of large-scale parallel applications. This will be an important area for our future work with the DOE laboratories and ASCI platforms.
Fig. 4. Scalable SAMRAI Profile Display.
To demonstrate ParaProf’s ability to handle data from large parallel applications in an informative manner, we applied ParaProf to TAU data obtained during the profiling of a SAMRAI [8] application run on 512 processor nodes. SAMRAI is a C++ class library for structured adaptive mesh refinement. Figure 4 shows a view of exclusive wall-clock time for all events. The display is fully interactive, and can be “zoomed” in or out to show local detail. Even so, some
ParaProf: A Portable, Extensible, and Scalable Tool
25
Fig. 5. SAMRAI Histogram Displays.
performance characteristics can still be difficult to comprehend when presented with so much visual data. Figure 5 shows one of ParaProf’s histogramming options that enable global data characteristics to be computed and displayed in a more concise form. In this case, we see binning of exclusive time data across the system for the routine whose total exclusive time is largest, MPI Allreduce(). We have included another histogram showing the floating point instructions of a SAMRAI module; this data was obtained using TAU’s interface to PAPI. Clearly, the histogram view is useful for understanding value distribution.
6
Conclusion
The ParaProf profile analysis framework incorporates important features from a rich heritage of performance profiling tools while addressing new challenges for large-scale paralle performance analysis. Although ParaProf was developed as part of the TAU performance system, our primary goals in ParaProf’s design were flexibility and extensibility. As such, we are positioning ParaProf to accept profile data from a variety of sources, to allow more complete performance analysis. We will also continue to enhance ParaProf’s displays to aid in performance investigation and interpretation. In particular, there are opportunities to apply three-dimensional graphics to visualize profile results for very large processor runs. Currently, we are developing an advanced visualization library based on OpenGL that can be used by ParaProf. There are two areas where we want to improve ParaProf’s capabilities. First, other parallel profile tools provide linkage back to application source code. The information needed for this is partly encoded in the profile event names, but ParaProf needs to have a standard means to acquire source mapping metadata (e.g., source files, and line and column position) to associate events to the program. We will apply our PDT [12] source analysis technology to this problem, and also hope to leverage the work in HPCView. In addition, source text display and interaction capabilities are required. The second area is to improve how performance calculations are specified and implemented in ParaProf. Our plan is to develop an easy to use interface to
26
R. Bell, A.D. Malony, and S. Shende
define analysis formalae, whereby more complex expressions, including reduction operations, can be created. These can then be saved in an analysis library for reuse in future performance profiling studies.
References 1. APART, IST Working Group on Automatic Performance Analysis: Real Tools. See http://www.fz-juelich.de. 2. S. Browne, J. Dongarra, N. Garner, G. Ho, and P. Mucci, “A Portable Programming Interface for Performance Evaluation on Modern Processors,” International Journal of High Performance Computing Applications, 14(3):189–204, Fall 2000. 3. L. DeRose, “The Hardware Performance Monitor Toolkit,” Euro-Par 2001, 2001. 4. T. Fahringer and C. Seragiotto, “Experience with Aksum: A Semi-Automatic Multi-Experiment Performance Analysis Tool for Parallel and Distributed Applications,” Workshop on Performance Analysis and Distributed Computing, 2002. 5. S. Graham, P. Kessler, and M. McKusick, “gprof: A Call Graph Execution Profiler,” SIGPLAN ’82 Symposium on Compiler Construction, pp. 120–126, June 1982. 6. R. Hall, “Call Path Profiling,” International Conference on Software Engineering, pp. 296–306, 1992. 7. J. Hoeflinger et al., “An Integrated Performance Visualizer for MPI/OpenMP Programs,” Workshop on OpenMP Applications and Tools (WOMPAT), July 2001. 8. R. Hornung and S. Kohn, “Managing Application Complexity in the SAMRAI Object-Oriented Framework, Concurrency and Computation: Practice and Experience, special issue on Software Architectures for Scientific Applications, 2001. 9. C. Janssen, “The Visual Profiler.” http://aros.ca.sandia.gov/ cljanss/perf/vprof/. 10. KAI Software, a division of Intel Americas, “GuideView Performance Analyzer,” 2001. http://www.kai.com/parallel/kapro/guideview. 11. D. Knuth, “An Empirical Study of FORTRAN Programs,” Software – Practice and Experience, 1:105–133, 1971. 12. K. Lindlan, J. Cuny, A. Malony, S. Shende, B. Mohr, R. Rivenburgh, C. Rasmussen, “Tool Framework for Static and Dynamic Analysis of Object-Oriented Software with Templates,” Proc. Supercomputing 2000, November, 2000. 13. J. Mellor-Crummey, R. Fowler, and G. Marin, “HPCView: A Tool for Top-down Analysis of Node Performance,” The Journal of Supercomputing, 23:81–104, 2002. 14. P. Mucci, “Dynaprof.” http://www.cs.utk.edu/ mucci/dynaprof 15. D. Reed, L. DeRose, and Y. Zhang, “SvPablo: A Multi-Language Performance Analysis System,” 10th International Conference on Performance Tools, pp. 352– 355, September 1998. 16. TAU (Tuning and Analysis Utilities). http://www.acl.lanl.gov/tau. 17. Unix Programmer’s Manual, “prof command,” Section 1, Bell Laboratories, Murray Hill, NJ, January 1979. 18. VTF, Virtual Test Shock Facility, Center for Simulation of Dynamic Response of Materials. http://www.cacr.caltech.edu/ASAP. 19. F. Wolf and B. Mohr, “Automatic Performance Analysis of SMP Cluster Applications,” Technical Report IB 2001-05, Research Centre J¨ ulich, 2001. 20. M. Zagha, B. Larson, S. Turner, and M. Itzkowitz, “Performance Analysis Using the MIPS R10000 Performance Counters,” Supercomputing ’96, November 1996.
On Utilizing Experiment Data Repository for Performance Analysis of Parallel Applications Hong-Linh Truong and Thomas Fahringer Institute for Software Science, University of Vienna Liechtensteinstr. 22, A-1090 Vienna, Austria {truong,tf}@par.univie.ac.at
Abstract. Performance data usually must be archived for various performance analysis and optimization tasks such as multi-experiment analysis, performance comparison, automated performance diagnosis. However, little effort has been done to employ data repositories to organize and store performance data. This lack of systematic organization of data has hindered several aspects of performance analysis tools such as performance comparison, performance data sharing and tools integration. In this paper we describe our approach to exploit a relational-based experiment data repository in SCALEA which is a performance instrumentation, measurement, analysis and visualization tool for parallel programs. We present the design and use of SCALEA’s experiment data repository which is employed to store information about performance experiments including application, source code, machine information and performance data. Performance results are associated with experiments, source code and machine information. SCALEA is able to offer search and filter capabilities, to support multi-experiment analysis as well as to provide well-defined interfaces for accessing the data repository and leveraging the performance data sharing and tools integration.
1
Introduction
Collecting and archiving performance data are important tasks required by various performance analysis and optimization processes such as multi-experiment analysis, performance comparison and automated performance diagnosis. However, little effort has been done to employ data repositories to organize and store performance data. This lack of systematic organization of data has hindered several aspects of performance analysis tools. For example, users commonly create their own performance collections, extract performance data and use external tools to compare performance outcome of several experiments manually. Moreover, different performance tools employ different performance data formats and they lack well-defined interfaces for accessing the data. As a result, the collaboration among performance tools and high-level tools is hampered.
This research is partially supported by the Austrian Science Fund as part of the Aurora Project under contract SFBF1104.
H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 27–37, 2003. c Springer-Verlag Berlin Heidelberg 2003
28
H.-L. Truong and T. Fahringer
Utilizing a data repository and providing well-known interfaces to access data in the repository can help to overcome the abovementioned limitations. We can structure the data associated with performance experiments thus performance results can always be associated with their source codes and machine description on which the experiment has been taken. Based on that, any other performance tool can store its performance data for a given application to the same repository thus providing a large potential to enable more sophisticated performance analysis. And then, any other tools or system software can easily access the performance data through a well-defined interface. To do so, we have investigated and exploited a relational-based experiment data repository in SCALEA [12,11] which is a performance instrumentation, measurement, analysis and visualization tool for parallel programs. In this paper, we present the design and use of SCALEA’s experiment data repository which is employed to store performance data and information about performance experiment which alleviates the association of performance information with experiments and source code. SCALEA’s experiment data repository has been implemented with relational database, SQL and accessed through interfaces based on JDBC. We demonstrate significant achievements gained when exploiting this data repository such as the capabilities to support search and multi-experiment analysis, to facilitate and leverage performance data sharing and collaboration among different tools. We also discuss other directions on utilizing performance data repository for performance analysis. The rest of this paper is organized as follows: Section 2 details the SCALEA’s experiment data repository. We then illustrate achievements gained from the use of the experiment data repository in Section 3. We discuss other directions to utilize the experiment data repository in Section 4. The related work is presented in Section 5 followed by the conclusion and future work in Section 6.
2 2.1
Experiment Data Repository Experiment-Related Data
Figure 1 shows the structure of the data stored in SCALEA’s experiment data repository. An experiment refers to a sequential or parallel execution of a program on a given target architecture. Every experiment is described by experimentrelated data, which includes information about the application code, the part of a machine on which the code has been executed, and performance information. An application (program) may have a number of code versions, each of them consists of a set of source files and is associated with one or several experiments. Every source file has one or several static code regions (ranging from entire program units to single statements), uniquely specified by their positions – start/end line and column – where the region begins and ends in the source file. Experiments are associated with virtual machines on which they have been taken. The virtual machine is a collection of physical machines to execute the experiment; it is described as a set of computational nodes (e.g. single-processor
On Utilizing Experiment Data Repository for Performance Analysis Application
name ...
VirtualMachine
Experiment
Version 1:n
29
ComputationalNode
1:n startTime endTime commandLine compiler compilerOptions ...
versionInfo ... m:n SourceFile
n:1
CodeRegion 1:n start_line start_col end_line end_col ...
name peakFlops benchmarkFlops ...
RegionSummary
1:n
computationalNode processID threadID codeRegionID ...
1:n
1:n
1:n
Network
NodeSharedMemoryPerf
name bandwidth latency ...
1:n 1:n
n:1
PerformanceMetric
NetworkMPColPerf name value
hostName hostAddresses systemModel physMem virtMem numCpu cpuType cpuSpeed osName peakFlops benchmarkFlops ...
Cluster
1:n name content location ...
1:n
name nodes[] nprocs[] ...
libName numNode barrier bcast[] ...
libName lock[] fork[] join[] ...
1:n NetworkMPP2PPerf
libName uniDirectionSync[] biDirectionSync[] aSync[] ...
Fig. 1. SCALEA’s Experiment-Related Data Model
systems, SMP nodes) of clusters. A Cluster is a group of computational nodes (physical machines) connected by specific networks. Computational nodes in the same cluster have the same physical configuration. Note that this structure is still suitable for a network of workstations as workstations can be classified into groups of machines having the same configuration. Specific information of physical machines such as memory capacity, peak FLOPS is measured and stored in the data repository. In addition, for each computational node, performance characteristics of shared memory operations (e.g. lock, barrier, fork/join thread) are benchmarked and stored in NodeSharedMemoryPerf. Similarly, for each network of a cluster, performance characteristics of message passing model are also benchmarked and stored in NetworkMPColPef and NetworkMPP2PPerf for collective and point-to-point operations, respectively. A region summary refers to the performance information collected for a given code region on a specific processing unit (consisting of computational node, process, thread). The region summaries are associated with performance metrics that comprise performance overheads, timing information, and hardware parameters. A region summary has a link to a parent region summary; this link reflects the calling relationship between the two regions recorded in the dynamic code region call graph [12].
2.2
Implementation Overview
Figure 2 depicts components that interact with the experiment data repository. The post-processing is used to store source programs and the instrumentation description file [12] generated by SCALEA instrumentation system [11] into the
30
H.-L. Truong and T. Fahringer
Instrumentation Description File
OpenMP, MPI, HPF, Hybrid Programs
Profile/Trace Files
Post Processing
Overhead Analyzer Single Experiment Analysis
System benchmark
Experiment Data Repository Searching
Multi-Experiment Analysis External tools
Fig. 2. Components interacting with SCALEA’s Experiment Data Repository
data repository. In addition, it filters raw performance data collected for each experiment and stores the filtered data into the repository. The overhead analyzer performs the overhead analysis according to the overhead classification [11] based on filtered data in the repository (or trace/profile files). The resulting overhead is then stored into the data repository. The system benchmark is used to collect system information (memory, harddisk, etc), to determine specific information (e.g. overhead of probes, time to access a lock), and to perform benchmarks for every target machine of interest. By using available MPI and OpenMP benchmarks, reference values of system performance are obtained for both message passing (e.g. blocking send/receive, collective) and shared memory operations (e.g. lock, barrier) with various networks (e.g. Fast-Ethernet, Myrinet), libraries (e.g. MPICH, PGI OpenMP) on different architectures (e.g. Linux x86, Sun Sparc). The obtained results aim to assist the correlation analysis between application- and system-specific metrics. Based on data availability in the repository, various analyses can be conducted such as single- and multi-experiment analysis, searching. Moreover, data can be exported into XML format which further facilitates accessing performance information by other tools (e.g. compilers or runtime systems) and applications. The experiment data repository is implemented with PostgreSQL [8] which is a relational database system supported on many platforms. All components interacting with the data repository are written in Java and the connection realizing between these components with the database is powered by JDBC. 2.3
Experiment-Related Data APIs
Each table shown in Fig. 1 is associated with a Java class. In addition, we define two classes ProcessingUnit and ExperimentData. The first class is used to describe the processing unit where the code region is executed; a process unit consists of information about computational node, process, thread. The latter
On Utilizing Experiment Data Repository for Performance Analysis
31
implements interfaces used to access experiment data. Bellow, we just highlight some classes with few selected data members and methods: public class PerformanceMetric { public String metricName; public Object metricValue; ... } public class ProcessingUnit { ... public ProcessingUnit(String node,int process, int thread) {...} } public class RegionSummary { ... public PerformanceMetric[] getMetrics(){...} public PerformanceMetric getMetric(String metricN ame){...} } public class ExperimentData { DatabaseConnection connection; ... public ProcessingUnit[] getProcessingUnits(Experiment e){...} public RegionSummary[] getRegionSummaries(CodeRegion cr, Experiment e) {...} public RegionSummary getRegionSummary(CodeRegion cr, ProcessingUnit pu, , Experiment e) {...} ... }
Based on the well-defined APIs, external tools can easily access data in the repository, using the data for their own purpose. For example, a tool identif ied overhead can compute total execution time ratio for code region 1 in computational node gsr1.vcpc.univie.ac.at, process 1, thread 0 of experiment 1 based on identified overhead (denoted by oall ident) and total execution time (denoted by wtime) as follows: CodeRegion cr = new CodeRegion(‘‘region1’’); Experiment e = new Experiment(‘‘experiment1’’); ProcessingUnit pu = new ProcessingUnit(‘‘gsr1.vcpc.ac.at’’,1,0); ExperimentData ed = new ExperimentData(new DatabaseConnection(...)); RegionSummary rs = ed.getRegionSummary (cr,pu,e); PerformanceMetric overhead=rs.getMetric(’’oall ident’’); PerformanceMetric wtime =rs.getMetric(‘‘wtime’’); double overheadRatio=((Double)overhead.metricValue).doubleValue()/ ((Double)wtime.metricValue).doubleValue();
3 3.1
Achievements of Using Experiment Data Repository Search and Filter Capabilities
Most existing performance tools lack basic search and filter capabilities. Commonly, the tools allow the user to browse code regions and associated performance metrics through various views (e.g. process time-lines with zooming and
32
H.-L. Truong and T. Fahringer
scrolling, histograms of state durations and message data [14,3]) of performance data. Those views are crucial but mostly require all data to be loaded into the memory, eventually making the tools in-scalable. Moreover, the user has difficulty to find out the occurrence of events with interesting criteria of performance metrics, e.g. code regions with overhead of data movement [11] larger than 50% of total execution time. Search of performance data will allow the user to quickly identify interesting code regions. Filtering performance data being visualized helps to increase the scalability of performance tools. Utilizing a data repository allows to power the archive, to facilitate search and filter with great flexibility and robustness based on SQL language and to minimize the implementation’s cost.
Fig. 3. Interface for Search and Filter in SCALEA
Fig. 4. Specify complex performance conditions
Figure 3 presents the interface for search and filter in SCALEA. The user can select any experiment for searching code regions under selection criteria. For each experiment, the user can choose code region types (e.g. send/receive,OpenMP loop), specify metric constraints based on performance metrics (timing, hardware parameter, overhead) and opt the processing unit (computational nodes, processes, threads) on which the code regions are executed. Metric constraints can be made in a simple way by forming clauses of selection conditions based on available performance metrics (see Fig. 3). They can also be constructed by selecting quantitative characteristics (Figure 4). For instance, the user may define characteristics for L2 cache miss ratio by expressing L2 cache L2 cache misses miss ratio as rL2 cache miss ratio = L2 cache accesses and then discretizing the L2 cache miss ratio as follows: good if rL2 cache miss ratio ≤ 0.3, average if 0.3 < rL2 cache miss ratio < 0.7, and poor if rL2 cache miss ratio ≥ 0.7. These quantitative characteristics can be stored into the experiment data repository for later use. SCALEA will transfer user-specified conditions into SQL language, perform the search and
On Utilizing Experiment Data Repository for Performance Analysis
33
Fig. 5. Results of Performance Search
return the result. For example, the searching result based on the conditions in Fig. 3 is shown in Fig. 5. In the top-left window, the code region summaries met the search conditions are visualized. By clicking into a code region summary, the source code and corresponding performance metrics of the code region summary will be shown in the top-right and bottom window, respectively. 3.2
Multi-experiment Analysis
Most existing performance tools [5,3,13] investigate the performance for individual experiments one at a time. To archive experiment data in the repository, SCALEA goes beyond this limitation by supporting multi-experiment analysis. The user can select several experiments, code regions and performance metrics of interest of which associated data are stored in the data repository (see Figure 6). The outcome of every selected code regions and metrics is then analyzed and visualized for all experiments. SCALEA’s multi-experiment analysis supports: – performance comparison for different sets of experiments: The user can analyze the overall execution of the application across different sets of experiments; experiments in a set are grouped based on their properties (e.g. problem sizes, communication libraries, platforms). – overhead analysis for multi-experiment: Various sources of performance overheads across experiments can be examined. – study parallel speedup and parallel efficiency at both program and code region level: Commonly, those metrics are applied only at the level of the entire program. SCALEA, however, supports to examine the scalability at both program and code region level.
34
H.-L. Truong and T. Fahringer
Fig. 6. Interface for Multi-Experiment Analysis
3.3
Data Sharing and Tools Integration
One of the key reasons for utilizing the data repository is the need to support data sharing and tools integration. Via well-defined interfaces, other tools can retrieve data from the experiment data repository and use the data for their own purpose. For example, AKSUM [2] which is a high-level semi-automatic performance bottleneck analysis has employed SCALEA to instrument user’s programs, to measure and analyze performance overheads of code regions of the programs. AKSUM then accesses performance data (overheads, timing, hardware parameters) of code regions on the data repository, computes performance properties [1] and conducts the high-level bottleneck analysis based on these properties. In another case, the PerformanceProphet [7] which supports performance modeling and prediction on cluster architectures has used data stored in the experiment data repository to build the cost functions for the application model represented in UML forms. The model is then simulated in order to predict the execution behaviour of parallel programs. Performance data can also be exported into XML format so that it can easily be transfered and processed by other tools. For example, performance data of code region 1 can be saved in XML format as follows: <metric name=’’wtime’’ value=’’1.09039995E8’’ /> <metric name=’’odata_send’’ value=’’2986000.0’’ /> <metric name=’’odata_recv’’ value=’’5.6923546E7’’ />
where each performance metric of the region is represented as a tuple (name,value). Note that wtime (stands for wall clock time), odata send (overhead of send operations), odata recv (overhead of recv operations) are unique
On Utilizing Experiment Data Repository for Performance Analysis
35
performance metric names described in a performance metric catalog which holds information of performance metrics (e.g. unique metric name, data type, welldefined meaning) supported by SCALEA.
4
Further Directions
In this section, we discuss further directions on utilizing the experiment data repository for performance analysis: – A query language for performance data can be designed to support ad hoc and interactive searching and filtering for occurrence of events with criteria of performance metrics and/or performance problems in order to facilitate flexible and efficient discovery of interesting performance information. Such a language can be implemented on top of search facilities provided by database systems and be based on performance property language [1]. – Automatic scalable analysis techniques such as decision trees, rule associations, clusters, classifications should be exploited to discover the knowledge of performance information on the repository. These techniques are particularly useful for executing very complex queries on non-main-memory data. However, currently these techniques are rarely used in performance analysis due to lack of systematic organization of performance data. – Providing standardization APIs for acquiring and exploiting performance data is one of the keys to bring the simplicity, efficiency and success to the collaboration among tools. The well-defined APIs should be independent on the internal data representation and organization of each tool but based on an agreement of well-defined semantics.
5
Related Work
Significant works on performance analysis have been done by Paradyn [10], TAU [5], VAMPIR [3], EXPERT [13], etc. SCALEA differs from these approaches by storing experiment-related data to a data repository, and by supporting also multi-experiment performance analysis. In [4], information about each experiment is stored in a Program Event and techniques for comparison between experiments are done automatically. A prototype of Program Event has been implemented, however, the lack of capability to export and share performance data has hindered external tools from using and exploiting data in Program Events. Prophesy [9] provides a repository to store performance data for performing the automatic generation of performance models. Data measured and analyzed by SCALEA can be used by Prophesy for modeling systems. USRA Tool family [6] collects and combines information of parallel programs from various sources at the level of subroutines and loops. Information is stored in flat files which can further be saved in a format understood by spreadsheet
36
H.-L. Truong and T. Fahringer
programs. SCALEA’s repository provides a better infrastructure for storing, querying and exporting performance data with a relational database system. APART proposes the performance-related data specification [1] which stimulates our experiment-related data model. Besides performance-related data, we also provide system-related data.
6
Conclusions and Future Work
The main contributions of this paper are centered on the design and achievements of the experiment data repository in SCALEA which is a performance analysis tool for OpenMP/MPI and mixed parallel programs. We have described a novel design of SCALEA’s experiment data repository holding all relevant experiment information and demonstrated several achievements gained from the employment of the data repository. The data repository has increasingly supported the automation of the performance analysis and optimization process. However, employing the data repository introduces extra overheads in comparison with other non-employing-data-repository tools; the extra overheads occur in filtering and storing raw data to and retrieving data from the database. In the current implementation, we observed the bottleneck in accessing the data repository with large data volume. We are going to enhance our access methods and database structure to solve this problem. In addition, we intend to work on the issues discussed in Section 4.
References 1. T. Fahringer, M. Gerndt, Bernd Mohr, Felix Wolf, G. Riley, and J. Tr¨ aff. Knowledge Specification for Automatic Performance Analysis, Revised Version. APART Technical Report, Workpackage 2, Identification and Formalization of Knowledge, Technical Report http://www.kfa-juelich.de/apart/result.html, August 2001. 2. T. Fahringer and C. Seragiotto. Automatic search for performance problems in parallel and distributed programs by using multi-experiment analysis. In International Conference On High Performance Computing (HiPC 2002), Bangalore, India, December 2002. Springer Verlag. 3. Pallas GmbH. VAMPIR: Visualization and Analysis of MPI Programs. http://www.pallas.com/e/products/vampir/index.htm. 4. Karen L. Karavanic and Barton P. Miller. Experiment Management Support for Performance Tuning. In Proceedings of Supercomputing’97 (CD-ROM), San Jose, CA, November 1997. ACM SIGARCH and IEEE. 5. Allen Malony and Sameer Shende. Performance technology for complex parallel and distributed systems. In In G. Kotsis and P. Kacsuk (Eds.), Third International Austrian/Hungarian Workshop on Distributed and Parallel Systems (DAPSYS 2000), pages 37–46. Kluwer Academic Publishers, Sept. 2000. 6. Insung Park, Michael Voss, Brian Armstrong, and Rudolf Eigenmann. Parallel programming and performance evaluation with the URSA tool family. International Journal of Parallel Programming, 26(5):541–??, ???? 1998.
On Utilizing Experiment Data Repository for Performance Analysis
37
7. S. Pllana and T. Fahringer. UML Based Modeling of Performance Oriented Parallel and Distributed Applications. In Proceedings of the 2002 Winter Simulation Conference, San Diego, California, USA, December 2002. IEEE. 8. PostgreSQL 7.1.2. http://www.postgresql.org/docs/. 9. V. Taylor, X. Wu, J. Geisler, X. Li, Z. Lan, R. Stevens, M. Hereld, and Ivan R.Judson. Prophesy:An Infrastructure for Analyzing and Modeling the Performance of Parallel and Distributed Applications. In Proc.of the Ninth IEEE International Symposium on High Performance Distributed Computing (HPDC’s 2000), Pittsburgh, August 2000. IEEE Computer Society Press. 10. Paradyn Parallel Performance Tools. http://www.cs.wisc.edu/paradyn/. 11. Hong-Linh Truong and Thomas Fahringer. SCALEA: A Performance Analysis Tool for Distributed and Parallel Program. In 8th International Europar Conference(EuroPar 2002), Lecture Notes in Computer Science, Paderborn, Germany, August 2002. Springer-Verlag. 12. Hong-Linh Truong, Thomas Fahringer, Georg Madsen, Allen D. Malony, Hans Moritsch, and Sameer Shende. On Using SCALEA for Performance Analysis of Distributed and Parallel Programs. In Proceeding of the 9th IEEE/ACM HighPerformance Networking and Computing Conference (SC’2001), Denver, USA, November 2001. 13. Felix Wolf and Bernd Mohr. Automatic Performance Analysis of Hybrid MPI/OpenMP Applications. In Proceedings of the Eleventh Euromicro Conference on Parallel, Distributed and Network-based Processing (PDP-11), pages 13– 22. IEEE Computer Society Press, February 2003. 14. Omer Zaki, Ewing Lusk, William Gropp, and Deborah Swider. Toward scalable performance visualization with Jumpshot. The International Journal of High Performance Computing Applications, 13(3):277–288, Fall 1999.
Flexible Performance Debugging of Parallel and Distributed Applications Jacques Chassin de Kergommeaux1 , Cyril Guilloud2 , and B. de Oliveira Stein3 1
Laboratoire LSR-IMAG, B.P. 72, 38402 St. Martin d’H`eres Cedex, France
[email protected], http://drakkar.imag.fr/˜chassin 2 Laboratoire ID-IMAG, ENSIMAG Antenne de Montbonnot ZIRST, 51 avenue Jean Kuntzmann, 38330 Montbonnot Saint Martin, France
[email protected], http://www-id.imag.fr/˜guilloud 3 Departamento de Eletrˆ onica e Computa¸ca ˜o, Universidade Federal de Santa Maria – RS, Brazil.
[email protected] Abstract. The Paj´e approach to help performance debugging of parallel and distributed applications is to provide behavioral visualizations of their program executions to programmers. This article describes how Paj´e was designed to be usable for a variety of programming models. Trace recording and trace manipulation facilities can be generated from a description of the events to be traced. The visualization tool includes a trace-driven simulator reconstructing the behavior of the observed executions of the applications being debugged. The Paj´e simulator is generic and can be specialized by the types of the objects to be visualized, the representation of these objects and their relations.
1
Introduction
Performance debugging requires that programmers of parallel and distributed applications identify the bottlenecks, overheads or out of balance computations resulting in inefficient use of parallel resources by their programs. Tools supporting the identification of such problems should help programmers understand the runtime behavior of their applications. Of particular interest are behavioral visualization tools which ease the understanding of the interactions of processes and/or threads during parallel and distributed executions. Behavioral visualization tools reconstruct an execution of a debugged application from execution traces. Such tools need to adapt easily to the variety and continuous evolution of programming models and visualization techniques. This article describes the Paj´e tracing and visualization tools, aimed at easing performance debugging of parallel and distributed applications. These tools adapt to a variety of programming models. Traces are recorded by tracing functions, inserted in the observed applications and automatically generated by “Tumit1 ” from a description of the events to be recorded. The Paj´e visualization 1
Inuit word meaning animal track, track and trace being the same word in French.
H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 38–46, 2003. c Springer-Verlag Berlin Heidelberg 2003
Flexible Performance Debugging of Parallel and Distributed Applications
39
tool is generic, allowing users to describe what they would like to see and how this should be visualized. The next section discusses the notion of behavioral observation of executions of parallel and distributed applications. The following gives a brief overview of the user-guided generation of trace monitoring and manipulation functions. The genericity of the Paj´e visualization tool is then briefly recalled and exemplified. The two last sections survey rapidly related work and conclude the article.
2 2.1
Behavioral Observation of Parallel and Distributed Executions Monitoring Techniques
Most monitoring tools are either clock Monitoring driven or event driven [4] (see figure 1). Clock driven monitoring amounts to have Event driven the state of the observed system regis- Clock driven tered at periodical time intervals. This Timing Counting Tracing technique may be useless to identify the origins of some overheads of parallel programs: bottlenecks, communication or id- Fig. 1. Classification of monitoring ling times. Event driven monitoring is tools triggered by the advent of observable events. In the timing approach, the time spent in various parts of the observed program is measured. In the counting approach, the number of occurrences of the observed events is recorded in global performance indices. Tracing is done by recording each of the observed events in a performance trace. Each record is composed of several parameters describing the event (see §3.2). 2.2
Tracing and Behavioral Visualization
Event driven monitoring is intensively used for performance debugging of parallel and distributed applications. Counting and timing are used to compute global performance indices whose evolution can sometimes be related to locations (procedures) [7] or execution phases [1] of the observed applications so that programmers can identify where and when the time goes. However, to reconstruct the behavior of the observed execution, more information is required and implies the use of execution traces. Traces can be used to reconstruct the dynamic behavior of an observed application so that the activity of processes, threads and processing elements and their communications can be checked along the execution time axis [12] (see figure 6). 2.3
Versatile Visualization Tool
Behavioral visualization tools include a trace-driven simulator to reconstruct the behavior of traced executions. Such simulators are usually dedicated to a given
40
J. Chassin de Kergommeaux, C. Guilloud, and B. de Oliveira Stein
programming model. However, there exist various programming models that are used to program parallel or distributed applications [2,9,15,13]. In addition, these applications are often programmed using layered middleware systems; explaining performance problems may imply to observe their execution at several layers simultaneously, to relate a performance problem observed at one layer — MPI program for example — to the behavior of a lower abstraction layer — transport layer for example. Therefore it is necessary that behavioral visualization tools can be adapted to various programming models and able to reconstruct executions at several abstraction layers simultaneously. 2.4
The Paj´ e Approach
Paj´e is a trace-driven behavioral visualization tool combining interactivity, scalability, extensibility and genericity properties [3,5]. Interactivity allows users to move back and forth in time, to obtain more details on displayed objects or relate a visualization to a source program. Scalability is mainly provided by zooming functionalities in a fully interactive way so that application developers can zoom in from data aggregated along the space (groups of threads, of nodes) or time dimension to a detailed visualization of where the problem appears to be. Extensibility is the possibility of adding easily new functionalities to the tool; it is provided in Paj´e by a careful modular design. Genericity is the faculty allowing users to adapt the Paj´e visualization to various programming models by describing in the input data what they wish to visualize and how this should be done (see §4). Paj´e could already be used in a variety of domains such as performance debugging of parallel [3] and distributed applications and monitoring of clusters [5]. The latest developments concern the production of execution traces which should be simple for all programming models that can be visualized with Paj´e. The Tumit tool, described in the next section, allows tracing functions to be generated automatically as well as trace processing functions. The aim is to provide at least the same flexibility for monitoring as is provided by Paj´e for visualization.
3
Tumit
The aim of Tumit is to help middleware or application developers to easily generate trace recording and trace manipulation functions. “Classical” tracing problems such as global dating or intrusion limitation and compensation [4] are not dealt with in this section. 3.1
Software Tracing Instrumentation Techniques
Instrumentation is the insertion of code to detect and record the application events. It can be done at several possible stages: in the source code of the traced application or at compile-time or by instrumenting the middleware used by the
Flexible Performance Debugging of Parallel and Distributed Applications
41
application or by direct instrumentation of compiled object code [4]. Compiletime and object code instrumentation contradict the objective of being easy to adapt: the former requires to have access to the source code of the compiler and modify it while the latter is dependent on the operating system and the target platform. Instrumentation of communication libraries or runtime systems is not difficult for the developers of these middleware systems. However, if tracing functions are not included in the middleware, they have to be realized by the implementors of visualization tools or by their users; these people need to have access to the source codes of the middleware to be instrumented and also to update the instrumentation each time a new version of this middleware is released. For these reasons, it was decided not to realize another tracer for Paj´e but instead to assist users in the development of ad hoc tracing facilities. From a description of the events to be traced, Tumit generates a set of tracing functions. Calls to these functions are inserted in the applications or middleware systems to be observed. Tumit also provides a set of data manipulation facilities to transform local into global dates or convert the monitored data from one format into another one (see §3.2). To guarantee the maximum flexibility, format conversion functions are not directly provided but instead generated from a set of rewriting rules provided by the users (see §3.3).
3.2
Data Format
A trace is composed of event records. Each record contains at least the following information: type of the event, (physical) date of the event and process identification of the process having performed the event. Some records contain additional parameters of the traced event such as receiver (sender) identification and message length in case of message emission (reception). Traces are usually recorded in a compact binary format to avoid memory buffer saturation and frequent disk accesses likely to alter the behavior of the observed application. They can be later processed — for example to correct the offsets and drifts of local clocks with respect to a global one [10] —, sorted and converted in the input data format of a visualization tool. The visualization tool input data format is a textual self defined data format (see [6] and §4). In Paj´e, trace format conversion is considered inevitable and therefore assisted by Tumit. Trace format conversion might involve more than coding the same data in a different way. Suppose for example that a blocking receive is recorded — by a “semantic-aware” tracer — as a single event whose format includes two dates bounding the period when the emitting thread is blocked. When converting into the Paj´e visualization input format, this event has to be transformed in two different visualization events, corresponding to the blocking and unblocking of the thread; the reason is that Paj´e does not have notions of programming model semantics and has no notion of blocking receive.
42
3.3
J. Chassin de Kergommeaux, C. Guilloud, and B. de Oliveira Stein
Tumit Instrumentation Technique
Tumit performs direct code instrumentation: the instructions generating the event records must be inserted in the programs before compilation (by the user of Paj´e or by a preprocessor). Traced events are user-defined which provides complete flexibility in numbers and types of the recorded parameters. Tracing user-defined events requires specific functions to record these events and process the event records. In Tumit, these functions are automatically generated from events definitions. To illustrate the Tumit approach, we present examples extracted from the tracing and visualization of a small program using the Taktuk [11] communication library. Taktuk uses binomial tree topologies to launch parallel programs in logarithmic time on large-sized clusters of computers and to establish efficient communication networks. It can be seen as a communication library using two levels of parallelism: nodes and threads.
Fig. 2. Definition of events
Events definition. In this short example (see figure 2), four events are defined inside one level called COM: events are grouped in levels to ease the filtering of monitored data or allow users to mix traces recorded at different abstraction levels. For each event we define its name and for each user parameter, its name and type. The identification of the process (or thread) emitting the event has to be provided explicitly by the user since this identification is not standard, process or nodes in some cases, nodes and local thread identification in other cases. Stamps are used to match POST and RECV events.
Fig. 3. Example Tumit conversion rules
Flexible Performance Debugging of Parallel and Distributed Applications
43
Fig. 4. Definition of a hierarchy in a Paj´e trace
Generation of tracing functions. From each event definition, Tumit generates a tracing function which can then be called from the user source code. The execution of a tracing function includes three phases: binding to a buffer and allocate another one if necessary; building an event record containing the parameters appearing in the events’ definitions; if necessary, flushing the buffer to disk. Generation of trace manipulation and conversion functions. Trace files need to be processed in order to be usable. Tumit generates functions allowing event records to be read (written) from (into) trace files or to be displayed under textual form. These functions are used in the implementation of more complex functionalities such as correcting the local time-stamps of the events to obtain globally coherent dates [10], sorting or filtering event trace files. As explained in §3.2, format conversion is often required. Tumit uses re-writing rules definitions to generate trace conversion functions. Figure 3 shows an example where these rules are used to transform monitoring data into the Paj´e visualization tool format and produce the example of §4. The output format is described as a character string where escape characters (@ sign) indicate substitutions by the recorded values of parameters. The corresponding parameters are written after the string. A rule with two arrows means that two events will be produced in the converted trace.
4
Genericity of the Paj´ e Visualization Tool
The Paj´e visualization tool can be specialized for a given programming model. The visualization to be constructed from the traces can be described by the user, provided that the types of the objects appearing in the visualization are hierarchically related and that this type hierarchy can be described as a tree
44
J. Chassin de Kergommeaux, C. Guilloud, and B. de Oliveira Stein
(see figures 4 and 6). The Paj´e self-defined input data format [5] is based on two generic types: containers and entities. While entities can be considered as elementary visual objects, containers are complex objects, each including a (potentially high) number of entities. An entity type is further specified by one of the following generic types: event, state, link or variable. Each entity also has a value, which allows its identification among other entities of the same type. In the example of figure 4, three hierarchically related containers are defined, named Prog, Node and Thread. The Thread container is composed of entities of type S or ThreadState of the generic type “state” and will therefore be represented as rectangles showing the successive states of the thread. These states may have three possible values, ThreadWorking or PackingTakCom or ReceivingTakCom. Another entity type named Comm of type “link”, connecting threads, is defined and will be represented as an arrow with a source and destination. Monitoring data, recorded during the execution of a small Taktuk program, using trace functions generated by Tumit from the event definitions of §3.3, was converted, using the rewrite rules presented in §3.3, in the Paj´e data format defined above. A small excerpt of this Paj´e file is given in figure 5: it exemplifies the creation of two threads and the sending of a message from one thread to the other. The identification of each thread is obtained by concatenation of a global node identification with a thread identification local to the node.
Fig. 5. Excerpt of converted trace file
5
Related Work
VampirTrace and Vampir [12] allow users to obtain easily behavioral visualizations of parallel programs executions. However these tools are bound to a programming model such as MPI and cannot be easily adapted. Tau [14] is a very flexible and portable performance monitoring tool which records information using mainly the counting and timing techniques (see §2.1). The tracing facility which should be provided through the definition of “user events” was not yet
Flexible Performance Debugging of Parallel and Distributed Applications
45
time
Prog
PACK_MSG
POST
node1 thread1
Node
COMM
Thread State
Comm
node2 thread1
RECV
END_UNPACK
Fig. 6. Simple Paj´e type hierarchy and visualization of Taktuk communication events
available in Tau at the time Tumit was designed. In the Paraver visualization and analysis tool [8], users can assign semantics — within a programming model paradigm — to the types of event records to indicate how trace file should be visualized.
6
Conclusion and Future Work
The Paj´e approach for performance debugging is based on event tracing and behavioral visualizations of the observed executions. One of the objectives of Paj´e is to face the multiplicity of parallel and distributed programming models. From a description of events to be traced, Tumit generates event recording functions and trace manipulation programs. The Paj´e visualization tool can be adapted to various programming models. Future work includes experimenting with multi-level tracing and visualizations — mixing hardware counters, thread management, communications and high level parallel programming model — or adapting global clock implementation techniques to scale to hundreds of processing elements or more. Acknowledgements. Jacques Briat suggested several of the proposed techniques. This work is partly supported by INRIA Rhˆ one-Alpes via the LIPS project (joint project of INRIA and BULL).
References 1. B. Buck and J. K. Hollingsworth. An API for runtime code patching. Journal of SuperComputing Applications and High Performance Computing, 14(4):317–329, Winter 2000. 2. R. Chandra et al. Parallel Programming in OpenMP. Morgan Kaufmann, 2000. 3. J. Chassin de Kergommeaux, B. de Oliveira Stein, and P. Bernard. Paj´e, an interactive visualization tool for tuning multi-threaded parallel applications. Parallel Computing, 26(10):1253–1274, aug 2000.
46
J. Chassin de Kergommeaux, C. Guilloud, and B. de Oliveira Stein
4. J. Chassin de Kergommeaux, E. Maillet, and J.-M. Vincent. Monitoring parallel programs for performance tuning in cluster environments. In J. Cunha et al., editors, Parallel Program Development for Cluster Computing, volume 5, chapter 6, pages 131–150. Nova Science, 2001. 5. J. C. de Kergommeaux and B. de Oliveira Stein. Flexible performance visualization of parallel and distributed applications. Future Generation Computer Systems, 19(5):735–748, 2003. 6. B. de Oliveira Stein, J. Chassin de Kergommeaux, and G. Mouni´e. Paj´e trace file format. Technical report, ID-IMAG, Grenoble, France, 2002. http://wwwid.imag.fr/Logiciels/paje/publications. 7. L. DeRose and D. Reed. Svpablo: A multi-language architecture-independent performance analysis system. In Procs of the International Conference on Parallel Processing (ICPP’99), pages 311–318, Fukushima, Japan, September 1999. 8. European Center for Parallelism of Barcelona, http://www.cepba.upc.es/paraver. Paraver: Parallel Program Visualization and Analysis Tool, 2001. 9. M. P. I. Forum. MPI: A message passing interface standard. International Journal of Supercomputer Applications, 8(3/4), 1994. ´ Maillet and C. Tron. On Efficiently Implementing Global Time for Perfor10. E. mance Evaluation on Multiprocessor Systems. Journal of Parallel and Distributed Computing, 28:84–93, 1995. 11. C. Martin and O. Richard. Parallel launcher for cluster of PC. In Proceedings of ParCo 2001. Imperial College Press, London, 2001. 12. W. Nagel et al. VAMPIR: Visualization and analysis of MPI resources. Supercomputer, 12(1):69–80, 1996. 13. A. Pope. The CORBA reference guide. Addison Wesley, 1998. 14. S. Shende, A. D. Malony, et al. Portable profiling and tracing for parallel scientific applications using C++. In Proc. of SPDT’98: ACM SIGMETRICS Symposium on Parallel and Distributed Tools, pages 134–145, Aug. 1998. 15. Sun Microsystems Inc., http://java.sun.com/docs/books/tutorial/. The Java(tm) Tutorial, 2003.
EventSpace – Exposing and Observing Communication Behavior of Parallel Cluster Applications Lars Ailo Bongo, Otto J. Anshus, and John Markus Bjørndalen Department of Computer Science, University of Tromsø {larsab, otto, johnm}@cs.uit.no
Abstract. This paper describes the motivation, design and performance of EventSpace, a configurable data collecting, management and observation system used for monitoring low-level synchronization and communication behavior of parallel applications on clusters and multi-clusters. Event collectors detect events, create virtual events by recording timestamped data about the events, and then store the virtual events to a virtual event space. Event scopes provide different views of the application, by combining and pre-processing the extracted virtual events. Online monitors are implemented as consumers using one or more event scopes. Event collectors, event scopes, and the virtual event space can be configured and mapped to the available resources to improve monitoring performance or reduce perturbation. Experiments demonstrate that a wind-tunnel application instrumented with event collectors, has insignificant slowdown due to data collection, and that monitors can reconfigure event scopes to trade-off between monitoring performance and perturbation.
1
Introduction
As the complexity and problem size of parallel applications and the number of nodes in clusters increase, communication performance becomes increasingly important. Of eight scalable scientific applications investigated in [12], most would benefit from improvements to MPI’s collective operations, and half would benefit from improvements in point-to-point message overhead and reduced latency. The performance of collective operations has been shown to improve by a factor of 1.98 by using better mappings of computation and data to the clusters [2]. Point-to-point communication performance can also be improved by tuning configurations. In order to tune the performance of collective and point-to-point communication, fine-grained information about the applications communication events is needed to compute how the application behaves.
This research was supported in part by the Norwegian Science Foundation project “NOTUR”, sub-project “Emerging Technologies – Cluster”
H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 47–56, 2003. c Springer-Verlag Berlin Heidelberg 2003
48
L.A. Bongo, O.J. Anshus, and J.M. Bjørndalen
In this paper we describe EventSpace, an approach and a system for online monitoring the communication behavior of parallel applications. It is the first step toward a system that can dynamically reconfigure an applications communication structure and behavior. For low-level performance analysis [3,11] and prediction [14,5], large amounts of data may be needed. For some purposes the data must be consumed at a high rate [9]. When the data is used to improve the performance of an application at run-time, low perturbation is important [8,10]. To meet the needs of different monitors the system should be flexible, and extensible. Also the sample rate, latency, perturbation, and resource usage should be configurable [8,5,11]. Finally, the complexity of specifying the properties of such a system must be handled. The approach is to have a virtual event space, that contains traces of an applications communication (including communication used for synchronization). Event scopes are used by consumers to extract and combine virtual events from the virtual event space, providing different views of an applications behavior. The output from event scopes can be used in applications and tools for adaptive applications, resource performance predictors, and run-time performance analysis and visualization tools. When triggered by communication events, event collectors create a virtual event, and store it in the virtual event space. A virtual event comprises timestamped data about the event. The EventSpace system is designed to scale with regards to the number of nodes monitored, the amount of data collected, the data observing rate, and the introduced perturbation. Complexity is handled by separating instrumentation, configuration, data collection, data storage, and data observing. The prototype implementation of the EventSpace system is based on the PATHS [1] system. PATHS allows configuring and mapping of an applications communication paths to the available resources. PATHS use the concept of wrappers to add code along the communication paths, allowing for various kinds of processing of the data along the paths. PATHS use the PastSet [13] distributed shared memory system. In PastSet tuples are read from and written to named elements. In EventSpace, an application is instrumented when the configurable communication paths are specified. Event collectors are implemented as PATHS wrappers integrated in the communication system. They are triggered by PastSet operations invoked through the wrapper. The virtual event space is implemented by using PastSet. There is one trace element per event collector. Event scopes are implemented using scalable hierarchical gather trees, used to gather data from a set of trace elements. PATHS wrappers to filter, sort, convert, or reduce data can be added to the tree as needed. To trade-off between performance and perturbation the tree can be mapped to the available resources, and wrapper properties can be set. This paper proceeds as follows. In section 2 we discuss related work. The architecture and implementation of EventSpace are described in sections 3 and 4. Monitors using EventSpace are described in section 5. The performance of
EventSpace – Exposing and Observing Communication Behavior
49
EventSpace is evaluated in section 6. In section 7 we draw conclusions and outline future work.
2
Related Work
There are several performance analysis tools for parallel programs [6]. Generally such tools provide coarse grained analysis with focus on processor utilization [11]. EventSpace supplements these tools by providing detailed information about the internal behavior of the communication system. NetLogger [11] provides detailed end-to-end application and system level monitoring of high performance distributed systems. Analysis is based on lifelines describing the temporal trace of an object through the distributed system. EventSpace provides data for similar paths inside the communication system. However, data is also provided for joined and forked paths forming trees, used to implement collective operations and barriers. There are several network performance monitoring tools [5,7]. While these often monitor low level network data, EventSpace is used by monitors monitoring paths that are used to implement point-to-point and collective communication operations. Such a path may in addition to a TCP connection, have code to process the data, synchronization code, and buffering. JAMM [10] is a monitoring sensor management system for Grid environments. Focus is on automating the execution of monitoring sensors and the collection of data. In JAMM sensors generate events that can be collected, filtered and summarized by consumers using event gateways. In EventSpace, events are generated by sensors integrated in the communication system, and consumers collect, filter, and preprocess data using event scopes. Event scopes are implemented by extending the PATHS system, allowing the data flow to be configured at a fine granularity. The Network Weather Service [14] is a system service producing short-term forecast for a set of network and computational resources. In EventSpace monitoring is per application, hence data is only collected for the resources used by the application simplifying the resource usage control. EventSpace, and many other monitoring systems [5,8,11,14] separate producers and consumers, and allow intermediates to filter, broadcast, cache, and forward data. Most monitoring systems also have a degree of configurability [5, 8,11,14]. In EventSpace focus is on allowing all parts of the monitoring to be configured and tuned at a fine granularity.
3
EventSpace Architecture
The architecture of the EventSpace system is given in figure 1. An application is instrumented by inserting event collectors into its communication paths. Each event collector record data about communication events, creates a virtual event based on the data, and stores it in a virtual event space. Different views of the
50
L.A. Bongo, O.J. Anshus, and J.M. Bjørndalen
Parallel application
thread thread
thread
event collector event collector
virtual event space view
00 000 11 111 00 11 111111 000000 11 00 111 000 view
Monitors event scope reduce gather filter sort
ConsumerA
thread
event collector event collector
ConsumerB
Fig. 1. EventSpace overview.
communication behavior can be provided by extracting and combining virtual events provided by different event collectors. Consumers use an event scope to do this. An event collector record operation type, operation parameters, and start and completion times1 of all operations sent through it. Typically, several event collectors are placed on a path to collect data at multiple points. EventSpace is designed to let event collectors create and store events, with low overhead introduced to the monitored communication operations. Shared resources used to extract and combine virtual events are not used until the data is actually needed by consumers. We call this lazy event processing. By using lazy processing we can, without heavy performance penalties, collect more data than may actually be needed. This is important because we do not know the actual needs of the consumers, and we expect the number of writes to be much larger than the number of reads [9]. EventSpace is designed to be extensible and flexible. The event collectors and event scopes can be configured and tuned to trade off between introduced perturbation and data gathering performance. It is also possible to extend EventSpace by adding other event collectors, and event scopes. The communication paths used by event collectors and event scopes can also be instrumented and monitored. Consequently, a consumer can monitor its own, another consumer’s, or an event collectors performance and overhead.
4
EventSpace Implementation
The implementation of EventSpace is built on top of PATHS and PastSet. Presently, the monitored applications must also use PATHS and PastSet. PastSet is a structured distributed shared memory system in the tradition of Linda [4]. A PastSet system comprises a number of user-level tuple servers hosting PastSet elements. An element is a sequence of tuples of the same type. Tuples can be read from and written to the element using blocking operations. PATHS supports mapping of threads to processes, processes to hosts, specifying and setting up physical communication paths to individual PastSet elements, 1
Using the Pentium timestamp counter.
EventSpace – Exposing and Observing Communication Behavior
51
and insertion of code in the communication paths. This code is executed every time the path is used. A path is specified by listing the stages from a thread to a PastSet element. At each stage the wrapper type and parameters used to initialize an actual instance of the wrapper are specified. A wrapper is typically used to run code before and after forwarding the PastSet operation to the next stage in the path. Paths can be joined or forked forming a tree structure. This supports implementation of collective operations, barriers and EventSpace gather trees. The threads, elements, and all communication paths used used by an application are specified in a pathmap. It also contains information about which paths are instrumented, and the properties of event collectors and trace elements. The path specifications are generated by path generate functions. As input to these functions three maps are used: (1) An application map describing which threads access which elements. (2) A cluster map, describing the topology and the nodes on each cluster. (3) An application to cluster map, that describes the mapping of threads and elements to the nodes. Threads can use elements in an access and location transparent manner, allowing the communication to be mapped onto arbitrary cluster configurations simply by reconfiguring the pathmap. Presently, run-time reconfiguration of the pathmap is not implemented.
4.1
Event Collectors
When triggered, an event collector creates a virtual event in the form of a trace tuple, and writes it to a trace element. A virtual event space is implemented by a number of trace elements in PastSet. Each trace element can have a different size, lifetime, and be stored in servers locally or remotely from where the communication event took place. The trace tuple is written to a trace element using a blocking PastSet operation. As a result it is important to keep the introduced overhead low. If the trace element is located in the same process as the event collector (a local server), the write only involves a memory copy and some synchronization code. To keep the introduced overhead low, trace tuples are usually stored locally. Tuples can be removed either by explicit calls, or automatically discarded when the number of tuples is above a specified threshold (specified on a per element basis). Presently, for persistent storage some kind of archive consumers are needed. The recorded data is stored in a 36 byte tuple. Since write performance is important, tuples are stored in binary format, using native byte ordering. For heterogeneous environments, the tuple content can be parsed to a common format when it is observed. By using PATHS to specify the path from an event collector wrapper to a trace element, we can specify the location of the trace element, its size, and its properties.
52
L.A. Bongo, O.J. Anshus, and J.M. Bjørndalen
Presently, the amount of tracing cannot dynamically be adjusted as in [8,11]. However, when a consumer decides to start monitoring a part of the system, a backlog of collected events can be examined. 4.2
Event Scopes
An event scope is used to gather and combine virtual events providing a specific view of an applications communication behavior. It can also do some preprocessing on the virtual events. An event scope is implemented using a configurable gather tree. The tree is built using PATHS wrappers. Even if a tree can involve many trace elements with wrappers added at arbitrary places, the complexity of building and configuring it is reduced due to the hierarchical nature of views, resulting in a regular structure of the tree. For example, the Heartbeat view in figure 2a, comprises two node views, each comprising two thread views. The tree can be built hierarchically by creating similar sub-trees for each sub-view. Any number of event scopes can be dynamically built by different consumers using the applications pathmap as input. The desired performance and perturbation of a gather tree are achieved by mapping the tree to available resources and setting properties of the wrappers. Data can be reduced or filtered close to the source, to avoid sending all data over a shared resource such as Ethernet, or a slow Internet link. Also some data preprocessing can be done on the compute clusters, thereby reducing the load on the node where the consumer is running. Since data is gathered on-demand, shared resources such as CPU, memory and networks are only used when needed.
Thread1 view TE1 TE2 TE3 reduce NodeA
Thread2 view reduce
NodeB view
T1
Thread3 Thread4 view view
TE1
gather convert
reduce
reduce
gather proxy proxy convert gather NodeB Heartbeat NodeC (a)
T2
T3
T4
TE2 TE3
reduce
TE4
reduce
TE5 proxy NodeA
TE6 proxy
TE7 TE8 global reduce
Contri− bute view
Departure view NodeB Arrival view NodeC
(b)
Fig. 2. (a) Heartbeat gather tree. (b) Data collected for a reduce operation tree.
5
Monitors – Consumers of Virtual Events
In this section we describe several monitors that use event scopes to extract and process virtual events.
EventSpace – Exposing and Observing Communication Behavior
5.1
53
Heartbeat Monitor
The Heartbeat monitor has the task of establishing whether a thread has reached a stable state where no further progress can take place or not. The monitor uses an event scope as shown in figure 2a, to find the last virtual event for each thread of the application. The event scope extract the latest virtual event for every communication path used by a thread, and then selects the last of these events (“reduce”). The output (“gather”) from the event scope is one single timestamp for each thread representing the last event happening in each thread. After a set time the monitor use the event scope to reduce and gather new timestamps. These are compared with the previous values. If no change is detected the thread has had no progress. 5.2
Get Event Monitor
GetEventMon just consumes virtual events without further processing of the data. In [3] the data collected by GetEventMon is used for performance analysis and visualization of the behavior of an application. GetEventMon uses an event scope with a gather tree similar to Heartbeat’s, except that all reduce wrappers are replaced with gather wrappers. 5.3
Collective Operation Monitor
ColOpMon monitors the performance of MPI type collective operation, implemented and instrumented using PATHS. As these operations use broadcast, gather, scatter, and reduce trees, we collect data about the activity in the trees for later extraction from the virtual event space, and analysis by ColOpMon. ColOpMon is a multi-threaded distributed application, with its own pathmap. It has threads monitoring the performance of each internal tree node, and one thread monitoring the contribution times of threads participating in the collective operation. In figure 2b the different views for (a part of) a reduce operation tree are shown, where four threads, T1 - T4 on nodes A and B, do a partial reduce on each node before the global reduce on node C. The path is instrumented using event collectors that store events in trace elements TE1 - TE8. An internal node monitor thread gathers, does clock synchronization, and analyzes events from the departure and arrival views. A contribution time monitor gathers and analyzes events from the contribute view.
6
Experiments
To demonstrate the feasibility of the approach and the performance of the EventSpace prototype, we monitor a wind-tunnel application. The hardware platform comprise two clusters: 4W: Eight four-way Pentium Pro 166 MHz, 128 MB RAM. 8W: Four eight-way Pentium Pro 200 MHz, 2GB RAM.
54
L.A. Bongo, O.J. Anshus, and J.M. Bjørndalen
The clusters use TCP/IP over a 100 Mbps Ethernet for intra and inter-cluster communication. Communication to and from the 4W cluster goes through a twoway Pentium II 300 MHz with 256 MB RAM, while the 8W nodes are directly accessible. The wind-tunnel is a Lattice Gas Automaton doing particle simulation. We use eight 2N ×N matrices. Each matrix is split into 140 slices, which are assigned to 140 threads. Each thread uses 12 PastSet elements to exchange the border rows of its slices with threads computing on neighboring slices. We use three different problem sizes: large, medium and small. Large is the largest problem size that does not trigger paging on the 4W nodes, while medium and small are 1/2 and 1/8 of the size of large 2 . For the large problem size, border rows are exchanged approximately every 300 ms (and thus virtual events are produced at about 3.3 Hz). For medium and small, rows are exchanged every 70 ms and every 5 ms respectively. The discard threshold is set to 200 tuples for all 2240 trace elements. Less than 0.5 MB of memory on a 4W node is used for trace tuples. 6.1
Event Collecting Overhead
The overhead introduced to the communication path by a single event collector wrapper is measured by adding event collector wrappers before and after it. The average overhead, calculated using recorded timestamps in these wrappers, is 1.1 µs on a 1.8 GHz Pentium 4 and 6.1 µs on a 200 MHz 8W node. This is comparable to overheads reported for similar systems [8,11]. For large and medium, the slowdown due to data collection is insignificant. For small, the slowdown is 1.03. Further experiments are needed to determine if the insignificant and small slowdowns are due to unused cluster resources, and the effect of the overhead introduced by event collectors on other applications. 6.2
Event Scope Perturbation and Performance
In this section we document how reconfiguring an event scope can be used to trade-off between the rate at which virtual events can be observed for a given view, and the perturbation of monitored application. We measure how many times a monitor can pull, using an event scope, one tuple from all trace elements in a view during a run of the wind-tunnel. All monitors are run on a computer outside the cluster3 , if not otherwise stated. GetEventMon consumes events from a single trace element, a thread view with 14 elements, and a node view with 266 elements (19 thread views) with no slowdown for large, and 1.03 for small. Events are consumed at 2500 Hz, 1500 Hz, and 200 Hz respectively. 2 3
As the size of the matrices, N, increase the computation increase by O(N 2 ), and the communication by N. For large, N is 7680. A 1.8 GHz Pentium 4 with 2 GB RAM.
EventSpace – Exposing and Observing Communication Behavior
55
When consuming events from a multi-cluster view with 1948 trace elements (140 thread views), the wind-tunnel has no slowdown for large. However when using the small problem size, the event scope introduces a slowdown of 1.36. Using a different gather tree shape increases the observe rate from 58 Hz to 106 Hz, and the slowdown to 1.50. Reconfiguring the event scope to use less threads, reduces the slowdown to 1.07, and the observe rate to 25 Hz. When consuming from another multi-cluster view for large, with only two trace elements per thread, there is no difference in sample rates and slowdown when consuming events sequentially and concurrently. However, when more processing are added to the communication paths the concurrent version is faster, due to better overlap of communication and computation. The event scopes used by the collective operation monitor, ColOpMon, results in a slowdown of 1.17. We discovered that an event scope actually perturbs the wind-tunnel more than the computation intensive internal node monitoring threads running on the clusters. By reconfiguring the event scope to use less resources the slowdown is reduced to 1.08 (the observe rates are also decreased by about 50%). When running four monitors concurrently, the slowdown is about the same as the largest slowdown caused by a single monitor (ColOpMon). For the consumers running on the computer outside the cluster, observe rates are reduced by 10-50%. For the ColOpMon internal monitor threads running on the clusters, observe rates are unchanged.
7
Conclusions and Future Work
The contributions of this work are two-fold: (i) we describe the architecture and design of a tunable, and configurable framework for low-level communication monitoring, and (ii) we demonstrate its feasibility and performance. The approach is to have event collectors integrated in the communication system, that when triggered by communication events, create a virtual event that contains timestamped information about the event. The virtual events are then stored in a virtual event space from where they can be extracted by consumers using different event scopes. Low-level communication monitoring is implemented by adding event collection code to communication paths. The data is stored in a structured shared memory. The data is extracted, combined and pre-processed using configurable communication paths. The architecture and its implementation, allows consumers, producers, and data management issues to be clearly separated. This makes handling the complexity simpler. We have described several monitors using EventSpace, and given initial performance results. The results show that a wind-tunnel application instrumented with event collectors, has insignificant slowdown due to data collection. We have also documented how event scopes can be reconfigured to trade-off between monitoring performance and perturbation.
56
L.A. Bongo, O.J. Anshus, and J.M. Bjørndalen
Currently, we are using other types of applications and monitors to evaluate the performance and configurability of the system. We are also using EventSpace to analyze and predict the performance of MPI type collective operations, with the purpose of dynamically adapting these for better performance.
References 1. Bjørndalen, J. M., Anshus, O., Larsen, T., and Vinter, B. Paths - integrating the principles of method-combination and remote procedure calls for run-time configuration and tuning of high-performance distributed application. Norsk Informatikk Konferanse (2001). 2. Bjørndalen, J. M., Anshus, O., Vinter, B., and Larsen, T. Configurable collective communication in LAM-MPI. Proceedings of Communicating Process Architectures 2002, Reading, UK (2002). 3. Bongo, L. A., Anshus, O., and Bjørndalen, J. M. Using a virtual event space to understand parallel application communication behavior, 2003. Technical Report 2003-44, Department of Computer Science, University of Tromsø. 4. Carriero, N., and Gelernter, D. Linda in context. Commun. ACM 32, 4 (April 1989). 5. Dinda, P., Gross, T., Karrer, R., Lowekamp, B., Miller, N., Steenkiste, P., and Sutherland, D. The architecture of the Remos system. In Proc. 10th IEEE Symp. on High Performance Distributed Computing (2001). 6. Moore, S., D.Cronk, London, K., and J.Dongarra. Review of performance analysis tools for MPI parallel programs. In 8th European PVM/MPI Users’ Group Meeting, Lecture Notes in Computer Science 2131 (2001), Y. Cotronis and J. Dongarra, Eds., Springer Verlag. 7. http://www.caida.org/tools/taxonomy/. 8. Ribler, R. L., Vetter, J. S., Simitci, H., and Reed, D. A. Autopilot: Adaptive control of distributed applications. In Proc. of the 7th IEEE International Symposium on High Performance Distributed Computing (1998). 9. Tierney, B., Aydt, R., Gunter, D., Smith, W., Taylor, V., Wolski, R., and Swany, M. A grid monitoring architecture. Tech. Rep. GWD-PERF-16-2, Global Grid Forum, January 2002. (2002). 10. Tierney, B., Crowley, B., Gunter, D., Holding, M., Lee, J., and Thompson, M. A monitoring sensor management system for Grid environments. In Proc. 9th IEEE Symp. On High Performance Distributed Computing (2000). 11. Tierney, B., Johnston, W. E., Crowley, B., Hoo, G., Brooks, C., and Gunter, D. The NetLogger methodology for high performance distributed systems performance analysis. In Proc. 7th IEEE Symp. On High Performance Distributed Computing (1998). 12. Vetter, J. S., and Yoo, A. An empirical performance evaluation of scalable scientific applications. In Proceedings of the 2002 ACM/IEEE conference on Supercomputing (2002). 13. Vinter, B. PastSet a Structured Distributed Shared Memory System. PhD thesis, University of Tromsø, 1999. 14. Wolski, R., Spring, N. T., and Hayes, J. The network weather service: a distributed resource performance forecasting service for metacomputing. Future Generation Computer Systems 15, 5–6 (1999).
A Race Detection Mechanism Embedded in a Conceptual Model for the Debugging of Message–Passing Distributed Programs* Ana Paula Cláudio and João Duarte Cunha 1
Faculdade de Ciências da Universidade de Lisboa Departamento de Informática, Campo Grande, Edifício C5, Piso 1, 1700 LISBOA Portugal
[email protected] 2 Laboratório Nacional de Engenharia Civil Av. do Brasil, 101, 1799 LISBOA CODEX, Portugal
[email protected] Abstract. An object-oriented conceptual model for the debugging of message-passing distributed programs incorporates several debugging facilities: generation of a space-time diagram showing the progression of the execution being studied, detection of race conditions, detection of particular kinds of predicates and representation of causality cones. The focus of this paper is the race detection mechanism. The proposed mechanism comprises two steps: detection of pairs of receive events in the same process potentially involved in race conditions and verification of the legitimacy of the potential race condition. The mechanism relies on the analysis of two arguments, process id and message tag, in receive events and consume events, which are considered as distinct types of occurrences in the conceptual model.
1
Introduction
This paper describes a race detection mechanism embedded in a conceptual model for message-passing distributed programs. This model was conceived according to the object-oriented methodology and incorporates several debugging facilities. Due to race conditions the internal work of a distributed program may be nondeterministic, that is, two successive executions of the program with the same input behave differently. Therefore, a bug observed in one execution may vanish in a later one and a mechanism for detecting these conditions is of utmost importance. However, it should be stressed that the programmer may intentionally introduce race conditions in order to obtain improved performances. So, only the programmer is able to distinguish between intentional and non-intentional races. The conceptual model is the core of MPVisualizer, a debugging tool for message-passing programs that is capable of operating either in reexecution mode or in post-mortem mode. Besides the race detection mechanism, the conceptual model incorporates the following debugging facilities which are available in the tool: * This work has been partially funded by FCT(Portugal) under Project POSI/39351/SRI/2001 Mago2. H. Kosch, L. Böszörményi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 57–65, 2003. © Springer-Verlag Berlin Heidelberg 2003
58
A.P. Cláudio and J.D. Cunha
• Generation of a space-time diagram showing the progression of the execution being studied. • Detection of local predicates as well as disjunction and conjunction of local predicates. Local predicates are defined by the user and, as explained in [1], a specific class must be rewritten in order to prepare the tool for the detection of each local predicate. Once the local predicates are defined no further programming is needed for the detection of disjunctive and conjunctive predicates. • Representation of causality cones. The tool has the capability of adding to the space-time diagram a representation of the causality cone [2] or causal history [3] of an event previously selected by the user. The causality cone of an event contains all the events in the program, which can possibly have influenced the event. Conjunctive and disjunctive predicates are evaluated in the frontier of a causality cone. The frontier of a causality cone includes the last event belonging to the cone in each process. Detection of disjunctive and conjunctive predicates as well as the representation of causality cones did not exist in the previous version of MPVisualizer [1]. Also, the race detection mechanism has undergone a major improvement in the current version. In the next section a brief description of MPVisualizer is given. The race detection mechanism is thoroughly explained in section three. Section four is dedicated to the conclusions.
2
MPVisualizer
As mentioned above, the core of MPVisualizer is the conceptual model. Conceived according to the object-oriented methodology, this model comprises two groups of classes: kernel classes and graphical classes. Classes in the first group are independent from the message-passing software. Classes in the second group are subclasses of the classes in the first group and encapsulate all the programming details that depend on the graphical software. MPVisualizer builds, either in reexecution or in post-mortem mode, a concrete model of the execution being studied. This model is an instance of the conceptual model and is called the functional model. MPVisualizer has three components: the visualization engine, the reexecution mechanism and the graphical interface. The visualization engine contains the functional model, the entity responsible for its management, called manager, and a set of processes with special tasks, the collectors and the central process. Collectors, one per machine, receive data from local processes in the execution being studied and forward them to the central process, which is in charge of sending to the manager all the information, needed to build the functional model. The code executed by collectors depends on the message-passing software used by the program being studied; everything else in the visualization engine is independent of it. The reexecution mechanism is based on Instant Replay [4], a well known mechanism for shared memory communication that we have adapted for message-passing communication. The reexecution mechanism includes two phases: trace and replay. In the trace phase, minimal information is stored to minimize the inevitable intrusion caused by
A Race Detection Mechanism Embedded in a Conceptual Model for the Debugging
59
tracing activities. Although minimal, the stored information is sufficient to ensure that, during the replay phase, each process will consume the same messages, in the same order. The monitoring code has been inserted in the standard libraries of the message-passing software. Program code is kept untouched but linking is different in the trace and replay phases. In the trace phase, the program code must be linked with one modified library, the trace library; for the replay phase, it must be linked with a second modified library, the replay library. The routines in this last library force the re-execution to respect the causal order of the communication events recorded during the original execution. They also force the processes to send blocks of information to the local collector process. The graphical interface is composed of a menu bar and a drawing area where the space-time diagram is shown. Space-time diagrams are built in a bidimensional orthogonal referential: one of the axis corresponds to the continuous variable time and the other axis to the set of all the processes in the execution which is a discrete variable. Each process is represented by a line-segment parallel to the time axis and growing accordingly to the evolution of the process. Every kind of communication event has its own graphical symbol drawn over the line–segment of the corresponding process in order to signal its occurrence. An oblique line-segment connects each consume event to the corresponding send event. The diagram is consistent with the happened-before relation defined by Lamport [5]. By appropriate manipulation of the graphical symbols in the diagram, the user can profit from all the facilities offered by MPVisualizer.
3
Race Detection Mechanism
The race detection mechanism comprises two steps: detection of pairs of receive events in the same process potentially involved in race conditions and verification of the legitimacy of the potential race condition. The original mechanism [1] included also the same two steps. However, the first step has been improved in order to enable the identification of a broader spectrum of race conditions. The mechanism, embedded in the conceptual model, performs the detection of race conditions while the construction of the functional model is taking place. The description of the mechanism is divided in three subsections: architecture, implementation and mechanism flaws. 3.1
Architecture
According to Netzer and Miller [6]: “Two messages race if either could have been accepted first by some receive, due to variations in message latencies or process scheduling”. Therefore, the entities involved in a race condition are: a couple of messages sent to the same process and at least one receive event that consumes the message arrived in the first place. Sometimes, there is a second receive event involved, which may or may not consume the message arrived in the second place.
60
A.P. Cláudio and J.D. Cunha
The proposed mechanism relies on the analysis of two arguments, process id and message tag, in receive and consume events. In the conceptual model, receive and consume events are considered as distinct types of occurrences: • A receive event occurs when a process intends to consume a message; it can be either blocking or non-blocking. A process id and a message tag are specified; both values are integers and one or both can have the value “–1” which means “any”. We assume that this value is strictly reserved for this meaning. • A consume event occurs when a process consumes a message. An event of this kind is always preceded by a receive event. It contains the tag of the message and its sender id. Both values are different from “-1”. A consume event is always preceded by a receive event but, it is possible that a receive event is not followed by a consume event. Therefore, for any pair of receive events, four distinct cases can be identified: • RCRC: both receive events, either blocking or non-blocking, are followed by consume events. • RCR-: the first receive event, either blocking or non-blocking, is followed by a consume event. The same does not happen with the second receive event which is not immediately followed by a consume event. If this receive event is a blocking one the execution of the process remains suspended. • R-RC: the first receive event is not followed by a consume event while the second one is. This case is not possible when the first receive is blocking. • R-R-: none of the two receive events is followed by a consume event. This case is not possible when the first receive is blocking. If the first receive event is non-blocking and the second receive event is blocking, the execution of the process is suspended. Potential race conditions are considered as non-legitimate, i.e. spurious, when one of the following situations holds: • Both messages have the same origin. Assuming that communication channels are FIFO, two messages sent by the same process to an equal destination can not be racing with one another. • Consumption of the first message “happens before” the sending of the second one, in the sense introduced by Lamport [5]. These situations are called false race condition [6]. The race detection mechanism compares the pair of values (process id, message tag) of a receive event with the equivalent pair of a later receive event in the same 4 process. This comparison deals with 16=2 different combinations of values. Table 1 details all the sixteen different situations. For referencing purposes, each situation is identified in the first column by an integer ranging from 1 to 16. The second and third columns correspond to the arguments process id and message tag of the first receive event; the fourth and fifth columns correspond to the same arguments of the second receive event. The conceptual model assumes that every message carries its origin and its tag, thereafter called message.pid and message.tag, respectively. Every consume event contains the pair of values that corresponds to the consumed message. These values are useful in both detection steps; however they are essential for the second one, when the legitimacy of potential races is verified.
A Race Detection Mechanism Embedded in a Conceptual Model for the Debugging
61
Table 1. Sixteen combinations of pairs (process id, message tag) in two receive events. nº 1 2 3 4 5 6
recv1 pid1 pid1 -1 -1 pid1 pid1
tag1 -1 tag1 -1 tag1 -1
recv2 pid2 pid2 pid2 pid2 pid2 pid2
tag2 tag2 tag2 tag2 -1 -1
7
-1
tag1
pid2
-1
8 9 10 11 12 13 14 15
-1 pid1 pid1 -1 -1 pid1 pid1 -1
-1 tag1 -1 tag1 -1 tag1 -1 tag1
pid2 -1 -1 -1 -1 -1 -1 -1
-1 tag2 tag2 tag2 tag2 -1 -1 -1
16
-1
-1
-1
-1
Comments There is no race condition There is no race condition There is a potential race condition if tag1=tag2 There is a potential race condition There is no race condition There is no race condition There is a potential race condition if both messages are consumed and both have a tag equal to tag1 or if the message consumed in the first place has a process id equal to pid2 There is a potential race condition There is no race condition There is no race condition There is a potential race condition if tag1=tag2 There is a potential race condition There is no race condition There is no race condition There is a potential race condition if both messages have been consumed and if they have equal tags There is a potential race condition
In the set of situations listed on table 1 four subsets can be identified: • Subset 1: situations 1, 2, 5, 6, 9, 10, 13 and 14. In these situations there are no race conditions because the message consumed in the first place has to be the one sent by the process specified in the corresponding receive event parameter. • Subset 2: situations 4, 8, 12 and 16. These situations correspond to potential race conditions. Since the first receive event does not specify either the process id of the origin or the tag of the expected message, the message consumed first is the one that arrived first. The second message may or may not be consumed by the destination process. • Subset 3: situations 3 and 11. In these situations there are potential race conditions if message tags are the same in both receive events. • Subset 4: situations 7 and 15. In these situations there are potential race conditions if both messages have been consumed and both verify message.tag = tag1. Additionally, in situation number 7, a potential race condition also occurs if the message consumed in the first place verifies message.pid = pid2.
3.2 Implementation The following piece of pseudo-code explains the implementation of the first step in the mechanism. In the code, SRWHQWLDOBUDFH denotes a boolean variable, UHFYWDJ and FRQVSLG correspond, respectively, to the arguments process id and message tag of the first receive event, ∃FRQVdenotes aboolean expression that is truewhen the consumption of the first message has taken place; expressions UHFYWDJ, FRQVSLG and ∃FRQVhave similar meanings.
62
A.P. Cláudio and J.D. Cunha
SRWHQWLDOBUDFH IDOVH
LI UHFYSLG { the condition is true in situations 3, 4, 7, 8, 11, 12, 15 and 16 } WKHQLI UHFYWDJ { the condition is true in situations 4, 8, 12 and 16 } WKHQSRWHQWLDOBUDFH WUXH HOVH{ recv1.tag≠ -1 } LI UHFYWDJ UHFYWDJ { the condition is true in 3 and 11 } WKHQSRWHQWLDOBUDFH WUXH HOVH^recv1.tag≠ recv2.tag } LI UHFYWDJ { the condition is true in 7 and 15 } WKHQ LI ∃FRQV WKHQLI FRQVSLG UHFYSLG { false in 15 } WKHQSRWHQWLDOBUDFH WUXH HOVH{ cons1.pid≠ recv2.pid } LI ∃FRQV WKHQLI FRQVWDJ FRQVWDJ WKHQSRWHQWLDOBUDFH WUXH
Condition SLG≠is sufficient to guarantee that we are not dealing with a pair of receive events involved in a race condition. This condition excludes situations 1, 2, 5, 6, 9, 10, 13 and 14. The condition on line 3 is evaluated for the remaining eight situations. It is true in situations 4, 8, 12 and 16, whose first receive events do not specify either the process id, or the tag of the expected message. These situations are signaled as potential race conditions, whether or not any of the corresponding consumptions have occurred. The condition on line 6 is evaluated for situations 3, 7, 11 and 15. It is false for situations 7 and 15 and it can be true or false in situations 3 and 11. So, conditions 3 and 11 that satisfy the condition on line 6 are classified as potential race conditions. Condition UHFY.WDJ , on line 9, is false for situations 3 and 11. This fact excludes these situations from the subsequent tests. The condition is true for situations 7 and 15. Condition ∃FRQV, on line 10, is true for situations 7 and 15 corresponding to cases RCR- or RCRC. Within this group, situations satisfying the condition FRQVSLG UHFYSLG on line 11 are classified as potential race conditions. This condition is false for situation 15 and true for situations 7 similar to the one illustrated in figure 1, or when two messages with the same origin have been consumed. Both cases are marked as potential race conditions, but the last one is discarded during the verification process since it corresponds to a spurious race. Condition ∃FRQV, on line 14, is true for all RCRC cases in situation 15 and for situation 7 when condition FRQVSLG UHFYSLG does not hold In all these situations, there is a potential race condition when the consumed messages have the same tag, that is, when condition FRQVWDJ FRQVWDJ on line 15,holds. A brief explanation of how this code has been incorporated in MPVisualizer follows. The mechanism is embedded in specific methods of class Process, the kernel class modeling any individual process in the execution. In situations 4, 8, 12 and 16, the detection of potential race conditions takes place when the first receive event is notified; in situations 3, 11 and some situations of type 7, the detection takes place
A Race Detection Mechanism Embedded in a Conceptual Model for the Debugging
63
Fig. 1. Situation 7 of table 1. Dashed arrows correspond to messages that were not consumed.
when the second receive event is notified; in situations 15 as well as a different set of situations of type 7, the detection occurs when the second consume is notified. Therefore, detection code is embedded in methods EBUHFY, QEBUHFY and PBFRQVXPHG, which are invoked respectively when a blocking receive, a non-blocking receive and a consume event occurs. Whenever any of these methods detects a potential race condition, it invokes a private method of Process called QRWLI\BSRWHQWLDOBUDFH which then verifies if the condition is a legitimate race. Two receive events in the same process are classified as adjacent when no receive event occurred between them. In the current version of MPVisualizer, for efficiency reasons, detection of adjacent race conditions is performed automatically while detection of non-adjacent race conditions is performed only on demand, when the user selects the graphical symbol of a UHFY event. 3.3 Mechanism Flaws The mechanism is embedded in a conceptual model and therefore limited by the amount of information contained in the model. Since the model does not contain information about messages that arrived at a process but were not consumed, legitimization is not always feasible and it is possible to identify race conditions that are not detectable by the mechanism.
64
A.P. Cláudio and J.D. Cunha
The mechanism is not able to legitimate potential races, namely when the second consumption does not occur, either because the receiver process did not execute the second receive event that could lead to the consumption of the late message, or because it did execute the second receive but the late message could not be consumed since its characteristics did not match the arguments of that receive event. However, even when legitimization is not feasible, the tool signals the occurrence of a potential race condition. Potential race conditions not detectable by the tool are those that involve messages that were not actually consumed. For instance in a situation of type 7, similar to the one illustrated in figure 1a) but without message consumptions because both receive events are non blocking and both messages arrived too late to be consumed.
4 Conclusions A race detection mechanism is a valuable debugging facility. Nevertheless, it is relatively unusual to have this facility available in debugging tools. Out of 12 debugging tools compared with MPVisualizer, only 4 claim to detect race conditions. The tools compared with MPVisualizer were: Atemp (the debugging tool of MAD) [7]; DDBG (a debugging tool previously used by GRADE) [8]; DETOP (the debugging tool of TOOL-SET) [9]; DIWIDE (the debugging tool of WINPAR and P-GRADE) [10]; p2d2 [11]; Panorama [12]; the debugging tool of Parascope [13]; PDBG (the debugging tool of DAMS) [14]; POET [15]; TotalView [16]; Xmdb [17] and XPVM [18]. Among these tools and environments only MAD, POET, XMDB, and the debugging tool of Parascope are able to detect race conditions. The debugging tool of Parascope detects data races during execution and the compiler in the same environment is able to signal potential races. MAD is able to detect race conditions based on the analysis of trace files produced by the trace/replay mechanism. In POET the detection is performed by a component in charge of the trace/replay mechanism. Both MAD and POET allow the user to test a different order of communication events and to conclude if the reordering of communication events causes different results. We consider that this is a useful feature to include in a future version of MPVisualizer, combined with the possibility of incremental reexecution. Users of Xmdb can, a priori, list the race conditions classified as harmless; these are not signaled by the tool. MPVisualizer opens a pop-up window every time a race condition involving two adjacent receptions is detected. Depending on the program being studied, an overwhelming volume of information can be displayed. Therefore, future versions, will include filtering mechanisms to avoid this kind of effect. The race detection mechanism described is based on an exhaustive study of all possible combinations of the arguments process id and message tag in two receive events in the same process. The mechanism is not perfect since it is possible to identify race conditions that are not detectable. However, this flaw is due to the fact that the mechanism is embedded in a conceptual model and therefore limited by the amount of information contained in the model. Without a thorough reformulation of the conceptual model itself this has to be considered as a feature.
A Race Detection Mechanism Embedded in a Conceptual Model for the Debugging
65
References 1.
2. 3. 4. 5. 6. 7.
8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18.
Cláudio, A.P., Cunha, J.D. Carmo, M.B.: Monitoring and Debugging Message Passing th Applications with MPVisualizer. Proceedings of the 8 Euromicro Workshop on Parallel and Distributed Processing, IEEE Computer Society, pages 376-382, Rodhes, January 2000. M. Raynal. About Logical Clocks for Distributed Systems. ACM Operating System Review, 26(1): 41-48, 1992. C. Fidge. Partial Orders for Parallel Debugging. Proceedings of the ACM SIGPLAN/SIGOPS Workshop on Parallel and Distributed Debugging, ACM SIGPLAN Notices: 24(1), pages183-194, 1989. T. Leblanc, J. Mellor-Crummey. Debugging Parallel Programs with Instant Replay. IEEE Transactions on Computers, C-36(4), pages 471-482, April 1987. L. Lamport. Time, Clocks and the Ordering of Events in a Distributed System. Communications of the ACM, 21(7): 558-565, July 1978. R. Netzer, B. Miller. Optimal Tracing and Replay for Debugging Message-Passing Parallel Programs. Proceedings of Supercomputing’92, Minneapolis, USA, pages 502-511, November 1992. D. Kranzlmüller, R. Hügl, J. Volkert. MAD - A Top Down Approach to Parallel Program Debugging. Proceedings of HPCN’99, LNCS 1593, Springer-Verlag, pages 1207-1210, April 1999. J.C. Cunha, J. Lourenço, T. Antão. An Experiment in Tool Integration: The DDBG Parallel and Distributed Debugger. Euromicro Journal of Systems Architecture, 45(11):897-907, Elsevier Science Press, 1999. R. Wismuller et al.. The Tool-Set Project: Towards an Integrated Tool Environments for nd Parallel Programming. Proceedings of the 2 Sino-German Workshop on Advanced Parallel Processing Technologies, Koblenz, Germany, September 1997. http://www.lpds.sztaki.hu. http://science.nas.nasa.gov/Groups/Tools/Projects/P2D2. J. May, F. Berman. Designing a Parallel Debugger for Portability. Proceedings of the International Parallel Processing Symposium, 1994. K. Cooper et al.. The Parascope Parallel Programming Environment. Proceedings of the IEEE: 81(2), February 1993. J.C. Cunha, J. Lourenço, J. Vieira, B. Moscão, D. Pereira. A Framework to Support Parallel and Distributed Debugging. Proceedings of HPCN’98, LNCS 1401, SpringerVerlag, pages 707-717, April 1998. http://styx.uwaterloo.ca/poet/. http://www.pallas.de/pages/totalv.htm. http://www.lanl.gov/orgs/cic/cic8/para-dist-team/mdb/mdb.html. http://www.netlib.org/utk/icl/xpvm/xpvm.html.
DIOS++: A Framework for Rule-Based Autonomic Management of Distributed Scientific Applications Hua Liu and Manish Parashar The Applied Software Systems Laboratory Dept of Electrical and Computer Engineering, Rutgers University, Piscataway, NJ 08854, USA {marialiu,parashar}@caip.rutgers.edu
Abstract. This paper presents the design, prototype implementation and experimental evaluation of DIOS++, an infrastructure for enabling rule based autonomic adaptation and control of distributed scientific applications. DIOS++ provides: (1) abstractions for enhancing existing application objects with sensors and actuators for runtime interrogation and control, (2) a control network that connects and manages the distributed sensors and actuators, and enables external discovery, interrogation, monitoring and manipulation of these objects at runtime, and (3) a distributed rule engine that enables the runtime definition, deployment and execution of rules for autonomic application management. The framework is currently being used to enable autonomic monitoring and control of a wide range of scientific applications including oil reservoir, compressible turbulence and numerical relativity simulations.
1
Introduction
High-performance parallel/distributed simulations are playing an increasingly important role in science and engineering and are rapidly becoming critical research modalities. These simulations and the phenomena that they model are large, inherently complex and highly dynamic. Furthermore, the computational Grid, which is emerging as the dominant paradigm for distributed computing, is similarly heterogeneous and dynamic, globally aggregating large numbers of independent computing and communication resources, data stores and sensor networks. As a result, applications must be capable of dynamically managing, adapting and optimizing their behaviors to match the dynamics of the physics they are modeling and the state of their execution environment, so that they continue to meet their requirements and constraints. Autonomic computing [3] draws on the mechanisms used by biological systems to deal with such complexity, dynamics and uncertainty to develop applications that are self-defining, self-healing, selfconfiguring, self-optimizing, self-protecting, contextually aware and open. This paper presents the design, prototype implementation and experimental evaluation of DIOS++, an infrastructure for supporting the autonomic adaptation and control of distributed scientific applications. While a number of existing systems support interactive monitoring
Support for this work was provided by the NSF via grants numbers ACI 9984357(CAREERS), EIA 0103674 (NGS) and EIA-0120934 (ITR), DOE ASCI/ASAP (Caltech) via grant numbers PC295251 and 1052856.
H. Kosch, L. B¨osz¨orm´enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 66–73, 2003. c Springer-Verlag Berlin Heidelberg 2003
DIOS++: A Framework for Rule-Based Autonomic Management
67
and steering capabilities, few existing systems (e.g. CATALINA [4] and Autopilot [2]) support automated management and control. DIOS++ enables rules and polices to be dynamically composed and securely injected into the application at runtime so as to enable it to autonomically adapt and optimize its behavior. Rules specify conditions to be monitored and operations that should be executed when certain conditions are detected. Rather than continuously monitoring the simulation, experts can define and deploy appropriate rules that are automatically evaluated at runtime. These capabilities require support for automated and controlled runtime monitoring, interaction, management and adaptation of application objects. DIOS++ provides: (1) abstractions for enhancing existing application objects with sensors and actuators for runtime interrogation and control, (2) a control network that connects and manages the distributed sensors and actuators, and enables external discovery, interrogation, monitoring and manipulation of these objects at runtime, and (3) a distributed rule engine mechanism that enables the runtime definition, deployment and execution of rules for autonomic application management. Access to an object’s sensors and actuators is governed by access control polices. Rules can be dynamically composed using sensors and actuators exported by application objects. These rules are automatically decomposed, deployed onto the appropriate processors using the control network, and evaluated by the distributed rule-engine. DIOS++ builds on the DIOS [1], a distributed object substrate for interactively monitoring and steering parallel scientific simulations, and is part of the Discover 1 computational collaboratory. Discover enables geographically distributed clients to collaboratively access, monitor and control Grid applications using pervasive portals. It is currently being used to enable interactive monitoring, steering and control of a wide range of scientific applications, including oil reservoir, compressible turbulence and numerical relativity simulations.
2
DIOS++ Architecture
DIOS++ is composed of 3 key components: (1) autonomic objects that extend computational objects with sensors (to monitor the state of an object), actuators (to modify the state of an object), access policies (to control accesses to sensors and actuators) and rule agents (to enable rule-based autonomic self-management), (2) mechanisms for dynamically and securely composing, deploying, modifying and deleting rules, and (3) a hierarchical control network that is dynamically configured to enable runtime accesses to and management of the autonomic objects and their sensors, actuators, access policies and rules. 2.1 Autonomic Objects In addition to its functional interfaces, an autonomic object exports three aspects: (1) a control aspect, which defines sensors and actuators to allow the object’s state to be externally monitored and controlled, (2) an access aspect, which controls accesses to these sensors/actuators and describes users’ access privileges based on their capabilities, 1
http://www.discoverportal.org
68
H. Liu and M. Parashar
and (3) a rule aspect, which contains rules that can autonomically monitor, adapt and control the object. These aspects are described in the following subsections. A sample object that generates a list of random integers (RandomList) is used as a running example. The number of integers and their range are allowed to be set at run time. Control aspect. The control aspect specifies the sensors and actuators exported by an object. Sensors provide interfaces for viewing the current state of an object, while actuators provide interfaces for processing commands to modify the object’s state. For example, a RandomList object would provide sensors to query the current length of the list or the maximum value in the list, and an actuator for deleting the current list. Note that sensors and actuators must be co-located in memory with the computational objects and must have access to their internal state, since computational objects may be distributed across multiple processors and can be dynamically created, deleted, migrated and redistributed. DIOS++ provides programming abstraction to enable application developers to define and deploy sensors and actuators. This is achieved by deriving computational objects from virtual base object provided by DIOS++. The derived objects can then selectively overload the base object methods to specify their sensors and actuators interfaces. This process requires minimal modification to the original computational objects and has been successfully used by DIOS++ to support interactive steering. Access aspect. The access aspect addresses security and application integrity. It controls the accesses to an object’s sensors and actuators and restricts them to authorized users. The role-based access control model is used, where users are mapped to roles and each role is granted specific access privileges defined by access policies. The DIOS++ access aspect defines three roles: owner, member, and guest. Each user is assigned a role based on her/his credentials. Owner can modify access policies, define access privileges for members and guests, and enable or disable external accesses. Polices define which roles can access a sensor or actuator and in what way. Owners can also enable or disable a sensor or actuator. Access polices can be defined statically during object creation using the DIOS++ API, or can be injected dynamically by the owner at runtime via secure Discover portals. Rule aspect. The DIOS++ framework uses user-defined rules to enable autonomic management and control of applications. The rule aspect contains rules that define actions that will be executed when specified conditions are satisfied. The conditions and actions are defined in terms of the control aspect (e.g. sensors and actuators). A rule consists of 3 parts: (1) Condition part, defined by the keyword “IF” and composed of conditions which are conjoined by logical relationships (AND, OR, NOT etc.), (2) Then action part, defined by the keyword “THEN” and composed of operations that are executed when the corresponding condition is true, (3) Else action part, defined by the keyword “ELSE” and composed of operations that are executed when condition is not fulfilled. For example, consider the RandomList object with 2 sensors: (1) getLength() to get the current length of the list and (2) getMaxValue() to get the maximal value in the list, and 1 actuator append(length, max, min) that creates a list of size length with random integers between max and min, and appends it to the current list.
DIOS++: A Framework for Rule-Based Autonomic Management
69
IF RandomList.getLength() x”) for available quantities of stock larger than y (“ps availqty > y”). We have varied the query restrictions to obtain different behaviors of the same query. This leads to the four different parameter sets that we call query executions, shown in Table 1. It is frequent to have queries with several joins on a central table (called Star Joins) as it happens here on Lineitem. This does not mean that our strategy can only be applied to this type of queries (see Section 2 for the applicability of our strategy). However, this is a good example for the evaluation of our technique. Figure 2.b also shows (in bold) the application of the PDBF in this query. There, we can see how the Scan on Lineitem looks up the Bit Filters of HJ1, HJ2 and HJ3, while the Scan on Part only looks up the Bit Filter of HJ2.
Agg
select p_name, p_partkey sum(l_quantity) from lineitem, orders, partsupp, part where l_orderkey = o_orderkey and l_partsupp = ps_partsupp and l_partkey = ps_partkey and l_partkey = p_partkey and o_orderdate > "x" and ps_availqty > "y" and p_brand = "z" group by p_name, p_partkey;
(a)
Group by
Sort
(HJ1)
Hash Join
(HJ2) Hash Join
Scan Orders
(HJ3) Scan
Hash Join
Partsupp
Scan
Scan
Lineitem
Part
(b)
Fig. 2. Query used (a), execution plan and use of Bit Filters (b).
332
J. Aguilar-Saborit, V. Munt´es-Mulero, and J.-L. Larriba-Pey
Table 1. Percentage of tuples eliminated by each combination of parameters and efficiency of the Bit Filters. (P) for Part, (PS) for Partsupp and (O) for Orders. set # 1 2 3 4
Fraction of Tuples eliminated P.”z” PS.”y” O.”x” 0.0 0.7 0.8 0.0 0.7 0.15 0.8 0.15 0.15 0.0 0.15 0.15
Efficiency of Bit Filter HJ3 HJ2 HJ1 0.0 0.16 0.81 0.0 0.16 0.18 0.81 0.0 0.18 0.0 0.0 0.18
We have used restrictions ”x”, ”y” and ”z”, to control the amount of tuples projected from the build Relation of each Hash Join. Table 1 shows the fraction of tuples eliminated with each restriction for each different query execution. The restriction is one of the factors determining the effectiveness of a Bit Filter. The efficiency of the Bit Filters (fraction of tuples filtered out from the scan where the Bit Filter is applied) is also shown in Table 1 for each Hash Join. We have computed these figures using the data obtained after the execution of the query. Software Setup. We have generated a TPC-H database with scale factor 1 (1 GB) and executed the 4 queries on it. We have used PostgreSQL to implement the use of Bit Filters and PDBF. Therefore, we have used three versions of the PostgreSQL query engine; the original engine without Bit Filters, the engine with Bit Filters and the engine with PDBF. The size of the Bit Filters we use is larger than the number of distinct values of the attributes used to build the Filters. The implementations do not include any modification of the Optimization engine. Instead, they only implement the query plan and the modifications of PostgreSQL necessary to execute the query that we designed for the tests. PostgreSQL decides the number of partitions in the Hybrid Hash Join at optimization time. With the application of our strategy, some Hybrid Hash Join nodes end up having a very small number of tuples in the build Relation. This means that the number of partitions could be smaller than planned by the optimizer of PostgreSQL. However, our implementations keep the number of partitions planned by the optimizer at the cost of making our strategy less efficient. The memory space assigned to each Hash Join node is called Sortmem in PostgreSQL. For the following analysis, we use different Sortmem sizes ranging from very restrictive to a size that allows the execution of Hash Joins in memory with one only data read per Relation. The Sortmem sizes range from 5 MB (very restrictive) to 60 MB (one data read). The Bit Filters that we have implemented are always used unless we know its effectiveness is zero, in which case, they are not created. The Bit Filters used are stored in a Bit Filter Heap that we created. Hardware Setup We run all the experiments on a 550M Hz Pentium-III based computer, with 512 MB of main memory and a 20 GB IDE Hard Disk. The Operating System used was Linux Red Hat 6.2.
Pushing Down Bit Filters in the Pipelined Execution of Large Queries
333
Traffic and Execution Time Figure 3 shows results of execution time, and tuple traffic. The plots show results for the basic implementation of PostgreSQL, for the use of Bit Filters in the nodes where they are created, and PDBF. The execution time plots show results for different Sortmem sizes while the traffic plots show the total traffic per node in the execution plan (the traffic is independent from the Sortmem size). The traffic of a node is equivalent to the output cardinality of that node. It is important to note that only the traffic of nodes HJ3, HJ2, Scan Lineitem and Scan Part are affected by PDBF. While the execution time is specific to the PostgreSQL implementation, the traffic avoided depends on the data and can be regarded as a general result of applicability to any DBMS. We divide the analysis into two groups of queries. Query executions #1 and #2: These are the query executions that obtain better results both in execution time and tuple traffic. As shown in the plots of those query executions, the major improvements in the traffic are obtained by nodes HJ2, HJ3 and the Scan of Lineitem. This is because the Bit Filter created in HJ1 with Relation Orders has significant efficiencies in query execution #1 (see Table 1). When pushing down that Bit Filter, the Lineitem table is filtered out significantly, reducing the traffic in all its upper nodes. The plots show that the smaller the traffic, the larger the reduction in execution time. Another aspect to understand is the fact that the size of the Sortmem has an insignificant influence on query execution #1 for our strategy while it has a somewhat more significant influence on the execution time of executions #2. The reason is that the Bit Filter of HJ1 has a large efficiency (81%) in query execution #1 which reduces the Lineitem table to an extent that minimizes the I/O to the compulsory reads from disk even for small Sortmem sizes. One final aspect to note is that the executions on Sortmem sizes of 60 MB show the reduction in CPU effort of the total execution time. In the three PostgreSQL implementations, the size of the Sortmem reduces the amount of disk reads to the compulsory ones. It is important to note that, even in those cases, the amount of CPU time saved is significant which gives strength to our strategy. Query executions #3 and #4: These executions achieve good results but are not as significant as those in the previous query executions. They also show that PDBF are not a burden even when there is little benefit to obtain. Query execution #3 has a significant restriction on Part. This means that HJ3 only has the compulsory reads from disk because the build Relation fits in memory. Therefore, the output cardinality of HJ3 is very small in any of the three engines tested. Also, almost all the tuples filtered out by our strategy on Lineitem are caused by the Bit Filter created in HJ3 and the only work saved is that of projecting the tuples from the Scan of Lineitem up to HJ3, which implies a very little amount of execution time. Query execution #4 does not have a restriction on Part and it has very weak restrictions on Partsupp and Orders. Most probably, Bit Filters would not
334
J. Aguilar-Saborit, V. Munt´es-Mulero, and J.-L. Larriba-Pey 10
2000
Traffic among nodes (#1)
Execution time (#1)
Original
9
Original
bit filters
bit filters
PDBF
8
PDBF
1500
Elapsed time (sec)
Tuples (Millions)
7 6 5 4
1000
3 500 2 1 0
0 HJ2
HJ3
Lineitem
Part
5
10
Node 10
15
20
60
SortMem (Mbytes) 2000
Traffic across nodes (#2)
Execution time (#2)
Original
9
Original
bit filters
bit filters
deep bit filters
8
deep bit filters
1500
Elapsed time (sec)
Tuples (Millions)
7 6 5 4
1000
3 500 2 1 0
0 HJ2
HJ3
Lineitem
Part
5
10
Node 10
15
20
60
SortMem (Mbytes) 2000
Traffic across nodes (#3)
Execution time (#3)
Original
9
Original
bit filters
bit filters
deep bit filters
8
deep bit filters
1500
Elapsed time (sec)
Tuples (Millions)
7 6 5 4
1000
3 500 2 1 0
0 HJ2
HJ3
Lineitem
Part
5
10
Node 10
20
Execution time (#4)
Original
9
60
Original bit filters
bit filters
deep bit filters
2500
deep bit filters
8 7
2000
Elapsed time (sec)
Tuples (Millions)
15
SortMem (Mbytes) 3000
Traffic across nodes (#4)
6 5 4
1500
1000
3 2
500
1 0
0 HJ2
HJ3
Lineitem
Part
Node
5
10
15
20
60
SortMem (Mbytes)
Fig. 3. Execution time and traffic across nodes for executions #1, #2, #3 and #4. PDBF stands for Pushed Down Bit Filters.
be used in this case (the optimizer would not trade memory for an uncertain execution time benefit). I/O for Query Execution #1 Figure 3 shows that PDBF on query execution #1 improves the execution time of the basic and Bit Filter implementations of PostgreSQL by more than a 50%. Now we analyze the implications of the I/O on that query execution. Figure 4 shows the I/O performed by each of the three Hash Joins involved in query execution #1. We do not include the I/O for the Scan operations because
Pushing Down Bit Filters in the Pipelined Execution of Large Queries 1000
1400
I/O in HJ1 Original
335
I/O data in HJ2
1200
bit filters
800
PDBF
1000
I/O data (Mbytes)
I/O data (Mbytes)
Original 600
400
bit filters
800
PDBF
600
400 200 200
0
0 5
10
15
20
5
60
10
1000
15
20
60
SortMem (Mbytes)
SortMem (Mbytes)
I/O data in HJ3
Original
4500 Original
I/O total data
bit filters PDBF
4000
bit filters
800
3500
PDBF
I/O data (Mbytes)
I/O data (Mbytes)
3000 600
400
2500 2000 1500 1000
200
500 0 5
10
15
20
60
0
SortMem (Mbytes)
5
10
15
20
60
SortMem (Mbytes)
Fig. 4. I/O for query execution #1. PDBF stands for Pushed Down Bit Filters.
it does not change. The amount of I/O in Hash Join operations varies depending on the amount of data in the build Relations and the amount of memory available for the Hash Joins. If the amount of data in the build Relations fits in memory, the amount of I/O is zero in those operations, as shown in Figure 4. The plots show a significant improvement of more than a 50% of PDBF compared to the other implementations. There is no improvement of our strategy in HJ1 compared to the only use of Bit Filters. This is because the Bit Filter is in the topmost Join and the Bit Filter already reduces the probe table before partitioning it. The amount of I/O on HJ2 and HJ3 is reduced due to the fact that the Bit Filters are applied in the bottom leaves of the plan. Notice that as the SortMem size is incremented, the I/O of the basic and Bit Filters executions is reduced but is still very far from that of the I/O of our strategy. As shown in Figure 4, the execution on 60 MB Sortmem just uses the amount of IO necessary for compulsory reads, even for the original implementation in PostgreSQL. However, PDBF show a significant reduction of execution time (see Figure 3) because the amount of CPU work incurred by the tuple traffic saved is significant.
4
Related Work and Discussion
The use of Bit Filters is a popular strategy to support decision support queries. They have been used both in parallel Database management systems [7,12,13] and in sequential DBMSs [2] in order to speed-up the execution of joins. The original idea of the Bit Filter comes from the work by Bloom [1] where he designed them to filter out tuples without join partners.
336
J. Aguilar-Saborit, V. Munt´es-Mulero, and J.-L. Larriba-Pey
The most relevant related work is [4]. Its aim is also to reduce the traffic between query nodes in order to reduce the I/O and the CPU costs. However, this proposal has a couple of limitations that we overcome with our Pushed Down Bit Filters. First, the algorithm proposed in [4] creates Bit Filters whenever an attribute belonging to one Relation or to the outcome of an operation can filter out tuples from another Relation. This leads to additional work and an extra need for memory space. Our technique relies on the Bit Filters already created, as we have shown in the paper. This saves time and memory space. Second, as a consequence of the exhaustive creation of Bit Filters, the algorithm in [4] requires a non-pipelined execution. This is so because Bit Filters have to be created in full in order to be used. Thus, operations that create Bit Filters have to materialize their results. Materialization of nodes always incurs additional I/O. Our technique does not change the pipelined model of query execution. The pipelined execution of Hash Joins yields a better performance for multi-join queries as shown in [14], which also concludes that pipelined algorithms should be used in dataflow systems. In addition, most of the commercial DBMSs such as DB2 or Oracle use the pipelined execution philosophy to implement their DBMS engines [8]. Our Pushed Down Bit Filters require a small effort to be included in modern pipelined execution engines. Generalized hash teams also use Bit Filters with a different purpose than we do here [9]. The idea comes from the original idea of hash teams in [6]. The technique uses a cascaded partitioning of the input tables using bitmaps. The partitioning is aimed at reducing the amount of I/O during the joining process and on the final aggregation. The bitmaps are used as a partitioning device. Our approach is not limited to multi-joins with aggregation but to a more general class of joins and allows one table being filtered by as many Bit Filters as different join attributes are projected from that table. Generalized hash teams also have the limitation of false drops. When two different values map into the same Bit Filter position there may be false drops. False drops may cause a considerable amount of I/O in skewed environments. Our approach, given that it is not based on partitioning, does not have the false drops problem. Other lines of related work include the use of Bit Filters as bitmap indexes [3, 10,15]. In those cases, Bit Filters are used to perform joins in data warehousing contexts with the only use of bitmaps that reduce the I/O cost.
5
Conclusions
In this paper we propose PDBF, a strategy to use the Bit Filters created in the upper leaves of pipelined query execution plans, in the Scan nodes of the plan. We can outline the following conclusions for our strategy. First, our strategy reduces the amount of tuple traffic of the query plan significantly. This has a direct influence on the amount of work to be performed by the nodes. Second, as a consequence of the reduction of traffic, it is possible to reduce significantly the amount of I/O of some queries. Third, the direct consequence of the reduction
Pushing Down Bit Filters in the Pipelined Execution of Large Queries
337
of traffic and I/O is the reduction of execution time whenever the strategy can be used. The amount of benefit varies depending on whether it comes only from the traffic or both the traffic and the amount of I/O. Fourth, our strategy is simple to use. It just uses the Bit Filters already created by the Join nodes of the plan. PDBF require very simple run time data structures that only have to be managed by the Scan operations of the plan. As a consequence of the benefits that can be obtained and the low implementation cost, it is most advisable to use PDBF in modern pipelined DBMSs. Topics of future research related to PDBFs are optimization and scheduling issues and a further understanding of their applicability.
References 1. Burton H. Bloom. Space/time trade-offs in hash coding with allowable errors. Communications of the ACM, 13(7):422–426, 1970. 2. K. Bratbergsengen. Hashing methods and relational algebra operations. In Proc. of the Conf. on Very Large Data Bases (VLDB), pages 323–333, 1984. 3. C.-Y. Chan and Y. E. Ioannidis. Bitmap index design and evaluation. In Proc. of the SIGMOD Conf. on the Management of Data, pages 355–366, 1998. 4. M.-S. Chen, H.-I. Hsiao, and P. S. Yu. On applying hash filters to improving the execution of multi-join queries. VLDB Journal: Very Large Data Bases, 6(2):121– 131, 1997. 5. P. Gongloor and S. Patkar. Hash joins: Implementation and tuning, release 7.3. Technical report, Oracle Technical Report, March 1997. 6. G. Graefe, R. Bunker, and S. Cooper. Hash joins and hash teams in microsoft sql server. In Proceedings of the 25th VLDB Conference, pages 86–97, August 1998. 7. H.-I. Hsiao, M.-S. Chen, and P. S. Yu. Parallel execution of hash joins in parallel databases. IEEE-Trans. on Parallel and Distributed Systems, 8(8):872–883, August 1997. 8. Roberto J. Bayardo Jr. and Daniel P. Miranker. Processing queries for first few answers. In CIKM, pages 45–52, 1996. 9. A. Kemper, D. Kossmann, and C. Wiesner. Generalized hash teams for join and group-by. In Proc. of the Conf. on Very Large Data Bases (VLDB), pages 30–41, September 1999. 10. P. O‘Neil and G. Graefe. Multi-table joins through bitmapped join indices. SIGMOD Record, 24(3):8–11, September 1995. 11. PostgreSQL. http://www.postgresql.org/. 12. Donovan A. Schneider and David J. DeWitt. A performance evaluation of four parallel join algorithms in a shared-nothing multiprocessor environment. In Proc. ACM SIGMOD, pages 110–121, 1989. 13. Patrick Valduriez and Georges Gardarin. Join and semi-join algorithms for a multiprocessor database machine. TODS, 9(1):133–161, 1984. 14. A. N. Wilschut and P. M. G. Apers. Dataflow query execution in a parallel mainmemory environment. Distributed and Parallel Databases, 1(1):103–128, 1993. 15. M.-C. Wu and A. P. Buchmann. Encoded bitmap indexing for data warehouses. In Intl. Conference on Data Engineering, pages 220–230, 1998.
Suffix Arrays in Parallel Mauricio Mar´ın1 and Gonzalo Navarro2 1
2
University of Magallanes,
[email protected] University of Chile, Center for Web Research (www.cwr.cl)
[email protected].
Abstract. Suffix arrays are powerful data structures for text indexing. In this paper we present parallel algorithms devised to increase throughput of suffix arrays on a multiple-query setting. Experimental results show that efficient performance is indeed feasible in this strongly sequential and very poor locality data structure.
1
Introduction
In the last decade, the design of efficient data structures and algorithms for textual databases and related applications has received a great deal of attention due to the rapid growth of the Web [1]. To reduce the cost of searching a full large text collection, specialized indexing structures are adopted. Suffix arrays or pat arrays [1] are examples of such index structures. They are more suitable than the popular inverted lists for searching phrases or complex queries composed of regular expressions [1]. A fact to consider in parallel processing natural language texts is that words are not uniformly distributed both in the text itself and the queries provided by the users of the system. This produces load imbalance. The efficient construction in parallel of suffix arrays has been investigated in [3,2]. In this paper we focus on query processing. We propose efficient parallel algorithms for (1) processing queries grouped in batches of Q queries, and (2) load balancing properly this process when dealing with biased collections of words such as in natural language.
2
Global Suffix Arrays in Parallel
The suffix array is a binary search based strategy. The array contains pointers to the document terms, where pointers identify both documents and positions of terms within them. The array is sorted in lexicographical order by terms. A search is conducted by direct comparison of the terms pointed to by the array elements. A typical query is finding all text positions where a given substring appears in and it can be solved by performing two searches that locate the delimiting positions of the array for the substring. We assume a broker (server) operating upon a set of P machines running the BSP model [4]. The broker services clients’ requests by distributing queries onto
Funded by Millenium Nucleus Center for Web Research, Grant P01-029-F, Mideplan, Chile.
H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 338–341, 2003. c Springer-Verlag Berlin Heidelberg 2003
Suffix Arrays in Parallel
339
the P machines implementing the BSP server. We assume that under a situation of heavy traffic the server processes batches of Q = q P queries. A suffix array can be distributed onto the processors using a global index approach in which a single array of N elements is built from the whole text collection and mapped evenly on the processors. In this case, each processor stands for an interval or range of suffixes (lexicographic partition). The broker machine mantains information of the values limiting the intervals in each machine and route queries to the processors accordingly. This fact can be the source of load imbalance in the processors when queries tend to be dinamically biased to particular intervals. We call this strategy G0. In the local index strategy, on the other hand, a suffix array is constructed in each processor by considering only the subset of text stored in its respective processor. Unlike the global approach, no references to text postitions stored in other processors are made. However, for every query it is necessary to search in all of the processors in order to find the pieces of local arrays that form the solution for a given interval defined by the query. It is also necessary to broadcast every query to every processor. We have found both theoretically and experimentally that the global index approach offers the potential of better performance. A drawback of the global index approach is related to the possibility of load imbalance coming from large and sustained sequences of queries being routed to the same processor. The best way to avoid particular preferences for a given processor is to send queries uniformly at random among the processors. We propose to achieve this effect by multiplexing each interval defined by the original global array so that if the array element i is stored in processor p, then the elements i + 1, i + 2, ... are stored stored in processors p + 1, p + 2, ... respectively in a circular manner. We call this strategy G1. In this case, any binary search can start at any processor. Once a search has determined that the given term must be located between two consecutive entries k and k + 1 of the array in a processor, the search is continued in the next processor and so on, where at each processor it is only necessary to look at entry k of the local arrays. In general, for large P , the inter-processors search can be done in at most log P additional BSP supersteps by performing a binary search accross processors. The binary search on the global index approach can lead to a certain number of accesses to remote memory. A very effective way to reduce this is to associate with every array entry the first t characters of the terms respectively. The value of t depends of the average length of terms. This reduced remote accesses to less than 5% in [2], and less than 1% in our experiments, for relatively small t. In G0 we keep in each processor an array of P strings of size marking the delimiting points of each interval of G0. The broker machine routes queries uniformly at random to the P real processors, and in every processor a log P time binary search is performed to determine to which processor send a given query (we do so to avoid the broker becoming a bottleneck). Once a query has been sent to its target processor it cannot migrate to other processors as in the case of G1. That is, this strategy avoids the inter-processors log P binary search.
340
3
M. Mar´ın and G. Navarro
Experimental Results and Conclusions
We now compare the multiplexed strategy (G1) with the plain global suffix array (G0). For each element of the array we keep t characters which are the t-sized prefix of the suffix pointed to by the array element. We found t = 4 to be a good value for our text collection. We also implemented the local index strategy, but it was always at least 3 times slower than G0 or G1. The text collection is formed by a 1GB sample of the Chilean Web retrieved by the search engine www.todocl.cl. We treated it as a single string of characters, and queries were formed by selecting positions at random within this string. For each position a substring of size 16 is used as a query. In the first set of experiments these positions were selected uniformly at random. Thus load balance is expected to be near optimal. In the second set of experiments we selected at random only the positions whose starting word character were one of the four most popular letters of the Spanish language. This produces large imbalance as searches tend to end up in a subset of the global array. Experiments with an actual set of queries from users of “todocl” produced results between the uniform and biased cases. The experiments were performed on a PC cluster of 16 machines. Runs with more than 16 processors were performed by simulating virtual processors. At each superstep we introduced 1024/P new queries in each processor. Most speedups obtained against a sequential realization of suffix arrays were super-linear. This was not a surprise since due to hardware limitations we had to keep large pieces of the suffix array in secondary memory whilst communication among machines was composed by a comparatively small number of strings. The whole text was kept on disk so that once the first t chars of a query were found to be equal to the t chars kept in the respective array element, a disk access was necessary to verify that the string forming the query was effectively found at that position. With high probability this required an access to a disk file located in other processor, case in which the whole query is sent to that processor to be compared with the text retrieved from the remote disk. Though we present running time comparisons below, what we considered more relevant to the scope of this paper is an implementation and hardware independent comparison between G0 and G1. This came in the form of two performance metrics devised to evaluate load balance in computation and communication. They are the average maximum across supersteps. During the processing of a query each strategy performs the same kind of operations, so for the case of computation the number of these ones executed in each processor per superstep suffices as an indicator of load balance for computation. For communication we measured the amount of data sent to and received from at each processor in every superstep. We also measured balance of disk accesses. Table 1.1 shows results for queries biased to the 4 popular letters. Columns 2, 3, and 4 show the ratio G1/G0 for each of the above defined performance metrics (average maximum for computation, communication and disk access). These results confirm intuition, that is G0 can degenerate into a very poor performance strategy whereas G1 is a much better alternative. G1 is independent
Suffix Arrays in Parallel
341
of the application but, though well-balanced, it tends to generate more message traffic due to the inter-processors binary searches. The differences among G1 and G0 are not significant for the case of queries selected uniformly at random. G1 tends to have a slightly better load balance. Table 1. G1/G0 ratios. P 2 4 8
comp comm disk P comp comm 0.95 0.90 0.89 16 0.39 0.35 0.49 0.61 0.69 32 0.38 0.29 0.43 0.45 0.53 64 0.35 0.27 (1) Performance metrics
disk 0.36 0.24 0.17
P Biased Uniform 4 0.68 0.78 8 0.55 0.78 16 0.61 0.86 (2) Running times
As speed-ups were superlinear due to disk activity, we performed experiments with a reduced text database. We used a sample of 1MB per processor which reduces very significantly the computation costs and thereby it makes much more relevant the communication and synchronization costs in the overall running time. We observed an average speed-up efficiency of 0.65. Table 1.2 shows running time ratios obtained with our 16 machines cluster. The biased workload increased running times by a factor of 1.7 approximately. In both cases G1 outperformed G0, but G1 loses efficiency as the number of processors increases. This is because, as P grows, the effect of the inter-processor binary searches becomes more significant in this very low-cost computation scenario. Note that the above G0 strategy can be extended to approximate G1 by partitioning the array in V = k P virtual processors and mapping the V pieces of the array in a circular manner on the P actual processors. Preliminary experiments show that this strategy tends to significantly reduce the imbalance of G0 at a small k value. Yet another method which solves both load imbalance and remote references is to re-order the original global array so that every element of it contains only pointers to local text (or most of them). This becomes similar to the local index strategy whilst it still keep global information which avoids the P parallel binary searches and broadcast per query. Unfortunately we now lose the capability of performing the inter-processors log P -cost binary search. We are currently investigating ways of performing this search efficiently using little extra space.
References 1. R. Baeza and B. Ribeiro. Modern Information Retrieval. Addison-Wesley., 1999. 2. J. Kitajima and G. Navarro. A fast distributed suffix array generation algorithm. In 6th Symp. String Processing and Information Retrieval, pages 97–104, 1999. 3. G. Navarro, J. Kitajima, B. Ribeiro, and N. Ziviani. Distributed generation of suffix arrays. In 8th Symp. Combinatorial Pattern Matching, pages 102–115, 1997. LNCS 1264. 4. L.G. Valiant. A bridging model for parallel computation. Comm. ACM, 33:103–111, Aug. 1990.
Revisiting Join Site Selection in Distributed Database Systems 2
Haiwei Ye1, Brigitte Kerhervé , and Gregor v. Bochmann 3 1
Département d’ IRO, Université de Montréal, CP 6128 succ Centre-Ville, Montréal Québec, Canada H3C 3J7
[email protected] 2 Département d'informatique, Université du Québec à Montréal, CP 8888, succ Centre-ville, Montréal Québec, Canada H3C 3P8
[email protected] 3 School of Information Technology & Engineering, University of Ottawa P.O. Box 450, Stn A, Ottawa Ontario, Canada K1N 6N5
[email protected] Abstract. New characteristics for e-commerce applications, such as highly distributed data and unpredictable system nature, require us to revisit query processing for distributed database systems. As join operations involve relations over multiple sites, the site to carry out the join operation can have a significant impact on the performance. In this paper, we propose a new join site selection method. Our method is different from traditional methods in the following ways. First, the candidate sites are not restricted to the operand sites or query site as adopted by most database systems. Second, it considers dynamic system properties such as available bandwidth and system loads. We also conducted some experiments to offer a comparison between our method and the traditional one. The results show the advantage of our method under various system statuses.
1 Introduction Given the explosive growth of data over the Internet and prevalence of web-based applications such as e-commerce, how to manage large volumes of data and provide fast and timely answers to queries become more challenging in today’s information systems. Data are usually distributed across multiple locations in a distributed database system for the purpose of scalability and availability. As a result, a database query issued against a distributed database system generally requires retrieval of data from several locations to compute the final answer. Typical operations to combine the data from different locations are binary operations such as join. Even if techniques such as semi-joins have been introduced to reduce the cost of queries [1], where to perform the join is still of interest since it does have a great impact on the overall system performance, especially in today’s Internet environment. The research dedicated to join site selection has not received adequate attention. Existing approaches of H. Kosch, L. Böszörményi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 342–347, 2003. © Springer-Verlag Berlin Heidelberg 2003
Revisiting Join Site Selection in Distributed Database Systems
343
join site selection do not consider the dynamic nature of the Internet environment. We propose an approach to integrate user-defined QoS requirements into a distributed query processing environment [2]. Join site selection is regarded as one of the essential steps in our QoS-aware query processing. In a recent survey [3], Kossmann summarized three site selection strategies for client-server architectures. Depending on whether to move the query to the data (execution at servers) or to move the data to the query (execution at clients), the strategy is called query-shipping or data-shipping. A hybrid approach of these two solutions is also possible. Traditionally [4, 5], two types of strategies were proposed: Move-Small and Query-Site. In the query-site strategy, all joins are performed at the site where the query was submitted. In the move-small strategy, for each join operation, the smaller (perhaps temporary) relation participating in the join is always sent to the site of the larger relation. Selinger and Adiba suggested another option in [5]. They mentioned that for a join of two given tables at different sites, they could move both tables to a “third” site yet to be specified. However, this “third” site strategy has not been completely studied and not been adopted by commercial database systems. The above strategies for join site selection in a distributed environment are inadequate in the sense that they are only suitable in a static environment where network performance and load of the database server are fixed and predictable. This is obviously not the case for today’s highly distributed and dynamic systems. In this paper, we address the issue of join site selection in distributed multidatabase systems deployed for e-commerce applications. We propose an approach where dynamic system properties such as the performance of the Internet and the load of the server are taken into account in cost-based decisions for join site selection. In the next section we describe our approach and we provide a performance analysis in Section 3.
2 Considering a Third-Site in Join Site Selection In order to select the site where the join operation will be processed, we first build the set of candidate sites where we consider both operand sites as well as possible third sites. In this set, we then select the optimal site. This decision is based on the cost model we propose, where dynamic system properties are considered. A QoS monitoring tool is used to periodically collect system status information and to provide dynamic system properties to the query optimizer. 2.1 Building the Set of Candidate Sites The key issue in site selection is to decide which site is the best (optimal) for each binary operator. The site selection process becomes complicated when several candidate sites are capable of handling the operator. In fact, the crucial question is how many candidates should be considered for the third site. There are three possible approaches to determine candidate sites:
344
H. Ye, B. Kerhervé , and G. v. Bochmann
1) Consider all the available sites in the system. This is simple but this will usually incur too much overhead for the optimizer. 2) We can shrink the above set to all the sites involved in this particular query. By considering these sites, we may benefit from the situation where the result of this join needs to be shipped to the selected third site for further calculation. However, if the number of locations involved in the query is large, we may have the same problem as above: too much optimization overhead. 3) We can apply some heuristics to further decrease the size of the candidate sites set. For example, we can restrict the third site for a particular join operator to its “close relatives”, such as niece/nephew sites in the join tree. In our approach, we combine options 2 and 3. A threshold for the number of sites is therefore used to describe the situation where option 3 should be used. That is, All the sites in the tree,
if N < =threshold
Candidate sites for a join = Close relative (children and niece), otherwise
where N is the total number of sites involved in the query. The value of threshold should be derived from the experiment. In the following algorithm, we use a procedure CandidateSitesSelection() to represent this procedure.
2.2 Algorithm The procedure of join site selection can be regarded as marking the site for each join node in the tree, usually done in a bottom-up fashion. Thus we can employ one of the standard bottom-up tree traversal algorithms for this purpose. In our algorithm, we use post order tree traversal to visit the internal nodes of the tree. We ignore the post order algorithm and only give the algorithm (SiteSelection()) to visit each node. Algorithm: SiteSelection (treenode) 1. { 2. if (hasChild(treenode) == true ) { 3. candidate_set[] = CandidateSitesSelection (treenode); 4. for each site_s in candidate_set 5. cost [s] = Cost-node (treenode, site_s);//Use the cost model to compute the cost if the join is performed on this site 6. min_site = select-min (cost[]); 7. treenode.join_site = min_site; 8. if (site_s is also marked as join site for another node m in treenode) 9. Cost-node (m, site_s); //recalculate cost for node m under the new added load introduced by choosing site_s 10. } // end if 11. }
The procedure of SiteSelection() picks up the join site based on the cost model and records the join site in the root node of the input tree. The procedure of SiteSelection() picks up the join site based on the cost model and records the join site in the root node of the input tree. In the algorithm (Line 8 and 9), we consider the impact of this site selection on other nodes of the tree. We then recalculate the cost for node m, taking
Revisiting Join Site Selection in Distributed Database Systems
345
into account the additional load introduced by choosing site_s. Due to the optimization overhead that would increase if we recursively consider the impact, in the current algorithm, we do not check whether the site selection is still optimal. This is part of our future work. 2.3 Cost Model and QoS Information The Cost-node() implements the cost model which provides an evaluation of the cost for each node in the query execution plan and enables the adaptation to the changing conditions in distributed systems. The cost function includes two major parts: the local part and the network part. The following formulas define our cost models to execute a join at a candidate site s:
Cost-node(treenode, site_s) = Local_cost site_s + Network_cost; Local_cost site_s = join-cost (node.left, node.right, load site_s); Network_cost =max{ship-cost (node.left, site_s), ship-cost (node.right, site_s)}; ship-cost (node, site_s) = node.data / bwd.site i, site_s + delay site i, site_s; where bwd node.site, i and delay node.site, i represent the available TCP bandwidth and delay, respectively, from the site of the node to site i. The accuracy of our cost models relies on the up-to-date information of the current system status. Therefore, to keep track of the current dynamic performance information about the underlying network and database server, we use the QoS monitoring tools developed in our work [6].
3 Experimentation In this section, we evaluate the performance of our join site selection strategy according to the framework proposed in the previous section. The objective of our experiment is to show that our strategy can adapt to workload changes (both server load and network load) and always provides a better performance as compared to traditional strategy. For the purpose of comparison, we also implement the traditional join site selection strategy where the chosen site is always the site where the larger table resides; we call this strategy “larger-site” strategy. Two types of system loads are used for our measurement: network load and server load. For network load, we mainly focus on the available bandwidth as the indication of network congestion level. For server load, we concentrate on the CPU utilization as the indication of server load. We observed the TCP traffic using IPERF [7]. As a result, the network congestion levels are classified into 6 levels: level 0 (no congestion) to level 5 (highest congestion). Concerning the server load, we degrade the performance of one server by loading it with additional processes. Each process simply eats up CPU and competes with the database system for CPU utilization. Additional load is
346
H. Ye, B. Kerhervé , and G. v. Bochmann
quantified by the number of these processes spawned on a server. We categorize this load into 4 levels: no load, low load, medium load, and high load. It should be noted that as an experimental prototype, our execution engine was designed for ease of implementation and was not tuned for performance. The main purpose is to demonstrate the feasibility of our ideas. We conducted a number of experiments on two-way join and collected performance data for the two sets of experiments identified previously. Figure 1 provides a comparison of various loads. 3rd-site vs larger site, changing server load no load (all users)
6,2
3rd-site (low load) 3rd-site (med load)
response time (sec)
5,2 3rd-site (high load) larger-site (low load)
4,2
larger-site (medium load)
3,2
larger-site (high load)
2,2 1,2 0,2 0
10000
20000
30000
40000
50000
60000
70000
resulting cardinality
Fig. 1. Third-site vs. Larger-site with various server loads
The number of records in the resulting table (noted as resulting cardinality) varied from 180 to 3075. In the experiment, the network condition is the same for the entire load test, and the result in Figure 1 is collected while the network bandwidth is 5Mbps for all links. We only load the servers taking part in the joins and the rest remains unloaded. This open up the possibility to ship joined tables to other nodes for the purpose of performance gain. We also conducted some experiments to compare the performance under different network congestion levels. In these tests, we assume that the links among the nodes involved in the join are congested while other links have the normal throughput (5Mbps). In addition, there is no load of the server during the experimental periods. The results show that, under the no load circumstance, the two strategies get the same performance. With the increasing load, the “third-site” selection strategy almost provides the same performance, while the “larger-site” strategy will experience higher response time. The advantage of third-site strategy increases with the increasing of server load. Traditional site selection fixes the join site to the larger site. As the network congestion level increases, the data transfer time will increase too, which will in turn affect the response time of the whole query. However, for the third site selection,
Revisiting Join Site Selection in Distributed Database Systems
347
when the network congestion level reaches a certain point (usually at level 3, called shifting point), the algorithm will suggest to avoid the congested link and ship the two tables to a third site. This also explains why the response times (for the third-site algorithm) after the shifting point remain the same. In the performance study of the site selection problem, our third-site strategy not only reduces the response time but also achieves good load balancing among different database servers. According to our algorithm, if a server is heavily loaded, then the cost to perform a join operation on that server might be higher. This leads to the optimizer to avoid using that database server. Both sets of experiments show the superiority of our “third-site” selection algorithm: it can pick up the fast response plan under different system conditions. In addition, it achieves good load balancing among different database servers. According to our algorithm, if a server is heavily loaded, then the cost to perform a join operation might be higher leading the optimizer to avoid using that server.
4 Conclusion In this paper, we have proposed a new join site selection strategy for our QoS-aware distributed query processing environment. The strategy is based on cost models which integrate dynamic system properties. The candidate sites set considered in our approach is not restricted to the two operand sites as used in the traditional way. For the moment, we focus on the performance aspect of the QoS parameters provided by the QoS monitor. In the future, we will consider other QoS parameters, such as money cost or data quality.
References 1. 2.
3. 4. 5. 6. 7.
Stocker, K., Kossmann, D., Braumandl, R., Kemper, A.: Integrating Semi-Join-Reducers into State of the Art Query Processors. ICDE 2001, pp. 575–584. Ye, H., Kerhervé, B., Bochmann, G.v., Oria, V.,: Pushing Quality of Service Information and Requirements into Global Query Optimization, the Seventh International Database Engineering and Applications Symposium (IDEAS 2003), Hong Kong, China, July 16–18 Kossmann, D.: The state of the art in distributed query processing, ACM Computing Surveys (CSUR), Volume 32, Issue 4, December 2000, pp 422–469. Cornell, D. W., Yu, P. S.: Site Assignment for Relations and Join Operations in the Distributed Transaction Processing Environment. ICDE 1988, pp. 100–108. Selinger P. G., Adiba, M.:Access path selection in distributed data base management systems. In Proc. of the International Conference on Data Bases, 1980, pp. 204–215. Ye, H.: Integrating Quality of Service Requirements in a Distributed Query Processing Environment, Ph.D thesis, University of Montreal, May 2003. National Laboratory for Applied Network Research, http://www.nlanr.net/
SCINTRA: A Model for Quantifying Inconsistencies in Grid-Organized Sensor Database Systems Lutz Schlesinger1 and Wolfgang Lehner2 1
University of Erlangen-Nuremberg Department of Database Systems Martensstr. 3, 91058 Erlangen, Germany,
[email protected] 2 Dresden University of Technology Database Technology Group 01062 Dresden, Germany,
[email protected] Abstract. Sensor data sets are usually collected in a centralized sensor database system or replicated cached in a distributed system to speed up query evaluation. However, a high data refresh rate disallows the usage of traditional replicated approaches with its strong consistency property. Instead we propose a combination of grid computing technology with sensor database systems. Each node holds cached data of other grid members. Since cached information may become stale fast, the access to outdated data may sometimes be acceptable if the user has knowledge about the degree of inconsistency if unsynchronized data are combined. The contribution of this paper is the presentation and discussion of a model for describing inconsistencies in grid organized sensor database systems.
1
Introduction
Data sets of sensor data are typically collected in sensor database systems ¨ ([BoGS01]). The concept of a federated ([OzVa91]) or data warehouse system ([Inmo96]) is the natural way to combine data of different sensor databases. However, the approaches suffer on the strong consistency property which cannot be maintained due to the high data refresh rate. Instead, we propose a combination of sensor database systems with the concept of grid organized computing structures ([FoKe99]), which is well known in the ‘number crunching’ area (the Globus Project; http://www.globus.org) but is not widely used for ‘data crunching’ where local nodes are represented by database systems. A grid provides an alliance of autonomous computing systems (figure 1). Each node locally stores sensor data. Since sensor data permanently change their values or add new values to already monitored data sets, we introduce the concept of recording periods. After a time interval or a maximum amount of recorded data the recording period is finished, the existing data are freezed and a new period starts. In comparison to the federated approach, we avoid the permanent data synchronization. In the opposite, the packaged updates do not guarantee the strong consistency property: An update of a local snapshot during H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 348–355, 2003. c Springer-Verlag Berlin Heidelberg 2003
SCINTRA: A Model for Quantifying Inconsistencies
349
Fig. 1. Architecture of the discussed grid scenario
query processing is time and cost consuming regarding the communication costs. If a fallback to a local existing but stale snapshot is possible, highest consistency is given up: The snapshots are not the most current ones and the locally cached snapshots need not to have the same valid time. The time difference between the current time and the time point of the selected snapshots should be quantified by a metric. A user may integrate this metric in the query specification to control the maximum inconsistency. Additionally, the metric may be used by the system to distribute queries. Contribution and Structure of the Paper In this paper we quantify the metric and propose a general model for dealing with inconsistencies in our grid organized sensor database system. Our SCINTRA (semi-consistent integrated time replicated approach) data model is presented in section 2. Section 3 discusses the usage of the model for distributing queries to a single node of the grid. An overview of related work is given in section 4. The paper closes with a conclusion.
2
The SCINTRA Data Model
In this section, we introduce the SCINRTA data model as a container model with disjoint fragments consisting of three parts. 2.1
System Model
As illustrated in figure 1 a grid database system consists of a set of nodes Ni , connections and communication costs cij between two neighboring nodes. The total communication costs between two arbitrary grid nodes are the sum of the communication costs of those nodes, which lie on the path between the communicating nodes:
350
2.2
L. Schlesinger and W. Lehner
Data Structure Concept
The data structure concept at a single node of the grid is divided into three layers: Atomic layer: Sensors measure (physical) signals like temperature, sales or bank account and send the data sets to a sensor database, where they are collected and made available for applications to analyze the data sets. Each such a (sensor) object monitors atomic data sets consisting of a timestamp and a value. An atomic object is notated as (x, t, d) or shortly x(t) = d with x as unique sensor identifier, which measures exactly one object of the real world, t as the timestamp and d as measured data. Micro Layer: At the micro layer a set of atomic objects, which are events in the real world, is grouped together. A micro object (MiO) is formed after a number of monitored atomic objects, a time interval or on demand. A MiO consists of a set of atomic objects and gets timestamps representing the time interval of the data set. A function may be defined to describe the value patterns of the real object, because the monitored atomic objects are only discrete, although the type of the data source may be discrete, partial steady or continous ([TCG+93]). At a single node the monitored atomic objects are exactly assigned to a single MiO, why two different MiOs do not overlap. Macro Layer: On the next layer, the macro layer, several MiOs are grouped to macro objects (MaOs). The schema of a MaO is built by schema integration of the schemas of the underlying MiOs. Missing instance values of the MiOs are set to NULL. A MaO forms a container for a set of MiOs. For each logical object (MiO or MaO), which exists exactly once in the grid, several physical representations may exist in the grid. Grid Layer: Now the focus is turned away from a local view to a global view: the grid layer. At a node, denoted as the data source or collecting database Ds (figure Fig. 2. Illustration of objects in the grid 2), the monitored atomic objects are grouped to MaOs which forms the local data DL . A copy of the MaO is sent as a snapshot to other nodes of the grid (consuming database Dt ). The new MaO is stored in addition to already existing MaOs from the same data source. All MaOs from foreign nodes form the foreign data DF . The time of storing the data locally is the transaction time. The valid time is either the upper time bound of the MaO or the transaction time if no other time is known. The object-time-diagram (figure 3) illustrates the situation.
SCINTRA: A Model for Quantifying Inconsistencies
2.3
351
Data Manipulation Concept
The data manipulation concept covers all operations based on the data structures presented in the last section. In this section the discussion is made in reverse order. Operations at the Grid Layer: At the grid layer general operations executed on a single node regard operations on the collecting and consuming database. The standard operations are insertion for adding a new data source including schema integration and the reverse operation deletion implying the removal of all MaOs of the data source existing in DF and the reduction of the integrated schema by the schema of the removed data source. Other operations are distribution for sending MaOs to other nodes and request for requesting the newest MaO from another node. Operations at the Macro Layer: At the macro layer algorithms for a parameterized selection of a set of snapshots (MaOs) and operations between the snapshots are defined. Usually the latest MaOs are selected and joined, which are the closest MaOs to the horizontal line tN OW reflecting the current time in the object-time-diagram (figure 3). In our approach, where an arbitrary number of MaOs of the same data source exists, a selection at any time tc (tc ≤ tN OW ), called the historic cut, is possible. The time distance between the MaOs, tc and tN OW and the data change rate is quantified as the inconsistency metric and defined in the following. Algorithms for selecting the MaOs under consideration of the metric and the age of the MaOs are discussed in [ScLe02].
Fig. 3. Example for the selection of historic cuts and the resulting inconsistency curve
Disregarding the data change rate for a moment, the inconsistency for a single data source is defined as the distance between the valid time of the snapshot time
352
L. Schlesinger and W. Lehner
(Skk ) and tc . For k selected snapshots the inconsistency I is defined on the basis of the Lp -metric: p k I= |(time(S j ) − t )| p
i
c
i=l
The example of figure 3 left shows snapshots of three data sources with p = 1. Figure 3 right illustrates the conflict between age and inconsistency: At point 7 the inconsistency is lower than the inconsistency at point 13, but the age is much higher. The formula is now extended with the data change rate ∆di , which reflects the data changes of the MaO and the underlying MiOs for each data source i. This signature regards existential and value based changes (values: 0–100%). ∆di is a parameter of the reusage rate ρ at a time point t with ts as the time point of the global oldest snapshot: 0 < ∆di ≤ 0, 5 : ρ(t) =
−1 1
−1
|t − (tN OW − ts ) − 1| 2∆di 1 · · (t − (tN OW − ts )) + 1 + 1 tN OW − ts −1 −1 · ·t+1 0, 5 ≤ ∆di < 1 : ρ(t) = −1 −1) tN OW − ts (t + 1) 2(∆di −1
The reusage rate of old snapshots depends on the data change rate (figure 4). At tN OW ρ has a value of 100% independently of ∆di and at ts the value is 0. The combination of the distance in time and ρ leads to an extended inconsistency formula, where αk denotes weights for each data source and Skjk are the selected snapshots: I(time(Sljl ), . . . , time(Skjk )) =
n k=l
Fig. 4. Dependencies of reusage degree and age
(αk · |time(Skjk ) − tc | · (1 − ρk (time(Skjk ))))
After selecting a single snapshot for each data source standard operations like selection or join may be defined in a nature manner. Operations at the Micro Layer: They are mostly realized by operations on atomic objects. The operations are standard operations focusing on sensor data sets under the consideration of the classification introduced by [TCG+93]. The operations selection, projection, aggregation and the unary operation are defined in a nature manner by applying the operation
SCINTRA: A Model for Quantifying Inconsistencies
353
to each atomic object of the MiO. A usual definition of the join operation would result in an atomic object with 6 elements. Since an atomic object consists of three elements, the join operation is defined in combination with a binary operation to get the 3-tuple. This special SCINTRA-join of two MiOs implies the problem of selecting the atomic objects, because the time points and the type of the sensor may be different. As illustrated in figure 5 the bullets mark atomic objects of a continous and a discrete sensor. The selection of two atomic objects depends on the join semantic according to the SEQ-model ([SeLR95]), while Fig. 5. Example for join semantics the resulting continuity type depends on the application. The time interval of the resulting MiO is determined by the atomic objects of the new MiO. Finally, the set operation on two MiOs is only defined, if the MiOs contain data from the same sensor. Operations at the Atomic Layer: At the atomic layer the operations introduced above are now realized in a nature manner by using atomic objects of one or two MiOs. Regarding the join it is mentioned that the time point of the resulting atomic object is a parameter of the operation and defined by the join semantic. The result of an aggregation operation is an atomic object whereby the aggregation operation is either applied to the time (operators min, max, . . . ) or on the data (operators sum, avg, . . . ). While these operations result in a single atomic object the set operations result in a set, which forms a logical micro object. The set operations are also defined in a nature manner.
3
Query Distribution in the Grid
Figure 6 shows a simple example of a grid. A user sends a query to join snapshots of two data sources D1 and D2 . If the query is routed to N1 , then the query can be evaluated at tc11 resulting in a computed inconsistency of 5. Another way is to transfer the snapshot S22 from N2 to N1 Fig. 6. Example for operations on the grid with transfer costs c21 = 50 resulting in an inconsistency of 1 at tc12 . If the query is routed to N2 , then a query evaluation is
354
L. Schlesinger and W. Lehner
possible, if S13 is copied from N1 to N2 with c12 = 20 and a resulting inconsistency of 1. Table 1 summarizes the facts. If the user specifies an upper bound of the allowed age of the historic cut, the transfer costs and the inconsistency, the system is able to decide on which node the query has to be evaluated. With this decision the snapshots are selected and operations between the elements of the snapshots are executed corresponding to a specification in the query. At first, the MiOs are selected, which are determined by a sensor identifier and a time interval and which are parts of the query declaration. In the same way the operations on the atomic objects are specified. For example, for a join operation the join semantic, the binary operator and the resulting continuity type may be specified. Thus the introduced operators and its parameters allow the user to control the query execution to receive the expected result. Table 1. Overview of computed data in the example historic cut transfer costs inconsistency t11 0 5 t12 20 1 t21 50 1
4
Related Work
Since our approach is influenced by many different research areas, we outline only two areas. At first, database middleware systems like Garlic ([IBM01]) or mediator systems ([Wied92]) provide a query engine with sophisticated wrapper and query optimization techniques ([RoSc97]). Secondly, introducing replicas in ¨ distributed database systems ([OzVa91]) is a way to speed up access to data. Since replicas are synchronized in the context of an update operation, our approach is much more related to the concept of database snapshots ([AdLi80]). The selection of the “best” set of snapshots from a consistency point of view however, has not been discussed prior to our work.
5
Conclusion
The concept of sensor grid database systems allows a grid member to cache data from other grid members. If from a user point of view a fallback to outdated data is acceptable during query evaluation, the query may be faster evaluated but the user gives up highest consistency. In this paper we introduce a model for a sensor grid database system and a consistency framework to quantify the inconsistencies. Our approach is a solid base to decide on which node a query should be evaluated. The model is further helpful for the user to specify how operations on the selected snapshots should be executed.
SCINTRA: A Model for Quantifying Inconsistencies
355
References Adiba, M.E.; Lindsay, B.G.: Database Snapshots. In: Proceedings of the 6th International Conference on Very Large Data Bases (VLDB’80, Montreal, Canada, October 1–3), 1980, pp. 86–91 [BoGS01] Bonnet, P.; Gehrke, J,; Seshadri, P: Towards Sensor Database Systems. In: Proceedings of the 2nd International Conference on Mobile Data Management (MDM’01, Hong Kong, China, January 8–10), 2001, pp. 3–14 [FoKe99] Foster, T.; Kesselman, C. (Hrsg.): The Grid: Blueprint for a New Computing Infrastructure. Morgan Kaufmann Verlag, San Francisco (CA) et al., 1999 [IBM01] N.N.: The Garlic Project, IBM Corp., 2001 [Inmo96] Inmon, W.H.: Building the Data Warehouse, 2nd edition. New York, Chichester, Brisbane, Toronto, Singapur: John Wiley & Sons, Inc., 1996 ¨ ¨ [OzVa91] Ozsu, M.; Valduriez, P.: Principles of Distributed Database Systems. Prentice-Hall, Englewoods Cliffs (New Jersey, USA), 1991 [RoSc97] Roth, M.T.; Schwarz, P.M.: Don’t Scrap It, Wrap It! A Wrapper Architecture for Legacy Data Sources. In: Proceedings of 23rd International Conference on Very Large Data Bases (VLDB’97, Athens, Greece, August 25–29), 1997, pp. 266–275 [ScLe02] Schlesinger, L.; Lehner, W.: Extending Data Warehouses by SemiConsistent Database Views. In: Proceedings of the 4th International Workshop on Design and Management of Data Warehouses (DMDW’02, Toronto, Canada, May 27), 2002 [SeLR95] Seshadri, P.; Livny, M.; Ramakrishnan, R.: SEQ: A Model for Sequence Databases. In: Proceedings of the 11th International Conference on Data Engineering (ICDE’95, Taipei, Taiwan, March 6–10), 1995, pp. 232–239 [TCG+93] Tansel, A.; Clifford, J.; Gadia, S.; Jajodia, S.; Segev, A.; Snodgrass, R.: Temporal Databases. Benjamin/Cummings Publishing, Redwood City, 1993 [Wied92] Wiederhold, G.: Mediators in the architecture of future information systems. In: IEEE Computer 25(3)1992, pp. 38–49 [AdLi80]
Topic 6 Grid Computing and Middleware Systems Henri Bal, Domenico LaForenza, Thierry Priol, and Peter Kacsuk Topic Chairs
Grid computing is originated from supercomputing in the mid nineties. The grand challenge problems were not solvable even the largest supercomputers. A natural idea was to run these high-end applications on several supercomputer resources in parallel. This concept led to the new direction of supercomputing, called metacomputing. The success of the first experiments in metacomputing initiated a more general view of metacomputing whereby not only supercomputers but any kind of computational, storage and other types of resources should be connected and exploited on demand. Even the objectives of metacomputing were widened and goals like supporting high-throughput computing, collaborative computing, tele-immersion, etc. became impor-tant aspects of this kind of generic view of metacomputing that was called later Grid computing. Grid computing, streamlined by the successful Global Grid Forum (GGF), has become a major new research area over the past few years, with strong involvement from both academia and the computing industry. Although much progress has been made in the deployment of grid infrastructures, many challenges still lie ahead of us before the ultimate goal of the grid can be realized. Recognizing the importance and potential impact of Grid computing on the whole society we decided to organize a workshop in the framework of Euro-Par where researchers could report their recent advances in Grid middleware design. The workshop received 28 submitted papers dealing with all possible aspects of Grid middleware research. Finally, after a careful review procedure we selected 6 regular and 4 short papers. The regular papers cover the following main issues: An RPC system called GrAD-Solve that supports execution of parallel applications over Grid resources; A Grid enabled computational toolkit that provides transparent and stable access to Grid compute resources from Matlab; An opportunistic job migration scheme to decide if job migration is feasible and worthwhile when a new Grid resource appears; Introducing semantic access control to improve security in medical applications over the Grid; An automated negotiation engine that identifies mutually acceptable terms and could be used in the Grid Notification Service; A Grid monitoring system for grid job and re-source monitoring. The short papers deal with the following problems: The design and implementation of a database toolkit for engineers, which has been incorporated into the Matlab environment; a resource accounting and charging system for a Condorbased Grid environment; a plug-in for the gSOAP Toolkit that allows development of Web Services exploiting the Globus Security Infrastructure; Optimisations of Java RMI for Grid applications. H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, p. 356, 2003. c Springer-Verlag Berlin Heidelberg 2003
Implementation of a Grid Computation Toolkit for Design Optimisation with Matlab and Condor Gang Xue, Matthew J. Fairman, Graeme E. Pound, and Simon J. Cox Southampton Regional e-Science Centre, School of Engineering Sciences, University of Southampton, Highfield, Southampton, SO17 1BJ, UK {gx, mjf, gep, sjc}@soton.ac.uk
Abstract. The process of design search and optimisation is characterised by its computationally intensive operations, which produce a problem well suited to Grid computing. Here we present a Grid enabled computation toolkit that provides transparent and stable access to Grid compute resources from Matlab, which offers comprehensive support for the design optimisation processes. In particular, the access and integration of the Condor resource management system has been achieved by using the toolkit components that are enabled by Web service and service enhancement technologies. The use of the computation toolkit for a four-dimensional CFD parameter study with Matlab and Condor is considered as an exemplar problem.
1
Introduction
Engineering design search and optimisation [1] is the process whereby engineering problems are modelled and analysed to yield improved designs. This process involves identifying design parameters that the engineer wishes to optimise, computing a measure of the quality of a particular design (the objective function) using an appropriate model, and using a number of design search algorithms to generate additional information about the behaviour of a model over the parameter space, as well as to optimise the objective function to improve the design quality. It is potentially computationally intensive, as lengthy and repetitive calculations of the objective function with regard to the design variables may be required. The demand for compute resources by the design optimisation process can be well satisfied by the adoption of Grid computing technologies. Grid computing provides the infrastructure and technologies to enable large-scale resource sharing and the construction of Virtual Organisations (VOs) that address various science and engineering problems [2]. When applied in design optimisation, it allows the process to discover and access resources from heterogeneous environments in a transparent and consistent manner for the required compute tasks. An important aspect of the use of Grid technology for this purpose is to establish links between the design optimisation process and various compute resource management systems, which organise and manage most of the compute resources on the Grid. Notable examples of such systems include Globus [3], Condor [4], Maui [5] and UNICORE [6]. In our previous work [7], access to the Globus system has been provided to Matlab users by H. Kosch, L. Böszörményi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 357–365, 2003. © Springer-Verlag Berlin Heidelberg 2003
358
G. Xue et al.
building on the Java CoG toolkit [8]. Here we focus on the integration into Matlab of another important target, the Condor system, using an approach more strongly grounded in Web service technologies. Condor is a resource management system that creates a High-Throughput Computing (HTC) environment by harnessing the power of UNIX and NT clusters and workstations [9]. Whilst it can manage dedicated clusters, it can also exploit preexisting resources under distributed ownership, such as computers sitting on people’s desks. For the design optimisation process, Condor provides a robust computational environment, which is able to find resources with specific requirements, carry out the compute operations, and manage the computation jobs for the process. The HTC style of Condor also suits the need of design optimisation, which can yield better results with further calculations. The integration of Grid resource management systems with the design optimisation process is mainly based within the Matlab environment [10], which is selected as the interface to the Geodise PSE [7]. The Matlab package provides a fourth generation language for numerical computation, built-in math and graphics functions and numerous specialised toolboxes for advanced mathematics, signal processing and control design. It is widely used in academia and industry for algorithm prototyping, and for data visualisation and analysis. From version 6.5 Matlab also contains a number of Just-In-Time (JIT) acceleration technologies to improve the performance of native Matlab code. In order to facilitate access to compute resources on the Grid, especially the Condor system, for users in Matlab and potentially other problem solving environments, a computation toolkit has been developed in the form of a set of client utilities and replaceable middleware components. The toolkit adopts a layered, highly flexible and easily configurable structure, which allows using a consistent user interface to access different compute resource systems with the help of corresponding middleware components. In particular, a middleware component based on Web service technology has been constructed to expose Condor as a Grid service, and hence make it accessible to the toolkit users. In the rest of the paper, we first focus on the design and implementation of the computation toolkit, including the Web service interface to the Condor system. We then demonstrate the application of the toolkit in the Matlab environment for engineering design optimisation problems.
2
The Computation Toolkit
The computation toolkit provides users with a set of basic functions to perform Grid based computation tasks, which are described below together with the technologies behind and the design of the toolkit. 2.1 Functions for Grid Based Computation In our computation toolkit, a number of functions are implemented both in the form of a low-level Java [11] class library, and a set of high-level commands for the Matlab environment, which enable users to query the resources, submit and manage the
Implementation of a Grid Computation Toolkit for Design Optimisation
359
compute jobs, perform data transmission, and configure security settings. The Java class library provides mappings between the APIs and the compute functions. It is therefore possible to perform a Grid based compute operation programmatically. The commands for Matlab users listed in Table 1 are written in the interpretive Matlab language, so that the compute functions are presented in a manner consistent with the behaviour and syntax of the Matlab environment. The commands can be used to start compute operations based on interactive user inputs, or directly on script files. The grid_createcredential command needs to be called before any other commands in the toolkit can be used. It sets up the user credentials according to the security requirement policy of the target compute resource. The user credential can be as simple as a username/password pair, or as complicated as a PKI (Public Key Infrastructure) [12] based user certificate. When all compute operations are finished, the grid_destroycredential command is used to disable the user credential, so that it won’t be misused or stolen. The grid_jobreqest, grid_sendfiles and grid_startjob commands represent the major steps in starting a compute job submission process. Users start the compute tasks with requests to the target resources for job submission. Once accepted, the job files are delivered to the resources using the data transfer command. Users then send out instructions to start the computation. In addition, a few job management commands are also provided to enable users to monitor and manage the running of jobs. Once the jobs are finished, job results can be easily retrieved using the grid_retrievefiles command. Table 1. Compute Commands Function Name grid_createcredential
Description Loads the user credential to the toolkit for compute operations
grid_destroycredential Destroys the user credential stored by the toolkit grid_jobrequest
Requests the submisson of a compute job with specification for the compute resources and description of the job. If accepted, a job handle is returned.
grid_startjob
Starts the submitted job identified by a job handle.
grid_killjob
Terminates the job identified by the job handle.
grid_getstatus
Queries the status of submitted jobs identified by the job handles.
grid_listjobs
Returns job handles for all jobs submitted to the target resource that belong to the user.
grid_sendfiles
Uploads the job files required for job execution to the target resource.
grid_retrievefiles
Retrieves all result data/files generated by the finished job identified by a job handler.
grid_jobsubmit
Automates the process of job request, job files transfer, and job start.
2.2 Design of the Computation Toolkit One of the most important features of Grid technologies is to provide transparency in accessing computational resources spread across the Grid. It is important because of the heterogeneity of the Grid environment. For instance, compute resource systems that can be targeted by the design optimisation processes include Globus, Condor, and
360
G. Xue et al.
UNICORE. Each of them has a different user interface, incompatible security policy, and diverse workflow model. Traditionally, separate client packages would be needed to access these different resources, which is clearly unscalable and not ideal for system integration. Moreover, these tightly bound client packages are less adaptable to potential changes made by the target systems, and would force frequent modification to design optimisation applications. To provide the essential transparency, the computation toolkit has been implemented in a distinctive structure, which separates the user interface from the underlying message processor that is responsible for interactions with different remote resources based on various protocols. The user interface is an API that represents the basic semantics of compute operations and is designed to be simple but stable. In contrast, the message processor is much more dynamic and powerful. It is implemented as two chains of filters for input and output message processing. Each filter is responsible for processing a specific message part, or even the entire message based on the protocol it is responsible for. At the end of the chains, there is a communication handler, which is responsible for direct interactions with the compute resource or compute service middleware. By using Java reflection, the filter chains are dynamically constructed at the runtime based on the information loaded from a configuration file, which is detached from the client applications and can be easily modified. It is therefore possible to access different resource systems or adapt to changes by loading different set of filters. Figure 1 shows the implementation of the file upload functionality when accessing the Condor service. The interaction is primarily based on the SOAP protocol [13], and needs to conform to the security regulation set by the service. For data transmission, the service uses DIME [14]. HTTP is used as the underlying communication protocol. Accordingly, a SOAP output filter, a Security handler and a DIME builder are loaded in turn to the output chain. Since the response is expected in plain SOAP format, only a SOAP input filter is loaded to the input chain. And at the end of the chains, an HTTP handler is loaded to handle the actual message exchanges.
&RQILJXUDWLRQ )LOH
DIME
WS-Security
SOAP Output
5HTXHVW
+773
&RQGRU :HE 6HUYLFH
Input Chain
5HVSRQVH
SOAP Input
Interface API in Java
"grid_sendfiles" command
Output Chain
Fig. 1. Implementation of Data Upload with the Message Filter Chains
The flexibility brought by the structure of the computation toolkit allows different compute resource systems accessible in a client-server mode to be integrated into the toolkit. Apart from the middleware component for the Condor system provided in the
Implementation of a Grid Computation Toolkit for Design Optimisation
361
current implementation, we will also construct message filters based on our previous work on Globus client tools, so that the Globus system can also be accessed using the toolkit.
3
The Web Service Enabled Interface to Condor
The computation toolkit consists of a set of Web service based middleware components that expose the Condor system. Several enhancements have been applied in addition to the standard Web service technologies to provide features required for Grid based operations. We have also exploited features of Condor and Web service inspection technology to support compute tasks with specific requirements on the resources. 3.1 The Web Service Interface The Web service enabled interface to the Condor system has been constructed in order to achieve programmatic, platform and language neutral access to the resources managed by Condor. It provides mappings between the computation toolkit functions and the resource management mechanisms of Condor. The interface is primarily an interpreter between XML messages representing the compute operations and the ClassAd language of Condor. It also hides from the users details of resource management that are proprietary to Condor, so as to achieve the desired transparency. In addition to interactions with Condor, the service also implements general Grid service functions that are not provided by the Condor system, such as management of security and process status. Furthermore, the service has also adopted several enhancements from future Web service technologies, which are described in detail in the following section. 3.2 Enhancements to the Service for Compute Operations The standard Web services technology collection, which includes XML, SOAP, WSDL and UDDI, has not supplied a solution to the management of service security. To address this problem and make the service compatible with common Grid security management practise such as GSI, we have employed the WS-Security specification in the Condor service implementation. WS-Security [15] is a maturing technology for Web service security management. It extends the simple structure of SOAP to establish a standard security mechanism on the message level, which is independent of the underlying transportation methods. Since WS-Security focuses mainly on the infrastructure, rather than detailed security techniques such as authentication and encryption, it allows established security solutions, including Kerberos [16], Public Key Infrastructure (PKI), and GSI [17] to be integrated so that they can be applied to Web services in a standard and consistent manner. In the Condor service, we have deployed PKI based asymmetric encryption in order to keep message confidentiality and perform user authentication. The exchanges of credential information, i.e. the X.509 certificates that contain public keys, are carried out following the WS-Security definition. In addition, we will also apply
362
G. Xue et al.
XML digital signature [18] to the service messages, as proposed by WS-Security, so as to achieve message integrity. Another problem about interfacing Condor with Web service is the degraded performance in data transfer. Traditionally, data transfer with SOAP is based on the Base64 encoding, which imposes heavy costs due to the serialisation/deserialisation of the Base64 strings. We address this problem in the Condor service by exploiting a recently proposed data transfer format named Direct Internet Message Encapsulation (DIME). DIME defines a MIME-like message format in which variously typed data that does not fit expediently or efficiently into XML can be directly contained and transmitted along with the standard SOAP messages. Significant improvement to the performance on data transfer can be achieved by using DIME, as overheads for data conversion are avoided. Test results for this have been demonstrated in [19]. The enhancements to the Condor service have been implemented with the help of Microsoft’s recently released WSE1.0 [20], which provides sound support for several emerging Web service technologies based on the .NET framework. Corresponding to the service enhancements, message filters for the client tools have also been constructed using related Java technologies [21] [22]. 3.3 Discoveries and Access of Resources with Special Capabilities We define resources to encompass compute facilities with certain features, e.g. memory, processor, disk capacities, database/archive facilities, and also specialist software environment or licensed applications. Design optimisation requires access to all of these resource types at various stages of the process. It is therefore necessary for the users to be able to discover all resources on demand, and manage access to these resources in a consistent way to familiar requests for particular compute capabilities. Our service interface to Condor provides the solution to this problem by combining features of Condor and inspection technology for Web services. The Condor system provides a convenient method for the discovery of resources with special capabilities in the Condor pool. When a computer joins the Condor pool, it declares its capabilities by adding corresponding attributes to its Condor system configuration file. These attributes will then be reflected in the information generated by the resource status query performed by Condor, which is passed on to the Web service. The service will therefore be able to make the discovery by identifying the target attributes. Figure 2(a) shows the Condor status query results for a machine that supports Java and MPI. Additional regulations that need to be enforced for computing on special resources can be presented in XML Schema and WSDL file that extends the original ones of the Condor service. Once the service makes the discovery, it follows the convention defined by the WS-Inspection specification [23] to locate the WS-I documents on the resources, which point to the extended schema and WSDL files. The service will then use the extended service definition to process the incoming requests. The entire process is illustrated in Figure 2(b). This model of resource discovery and inspection can be extended to suit the situation where resources are presented to the Web service as Web service themselves. In that case the sub-services are registered directly with the interface service, which becomes a service broker in addition to its original functions.
Implementation of a Grid Computation Toolkit for Design Optimisation 6HUYLFH &OLHQW
1 Condor Service
6HDUFK
&RQGRU 6\VWHP
363
/R FD WH 2
Request Processor 6HUYHU
5HTXHVW
3 Inspect
:6,QVSHFWLRQ 'RFXPHQW
5HTXHVW
4 Load
(a)
6HUYLFH 'HVFULSWLRQ
(b)
Fig. 2. (a) Discovering Resources of Special Capabilities in Condor. (b) Process of Special Capability Discovery through the Service
4
The Computation Toolkit Application Exemplar
To demonstrate the possible use of our Grid-enabled computation toolkit in Matlab, we choose a simple problem of fluid dynamics, which is a parameter study of the fluid dynamics of a 2D parameterised geometry of the nacelle for an aircraft engine. A suitable objective function was calculated across four design variables which describe curves on the upper and lower surface of the nacelle. In this example, each simulation is set up as a compute task, which is submitted to a Condor managed resource from Matlab using the toolkit command, as shown in Figure 3(a). Figure 3(b) shows the results of 648 simulations that were performed. The jobs were distributed by the Condor system over a cluster of 12 NT nodes. A parameter study such as this allows the engineer to easily evaluate the impact of different design variables upon the quality of a design.
Fig. 3. (a) Starting Grid Computation Tasks Using the Computation Toolkit. (b) Visualised Result of a Four Dimensional CFD Parameter Studies
364
G. Xue et al.
Once an engineer has developed a Matlab script based on our toolkit for design studies like this, improvements to the robustness and reliability of the underlying Condor service, or additional Condor-enabled resources and capabilities can be exploited in a seamless and transparent way without further modification to the script.
5
Conclusion and Future Work
The significant demand for computation resources in the design optimisation process can be satisfied in a transparent way using Grid computing. The computation toolkit presented in this paper facilitates access to computation resources, in particular the Condor system, from an engineering PSE such as Matlab. The implementation of the toolkit is based on a highly flexible structure and a few future Web service technologies. As an exemplar, we have demonstrated the use of the toolkit for a simple CFD design optimisation problem. Future work on the toolkit will mainly focus on the implementation of filter and middleware components for the access of different computation resources.
Acknowledgements. This work is supported by the Geodise e-Science pilot project (UK EPSRC GR/ R67705/01). We thank the authors of the Condor system at the University of Wisconsin, Madison.
References [1] [2]
[3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18]
The Geodise Project. http://www.geodose.org I. Foster, C. Kessleman, and S. Tuecke. The Anatomy of the Grid: Enabling Scalable Virtual Organisations, International Journal of Supercomputer Applications, 15(3):200222, 2001 The Globus Project. http://www.globus.org The Condor Project. http://www.cs.wisc.edu/condor/ The Maui Scheduler. http://supercluster.org/maui/ UNiform Interface to COmputing Resources. http://www.unicore.de/ G. E. Pound, M. H. Eres, et al. A Grid-Enabled Problem Solving Environment (PSE) for Design Optimisation within Matlab. Proceeding of IPDPS 2003, Nice, France. Commodity Grid Kits. http://www.globus.org/cog/ Miron Livny, and the Condor Team. Condor User Manual. http://www.cs.wisc.edu/condor Matlab 6.5. http://www.mathworks.com Java 2. Sun Microsystems Inc., http://java.sun.com Public Key Infrastructure. http://www.ietf.org/html.charters/pkix-charter.html Simple Object Access Protocol. http://www.w3.org/2000/xp/Group/ DIME. http://www.ietf.org/internet-drafts/draft-nielsen-dime-02.txt The WS-Security Specification. http://www.ibm.com/developerworks/library/ws-secure/ Kerberos: The Network Authentication Protocol. http://web.mit.edu/kerberos/www/ Grid Security Infrastructure. http://www.globus.org/security/ The IETF/W3C XML Signature Work Group. http://www.w3.org/Signature/
Implementation of a Grid Computation Toolkit for Design Optimisation
365
[19] G. Xue, G. E. Pound, S. J. Cox. Performing Grid Computation with Enhanced Web Service and Service Invocation Technologies. Proceedings of ICCS 2003, Melbourne, Australia. [20] Web Services Enhancements 1.0 for Microsoft .NET. http://msdn.microsoft.com/ [21] Java DIME Library v1.0.2. http://onionnetworks.com/dime/javadoc/ [22] Java Cryptography Extension (JCE). http://java.sun.com/products/jce/ [23] Web Service Inspection Language 1.0. http://www.ibm.com/developerworks/Webservices /library/ws-wsilspec.html
Grid Resource Selection for Opportunistic Job Migration Rub´en S. Montero1 , Eduardo Huedo2 , and Ignacio M. Llorente1,2 1
2
Departamento de Arquitectura de Computadores y Autom´ atica, Universidad Complutense, 28040 Madrid, Spain. Centro de Astrobiolog´ıa (Associated to NASA Astrobiology Institute), CSIC-INTA, 28850 Torrej´ on de Ardoz, Spain.
Abstract. The ability to migrate running applications among different grid resources is generally accepted as the solution to adapt to dynamic resource load, availability and cost. In this paper we focus on opportunistic migration when a new resource becomes available in the Grid. In this situation the performance of the new host, the remaining execution time of the application, and also the proximity of the new resource to the needed data, become critical factors to decide if job migration is feasible and worthwhile. We discuss the extension of the GridWay framework to consider all the previous factors in the resource selection and migration stages in order to improve response times of individual applications. The benefits of the new resource selector will be demonstrated for the execution of a computational fluid dynamics (CFD) code.
1
Introduction
Computational Grids are inherently dynamic environments, being characterized by unpredictable changing conditions, namely: high fault rate, dynamic resource availability, dynamic resource load, and dynamic resource cost. Consequently, in order to obtain a reasonable degree of both application performance and fault tolerance, a job must be able to migrate among the Grid resources adapting itself according to their characteristics, availability, load, and cost. Probably, the most sensitive step to the above conditions in job scheduling is resource selection, which in turn relies completely in the dynamic information gathered from the Grid. Resource selection usually takes into account the performance offered by the available resources, but it should also consider the proximity between them [1,2]. The size of the files involved in some application domains, like Particle Physics or Bioinformatics, is very large. Hence, the quality of the interconnection between resources, in terms of bandwidth and latency, is a key factor to be considered in resource selection [3]. This fact is specially relevant in the case of adaptive job execution, since job migration requires the transfer of large restart files between the compute hosts.
This research was supported by Ministerio de Ciencia y Tecnolog´ıa through the research grant TIC 2002-00334 and Instituto Nacional de T´ecnica Aeroespacial (INTA).
H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 366–373, 2003. c Springer-Verlag Berlin Heidelberg 2003
Grid Resource Selection for Opportunistic Job Migration
367
In this paper, we focus on the opportunistic migration of jobs when a “better” resource is discovered, because either a new resource is added to the Grid, or because the completion of an application frees a Grid resource. Opportunistic migration has been widely studied in the literature [4,5,6,7], previous works have clearly demonstrated the relevance of considering the amount of the computational work already performed by the application, the need of a metric to measure the performance gain due to migration, and the critical factor of dynamic load information of Grid resources. However, previous migration frameworks do not consider the proximity of the computational resources to the needed data, and therefore the potential performance gain can be substantially decremented by the overhead induced by job migration. The migration and brokering strategies presented in this work have been implemented on top of the GridWay framework [8], whose architecture and main functionalities are briefly described in Section 2. In Section 3 we discuss the extension of the GridWay framework to also consider resource proximity in the resource selection stage. This selection process is then incorporated to the GridWay migration system in Section 4. The benefits of the new resource selector will be demonstrated in Section 5 for the adaptive execution of a CFD code on a research testbed. Finally, Section 6 includes some conclusions and outlines our future work.
2
The GridWay Framework
GridWay is a new Globus-based experimental framework that allows an easier and more efficient execution of jobs on a dynamic Grid environment in a “submit and forget” fashion. The core of the GridWay framework is a personal submission agent that automatically performs the steps involved in job submission: system selection, system preparation, submission, monitoring, migration and termination. The user interacts with the framework through a request manager, which handles client requests (submit, kill, stop, resume...) and forwards them to the dispatch manager. The dispatch manager periodically wakes up at each scheduling interval, and tries to submit pending and rescheduled jobs to Grid resources (obtained through the resource selector module). Once a job is allocated to a resource, a submission manager and a performance monitor are started to watch over its correct and efficient execution (see [8] for a detailed description of the architecture of the GridWay framework).
3
The Resource Selector
Due to the heterogeneous and dynamic nature of the Grid, the end-user must establish the requirements that must be met by the target resources (discovery process) and criteria to rank the matched resources (selection process). The attributes needed for resource discovery and selection must be collected from the information services in the Grid testbed, typically the Globus Monitoring and Discovery Service (MDS). Usually, resource discovery is only based on static
368
R.S. Montero, E. Huedo, and I.M. Llorente
attributes (O.S., architecture...) collected from the Grid Information Index Service (GIIS), while resource selection is based on dynamic attributes (disk space, processor load...) obtained from the Grid Resource Information Service (GRIS). The dynamic network bandwidth and latency between resources will be also considered in the resource brokering scheme. Different strategies to obtain these network performance attributes can be adopted depending on the services available in the testbed. For example, MDS could be configured to provide such information by accessing the Network Weather Service (NWS) [9] or by activating the reporting of GridFTP statistics [10]. Alternatively, the end-user could provide its own network probe scripts or static tables. The brokering process of the GridWay framework is shown in figure 1. Initially, available compute resources are discovered by accessing the GIIS server and, those resources that do not meet the user-provided requirements are filtered out. At this step, an authorization test (via GRAM ping request) is also performed on each discovered host to guarantee user access to the remote resource. Then, the dynamic attributes of each host are gathered from its local GRIS server. This information is used by an user-provided rank expression to assign a rank to each candidate resource. Finally, the resultant prioritized list of candidate resources is used to dispatch the job.
JOB DISPATCH
Requirements & Preferences
Ranked resource list GRID INFORMATION SERVICE
RESOURCE DISCOVERY
GIIS Filtered LDAP search
Preferences & Candidate resource list
Globus MDS
LDAP registrations
GRIS
GRIS
Ranked resource list
RESOURCE SELECTION
Multiple LDAP searchs
GRIS
Fig. 1. The brokering process scheme of the GridWay framework
The new selection process presented in this paper considers both dynamic performance and proximity to data of the computational resources. In particular, the following circumstances will be considered in the resource selection stage: – The estimated computational time on the candidate host being evaluated when the job is submitted from the client or migrated from the current execution host. – The proximity between the candidate host being evaluated and the client will be considered to reduce the cost of job submission and monitoring, and file staging.
Grid Resource Selection for Opportunistic Job Migration
369
– The proximity between the candidate host being evaluated and a remote file server will be also considered to reduce the transfer costs when some input or output files, specified as a GridFTP URL, are stored in such server. – The proximity between the candidate host being evaluated and the current execution host will be also considered to reduce the migration overhead. 3.1
Performance Model
In order to reflect all the circumstances described previously, each candidate host (hn ) will be ranked using the total execution time (lowest is best) when the job is submitted or migrated to that host at a given time (tn ). In this case, we can assume that the total execution time can be split into: Texe (hn , tn ) = Tcpu (hn , tn ) + Txf er (hn , tn ),
(1)
where Tcpu (hn , tn ) is the estimated computational time and Txf er (hn , tn ) is the estimated file transfer time. Let us first consider a single-host execution, the computational time of the application on host h at time t can be estimated by: Op if CP U (t) ≥ 1; s (h, t) = F LOP SOp (2) Tcpu F LOP S·CP U (t) if CP U (t) < 1. where F LOP S is the peak performance achievable by the host CPU, Op is the number of floating point operations of the application, and CP U (t) is the total free CPU at time t, as provided by the MDS default scheme. However, the above expression is not accurate when the job has been executing on multiple hosts and then is migrated to a new one. In this situation the amount of computational work that have already been performed must be considered [5]. Let us suppose an application that has been executing on hosts h0 . . . hn−1 at times t0 . . . tn−1 and then migrates to host hn at time tn , the overall computational time can be estimated by: n−1 n−1 i t cpu s Tcpu (hn , tn ) = ticpu + 1 − (hn , tn ), (3) Tcpu s (h , t ) T i i cpu i=0 i=0 s (h, t) is calculated using (2), and ticpu is the time the job has been where Tcpu executing on host hi , as measured by the framework. Note that, expressions 2 and 3 become equivalent when n = 0. Similarly, the following expression estimates the total file transfer time:
Txf er (hn , tn ) =
n−1 i=0
tixf er +
Datah ,j n j = client, f ile server, exec host, bw(h , j, tn ) n j
(4) where bw(h1 , h2 , t) is the bandwidth between hosts h1 and h2 at time t, Datah1 ,h2 is the file size to be transferred between them, and tixf er is the file transfer time on host hi , as measured by the framework.
370
4
R.S. Montero, E. Huedo, and I.M. Llorente
GridWay Support for Adaptive Job Execution
The GridWay framework supports job adaption to changing conditions by means of automatic job migration. Once the job is initially allocated, it is dynamically rescheduled when one of the following events occurs: – – – – –
a new “better” resource is discovered (opportunistic migration), the remote host or its network connection fails, the submitted job is canceled or suspended, a performance degradation is detected, the requirements or preferences of the application changes (self-migration).
In this work we will concentrate on opportunistic migration. The dispatch manager wakes up at each discovery interval, and it tries to find a better host for each job by activating the resource selection process described in Section 3. In order to evaluate the benefits of job migration from the current execution host (hn−1 ) to each candidate host (hn ), we define the migration gain (Gm ) as: Gm =
Texe (hn−1 , tn−1 ) − Texe (hn , tn ) , Texe (hn−1 , tn−1 )
(5)
where Texe (hn−1 , tn−1 ) is the estimated execution time on current host when the job was submitted to that host, and Texe (hn , tn ) is the estimated execution time when the application is migrated to the new candidate host. The migration is granted only if the migration gain is greater than an user-defined threshold, otherwise it is rejected. Note that although the migration threshold is fixed for a given job, the migration gain is dynamically computed to take into account the dynamic data transfer overhead, the dynamic host performance, and the application progress. In the experiments presented in Section 5 the migration gain has been fixed to 10%.
5
Experiments
The behavior of the resource selection strategy previously described is demonstrated in the execution of a CFD code, that solves the 3D incompressible Navier-Stokes equations using an iterative multigrid method. In the following experiments, the client host is ursa, which holds an input file with the simulation parameters, and the f ile server is cepheus, which holds the executable and the computational mesh. The output file with the velocity and pressure fields is transferred back to the client, ursa, to perform post-processing. Table 1 shows the available machines in the testbed, their corresponding CPU performance (MFLOPS), and the maximum bandwidth (MB/s) between them and the hosts involved in the experiment. We will impose two requirements on the compute resources: a minimum main memory of 128MB, enough to accommodate the CFD simulation; and a total free CPU higher than 90% to prevent oscillating migrations. Initially, the application is submitted to draco, since it is the only resource that meet the previous
Grid Resource Selection for Opportunistic Job Migration
371
Table 1. Available machines in the testbed, their CPU performance, and bandwidth between them and the machines involved in the experiment (client=ursa, f ile server=cepheus and exec host=draco). Bandwidth host
Model
ursa draco columba cepheus solea
Sun Blade 100 Sun Ultra 1 Pentium MMX Pentium Pro Sun Enterprise 250
CP U
OS
330 175 225 325 350
Solaris 8 Solaris 8 Linux 2.4 Linux 2.4 Solaris 8
Memory client f ile server exec host ∞ 0.4 0.4 0.4 0.2
256MB 128MB 160MB 64MB 256MB
0.4 0.4 0.4 ∞ 0.2
0.4 ∞ 0.4 0.4 0.2
requirements. We will evaluate rescheduling strategies based on expressions 3 and 1, when the artificial workload running on columba and solea completes at different execution points (iterations) of the application running on draco. Let us first suppose that the application is rescheduled using expression 3 (figure 2, left-hand chart). In this case, as the file transfer time is not considered, migration to both hosts always presents a performance gain. The dispatch manager will consider feasible the migration to the best ranked host, solea, until the eight iteration (Texe = 302) is reached. Figure 2 (right-hand chart) shows the dynamic ranks of solea and columba when the application is rescheduled using expression 1. In this situation, migration to solea only will be granted until the second iteration (Texe = 325) is reached. Note that from fifth iteration, the performance gain offered by solea and columba is not high enough to compensate the file transfer overhead induced by job migration. Moreover, from sixth iteration the best ranked host is columba (nearest host) although it presents a CPU performance lower than solea, as proximity to data becomes more important as the application progresses.
360
500
340 450
300
Time (seconds)
Time (seconds)
320 280 260 240 220 Rank Columba Rank Solea Rank Draco Min. migration gain
200 180 160 0
2
4
6
Number of Iterations
8
400 350 300 Rank Columba Rank Solea Rank Draco Min. migration gain
250 200 10
0
2
4
6
8
10
Number of Iterations
Fig. 2. Estimated execution times (ranks) of the application when it is migrated from draco to different machines at different execution points, using expressions 3 (left-hand chart) and 1 (right-hand chart)
372
R.S. Montero, E. Huedo, and I.M. Llorente
Figure 3 shows the measured execution profile of the application when it is actually migrated to solea and columba at different iterations, and the execution profile on draco without migration. These experimental results clearly show that reschedules based only on host performance and application progress (expression 3) may not yield in performance benefits. In particular, rescheduling the job based on expression 1 results in a performance gain of 13% (12% predicted). This resource selection strategy will reject job migration from third iteration, and prevents performance loses up to 15%, that would occur with the rescheduling strategy based on expression 3. Note that this drop in performance can always be avoided with a pessimistic value of Gm . However using expression 1 allows a more aggressive value of the migration gain threshold, and therefore an improvement in the response time of the application. 400
Prolog
Time (seconds)
350 300
Execution on draco
250
Migration
200
Execution on new host
150
Epilog
100 Min. migration gain
50
Migration at iteration 2
b so a lea
co
lum
b so a lea
lum co
b so a lea
lum co
Ex e (w cuti ith on ou on t m dr igr aco ati on ) co lum b so a lea
0
Migration at Migration at Migration at iteration 4 iteration 6 iteration 8
Fig. 3. Execution profile of the application when it is migrated from draco to solea or columba at different execution points
6
Conclusions and Future Work
In this work we have analyzed the relevance of resource proximity in the resource selection process in order to reduce the cost of file staging. In the case of opportunistic migration the quality of the interconnection network has also a decisive impact on the overhead induced by job migration. In this way, considering resource proximity to the needed data is, at least, as important as considering resource performance characteristics. We would like to note that the decentralized and modular architecture of the GridWay framework guarantees the scalability of the brokering strategy, as well as the range of application, since it is not specialized for a specific application set. We are currently applying the same ideas presented here to develop a storage resource selector program that considers the proximity to a set of replica files belonging to a logical collection. The storage resource selection process is
Grid Resource Selection for Opportunistic Job Migration
373
equivalent to the one presented in figure 1, although the discovery process is performed by accessing the Globus Replica Catalog.
References 1. Liu, C., Yang, L., Foster, I., Angulo, D.: Design and Evaluation of a Resource Selection Framework for Grid Applications. In: Proceedings of the 11th IEEE Symposium on High-Performance Distributed Computing. (2002) 2. Kennedy, K., et al.: Toward a Framework for Preparing and Execution Adaptive Grid Applications. In: Proceedings of NSF Next Generation Systems Program Workshop, International Parallel and Distributed Processing Symposium. (2002) 3. Allcock, W., Chervenak, A., Foster, I., Pearlman, L., Welch, V., Wilde, M.: Globus Toolkit Support for Distributed Data-Intensive Science. In: Proceedings of Computing in High Energy Physics (CHEP ’01). (2001) 4. Evers, X., de Jongh, J.F.C.M., Boontje, R., Epema, D.H.J., van Dantzig, R.: Condor Flocking: Load Sharing Between Pools of Workstations. Technical Report DUT-TWI-93-104, Delft, The Netherlands (1993) 5. Vadhiyar, S., Dongarra, J.: A Performance Oriented Migration Framework for the Grid. In: Proceedings of the 3rd IEEE/ACM Int’l Symposium on Cluster Computing and the Grid (CCGrid). (2003) 6. Wolski, R., Shao, G., Berman, F.: Predicting the Cost of Redistribution in Schedulling. In: Proceedings of the 8th SIAM Conference on Parallel Processing for Scientific Applications. (1997) 7. Allen, G., et al.: The Cactus Worm: Experiments with Dynamic Resource Discovery and Allocation in a Grid Environment. International Journal of HighPerformance Computing Applications 15 (2001) 8. Huedo, E., Montero, R.S., Llorente, I.M.: An Experimental Framework for Executing Applications in Dynamic Grid Environments. Technical Report 2002-43, ICASE NASA Langley (2002) submitted to Intl. J. Software Practice & Experience. 9. Wolski, R., Spring, N., Hayes, J.: The Network Weather Service: A Distributed Resource Performance Forecasting Service for Metacomputing. Journal of Future Generation Computing Systems 15 (1999) 757–768 10. Vazhkudai, S., Schopf, J., Foster, I.: Predicting the Performance of Wide-Area Data Transfers. In: Proceedings of 16th Int’l Parallel and Distributed Processing Symposium (IPDPS 2002). (2002)
Semantic Access Control for Medical Applications in Grid Environments Ludwig Seitz, Jean-Marc Pierson, and Lionel Brunie LIRIS, INSA de Lyon 7, av. Jean Capelle, 69621 Villeurbanne cedex, FRANCE {ludwig.seitz,jean-marc.pierson, lionel.brunie}@liris.insa-lyon.fr
Abstract. Access control is the field of security which deals with permissions to access resources, where resources may be computing power, storage capacity and data. On the other hand computational grids are systems, where users share those resources in a mostly transparent way. Grid access control poses novel challenges, since the distributed nature of grids make it difficult to manage access control by a central authority. Numerous overlapping domains with different access control policies exist and the sharing of storage resources makes it possible that data leaves the domain of its owner. To enable the owner to enforce his access control policy in such cases, access control solutions adapted to grid environments are needed. In this article we introduce Semantic Access Certificates as an extension to existing access control solutions for grids, to solve some problems that arise when grids are used to process medical data.
1
Introduction
Grid computing [1] is becoming a very popular solution for researchers looking for vast storage and computing capacity. A computational grid is a set of computing elements and data storage elements, heterogeneous in hard- and software at geographically distant sites which are connected by a network and use mechanisms to share resources as computing power, storage capacity and data. The usage of those computational grids is called grid computing. Different interested parties like high-energy physics, terrestrial observation and genome decoding have recognized the potential value of such tools (see [2], [3] or [4] for application examples). Another field of application is to use grid infrastructures for medical data processing (see [5], [6]). Goals of such infrastructures can be to provide researchers with a broad spectrum of data for analysis and to make it possible for patients to gain access to their data regardless where they are and where it is stored. It is clear that in medical data processing privacy protection needs to be enforced much more rigorously than in other application areas. Moreover, the
This work is partly supported by the R´egion Rhˆ one-Alpes and the French ministry for research ACI-GRID project (http://www-sop.inria.fr/aci/grid/public/)
H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 374–383, 2003. c Springer-Verlag Berlin Heidelberg 2003
Semantic Access Control for Medical Applications in Grid Environments
375
users of such systems (patient, medical staff) will mostly not be trained in computer security and thus need easy to use and mostly transparent services. Thus we need tools for classical security tasks like authentication, data integrity, private communications and access control in grid environments. Authentication is the task of proving ones identity over a computer network. Data integrity means to ensure that data has not been fraudulently manipulated, which can be done through electronic signatures. In private communications, involved parties want to communicate confidential data over an insecure channel, here encryption is the technique that is most commonly used. These fields of security are well covered in common applications and have been adapted to grid environments without major changes (see [7] or [8] for examples). This article focuses on access control for medical data in grid environments. We define access control as the domain of security, which deals with the permissions of entities to access resources, where entities may be users, groups of users, programs, daemons etc. and resources may be computing power, storage capacity and data. Unlike the other security aspects the structure of the grid poses novel problems for access control that can not be dealt with by classical solutions. These problems arise from the distributed nature of grid users and grid resources, where user activity may cross organizational boundaries, and data may be stored outside the user’s domain of control. This also means that users can have to handle the access rights of data belonging to the same semantic context but physically distributed on different storage elements. As we consider the patient to be the owner of his data in our medical application, flexible mechanisms are needed, that permit the patient to manage the rights of his data, even when he is not connected to its storage site. The structure of the article is as follows: In section 2 we describe the access control problems, arising specially in medical applications on grid environments. We then extract some desirable functionalities of an access control system for those applications. In section 3 we present classical approaches to access control while section 4 gives grid specific solutions. In section 5 we propose an extension to the existing grid mechanisms, designed to provide the desirable functionalities defined in section 2: access control through Semantic Access Certificates. Finally we summarize and give directions of our future work in section 6.
2
Medical Data on Grids
We present a somewhat constructed but still realistic use-case, to illustrate a class of access control problems we want to address in this article. Imagine a patient who visits a new doctor (for instance a patient in vacation visiting a doctor far from home). He wants to give this doctor access to some of his medical data, for example radiographs made at a certain hospital and the medical files of his former doctor. He doesn’t know the storage element, neither does he know the filenames or directories where his data is. The patient only wants to interact with the doctor and not with the storage site to grant the access. If a large number of medical files are concerned, he would prefer to grant
376
L. Seitz, J.-M. Pierson, and L. Brunie
access to all those files at once through some kind of classification mechanism, and not separately for every single file. On the other hand the doctor wants to use the rights given to him to locate and retrieve the data. If the treatment is done by a team of doctors, all team members should be able to use the access rights. The doctor also needs to be able to access future data of the patient, necessary for treatment and diagnosis. He doesn’t want to have to take difficult security measures to protect the permissions that were given to him by his patients, since this is not his specialty. From this example we extract the following list of desirable functionalities specific to medical data processing. – Group granting: It should be possible to specify permissions for groups of doctors, thus a patient could give medical teams access to necessary data in one operation. – Class granting: It should be possible to specify permissions to classes of data, to enable the patient to give access to all data concerning a certain field of medicine in one operation. – Storage location independence: Patients should be able to specify permissions without exact knowledge of the resource storage sites, because a system that requires the patient to keep track of his medical data and remember how to index it in the system would neither be accepted nor would it be usable. – Offline delegation: Patients should be able to assign permissions offline, without being connected to the storage sites. Since the patient is the owner of his medical data, and the storage site is mostly some kind of laboratory or clinic it would be cumbersome, if the patient had to contact the storage site, every time he wants to grant access to some of his data. – Personalized permissions: The assigned permissions should be bound to a specific entity or group of entities, for a given amount of time and not be usable by unauthorized entities. This minimizes the impact of loss or theft of those permissions. In the two following sections, we explain why classical access control schema do not address completely all these problems before giving a possible solution through enriched access certificates.
3
Classical Access Control
The UNIX style access control [9] with read, write and execute permissions and its user, group and other classification has the benefit of simplicity. However it does not provide any functionality to deal with distributed data of the same class. Access rights management has to be done online and there is no way to define specific access rights for a user that has no account on the system. Using access control lists (ACL) [10] adds some flexibility, however it is a local access control system as well and does neither provide offline access rights granting and distributed data classes management.
Semantic Access Control for Medical Applications in Grid Environments
377
Role based access control (RBAC) [11] groups entities and resources by so called roles. Those are collections of entities and access rights grouped together based on different tasks they perform in the system environment. Entities are given the right to take certain roles and thus use their permissions, to be able to use the resources that are necessary for their tasks. The RBAC approach permits to group access rights by semantic meanings, but to deal with distributed data, extensions of the classical RBAC approaches are needed. Also pure RBAC is somewhat non flexible for the granting of specific rights, since it only permits to grant rights by defining an appropriate role and assigning users the right to use it. One such extension, that aims specifically at distributed systems is OASIS [12],[13]. OASIS differs from other RBAC schemes in three major ways. First the role management is decentralized, which means that every service in the distributed system may define their own roles and control the conditions how those can be used. The services interoperate through Service Level Agreements (SLA). Second OASIS permits to parameterize roles, which allows to define generic roles (e.g. A is doctor of B) were the parameters can be chosen for specific applications. Third it replaces priviledge delegation by a mechanism called appointment. Certain roles have the function to permit the issuing of appointment certificates, that permit to activate roles together with an authentication mechanism. The OASIS approach permits to manage access control for distributed data of the same semantic class, however it does not permit offline granting of access rights, since one has to activate an appointment role to grant rights.
4
Grid Access Control Solutions
This section deals with today grid solutions for access control. We present three common grid computing environments, Condor [14], Legion [15] and Globus [16] and we argue why their solutions have to be extended with further functionality for medical grid applications. Condor [14] is a specialized workload management system for computeintensive jobs. Its resource management mechanisms are similar to UNIX style access control. Main differences are some additional modes of access besides traditional read and write permissions. Also the classical meanings of the read and write permissions are slightly changed, since all permissions are not relative to files, but to the Condor system itself. Therefore read access means the permission to access information from Condor, write to change settings etc. The Legion project [15] uses an object oriented approach to grid computing. Thus resources, as files, services and devices are considered as objects and access to those are through functions of these objects. The Legion access control approach is that each object is responsible for enforcing own access control policy. Thus each object has a MayI function, invoked before any other functions of the object may be called. This function allows resource owners to define their own access control mechanisms. A default MayI implementation exists that is based on ACL’s and credential checking (authentication).
378
L. Seitz, J.-M. Pierson, and L. Brunie
The Globus grid toolkit [16] proposes mechanisms for translating users grid identities into local identities. This would allow users to sign onto the grid and then use resources without further authentication (single sign-on). The benefit for local resource providers would be that through the translations mechanism, local access control policies can be enforced. More recently Pearlman et al.[17] have also been working on an access control system that relies on a Community Authorization Service (CAS). The idea is that users shall be able to request access right at a CAS server, which will assign them capabilities, based on the request and the user’s role within the community. The user can present those capabilities at a resource server to gain access on behalf of the community. Thus this approach integrates the benefits of RBAC into a grid access control system. All approaches described above lack some mechanisms for managing access to distributed data belonging to a common semantical class and they do not provide mechanisms for offline granting of rights. Therefore they have to be extended for use in our medical grid applications.
5
Semantic Access Certificates
A number of classical approaches exist that grant access to data through the use of electronic documents, like the Kerberos V5 protocol [18] access tickets, however these approaches do not permit the management of transparent access to distributed data, since the storage site has to be known, to issue a valid ticket. We have therefore decided to enrich the concept of access tickets to get rid of those limitations. 5.1
The Structure of a Semantic Access Certificate
We will use a doctor/patient scenario as in our motivation to describe the structure and use of access certificates, this does of course not mean, that we limit ourselves to use in this context, but serves to clarify the relations between the involved entities. We first emphasize the medical data we want to give (or refuse !) access to. Upon creation of a medical data file, it will be assigned a certain set of metadata. This set necessarily contains an unique identifier of the patient that owns the data (since one of our assumption is that one data belong to one patient), the date of creation, and a hash of the data signed by the patient to guaranty integrity. Additionally, data can be associated to classes, to allow for more coarse grained access control decisions. In medical applications, classifying data by medical specialty allows a patient to give access rights to a doctor for the specific class of data that is needed for his treatment or diagnosis. A patient visiting a dentist should give him access to all dental data and medicine allergy, while not giving access to other classes of information such as psychological analysis or lung radiographs. Some ontology of the medical domain should allow for a fine
Semantic Access Control for Medical Applications in Grid Environments
379
classification of the data produced at different medical services (but note here that such an ontology does not currently exist globally, but this research field is very active today, see the US Unified Medical Language System for instance [19]). We define Semantic Access Certificates (SAC) as electronic documents that contain the following information (see figure 1): – – – – – –
an unique identifier of the doctors that are granted access, an unique identifier of the patient granting access, an unique identifier of the data or class of data to which access is granted, the allowed modes of access, a validity period, and the electronic signature of the entity granting the rights.
Unique identifier of entities granted access rights Unique identifier of owner of the concerned data
concatenation
Cryptographical Hash (e.g. SHA1, MD5 etc.) Hash value
Identifier of the data access is granted to Allowed modes of access
Encryption
Owners private key
Certificate’s signature Signed Access Certificate
Certificate validity period
Fig. 1. Components of a signed access certificate
Let us now detail and argue each of the fields present in this certificate. First appears the doctor (or the group) for whom the certificate has been issued : For a single doctor its public key may be used as unique identifier, management of groups of medical staff is done through trusted authorities that issue group membership certificates. Similarly, the unique identifier of the patient may be its public key. Creating unique identifiers for files can be done by concatenating the patient identifier with a hash (MD5, SHA1, ...) of the file content (or the semantic class name). The semantic value transported with the certificate when classes of access are given that way allows for a much more flexible and easy to use system. The users can thus effectively give access to entire part of his medical data. The classification of data allows to give access rights based on semantical criterions,
380
L. Seitz, J.-M. Pierson, and L. Brunie
as it is independent of storage sites, even access to distributed data of the same semantical context may be managed this way. Fourth, the allowed access mode should be as simple as “read”, “write” or “read/write” or more complicated such as “read anonymised” (for instance when an user wants to give access to some of its medical information to medical researchers for epidemiologic studies) or “append information” (when an user wants the doctor to add new information to its medical file). The handling of these different access mode lies beyond the scope of this article. The validity period in access certificates is generally set so that a hacker stealing the certificate won’t be able to decode it and use it before it expires. We see another interest for us : a patient should want a doctor to access his medical data during a given period of time (for instance while in the doctor office or for the treatment duration, but not longer). Finally, the electronic signature allows classically for the integrity of the issued certificate. To revoke SACs the patient will be able to issue a SAC revocation request, that uses the identifiers of the patient and the doctor holding the SAC with the signature to form an unique identifier of the SAC. This revocation request will be broadcasted to the resource server sites, in a similar way as certificate revocation lists. As the patient owns his data, he is the only one that can issue access certificates or change the data. A copy of the unique file identifiers of all his data need to be stored with him, so he can issue access certificates to specific data files. If the data is associated to a class, the classification also needs to be stored with him. For the revocation mechanism, the unique identifiers of the certificates need to be stored with the patients too. Such information will be stored on a smartcard or a similar cryptographic token along with the patients public/private key pair. Those tokens will be kept by the patients like today medical insurance cards. 5.2
Usage of SAC
Let’s imagine a scenario where a doctor has successfully acquired a valid SAC and wants to access some data. The process would have the following actors: The doctor requesting some data, the doctors grid interface, a resource broker mechanism provided by the grid infrastructure to locate resources and resource server sites that offer access to resources. The request formulated by the doctor would be translated by his grid interface into a grid readable format, like for example a request to the Globus GASS system [20]. The exact format of the request lies beyond the scope of this article and shall thus not be treated any further. The whole process that follows the submission of a request should remain transparent to the doctor, until the request results in either success or definitive failure. Figure 2 illustrates the use of access certificates and the protocol is specified here:
Semantic Access Control for Medical Applications in Grid Environments Patient (Data owner)
Resource Brooker (RB)
Doctor (Data user)
381
Resource Server (RS)
Issues signed access certificate (SAC) SAC
Grid Interface (GI)
1. Doctor specifies request to GI
2. GI requests the data 3. RB Finds best 4. RB returns address RS of RS 5. GI Authenticates at RS 6. GI Requests Data 7. Negociation of necessary SACs 8. GI Sends appropriate SACs 9. RS Verifies that SAC corresponds to 10. RS Grants access to data data and its onwer (if verification succeeds) SAC
Fig. 2. Usage of access certificates
1. The doctor specifies what data it wants to his grid user interface. 2. The doctor’s grid interface contacts the resource broker and submits the data request. 3. The resource broker locates the best resource server1 that has the requested data available. 4. The resource broker transmits the address to the doctor’s grid interface. 5. The doctor’s grid interface authenticates at the resource server. 6. The doctor’s grid interface transmits the data request. 7. The doctor’s grid interface and the resource server negotiate which access certificates are needed for the requested access. The negotiation is carried out in the following steps: a) The doctor’s grid interface submits the request to the resource server b) The resource server locates the requested data’s meta-data section and extracts the patient’s unique identifier and the data classification if one exists. It then returns those to the grid interface. c) The grid interface uses the provided meta-data to scan the doctor’s access certificate repository for appropriate access certificates and returns those to the resource broker. 1
The determination of the best server could be done by various criterions like: the one with the fastest connection, one in the same country for legal reasons, a trusted site etc.
382
L. Seitz, J.-M. Pierson, and L. Brunie
8. The doctor’s grid interface sends the appropriate access certificates to the resource server. 9. The resource server checks that the access certificates bears a valid signature by the patient owning the data. 10. If access can be granted the resource server gives access to the grid interface, otherwise it returns an appropriate request failure message. 11. If the access is granted the grid interface proposes the access to the doctor, if the request failed due to insufficient permission the doctor is notified, if the request failed for other reasons, the grid interface resubmits the request to the resource broker, excluding the addresses of the previously contacted resource servers in this new request. In section 2, we have exhibited five desirable functionalities of access control in medical area. We now address their adequation with the SAC definition and protocol. The SACs allow directly, from their definition, for Group granting and Class granting, a certificate being issued for a doctor or group of doctors and for a data or a given class of data). Storage location independence is assured since the identifier depends only on the patient and the content of the data. Thus any replica of the data stored on the grid can be accessed with the same SAC. Access certificates can be granted offline (Offline delegation) i.e. without being connected to the resource server site. The patient simply has to prepare the electronic document in the right format, to assure he has the right identifiers and to sign it digitally. From this point on, the certificate is bound to the doctor or group of doctors (Personalized permissions) it was issued for and can be transmitted even through an insecure channel. The electronic signature of the patient ensures its integrity and thus no unauthorized person can use the SAC.
6
Conclusion
We have proposed in this article a novel approach to access right management on grids, in the medical data field. Our contribution is the definition of a Semantic Access Certificate (SAC), holding the access control policy not only based on roles (patient, doctor, nurse, etc.) but also on the semantic of the stored data (the specialty it belongs to). While being illustrated here in the medical field, the proposed schema can be used in any application where privacy protection have to be enforced. However this article only covers a part of the access control problems arising in medical data processing on the grid. Special modes of access like anonymized read for medical research, problems of access logging and data persistence are not treated in this article. Therefore it will need to be expanded and should not be used alone. We are currently working on the design of reactive documents that will carry their special access mode programs, access logging and perenity enforcement mechanisms with them. These will allow the users to specify their own access modes, logging and perenity conditions and to enforce them, even if the data is stored at untrusted sites.
Semantic Access Control for Medical Applications in Grid Environments
383
References 1. Foster, I., Kesselman, C., eds.: The Grid Blueprint for a New Computing Infrastructure. Morgan Kaufmann Publishers, Inc. (1999) 2. European Data Grid: The datagrid project. http://eu-datagrid.web.cern.ch/eudatagrid/ (2001) 3. Cactus Community: The cactus code. http://www.cactuscode.org/ (2002) 4. National Institute of Advanced Industrial Science and Technology: Grid technology research center. http://www.aist.go.jp/ (2002) 5. MEDIGRID: Medical data storage and processing on the GRID. http://creatiswww.insa-lyon.fr/MEDIGRID (2002) 6. University of Manchester: Manchester Visualization Centre: ix-grid. http://www.sve.man.ac.uk/mvc/Research/iX-Grid/ (2002) 7. Foster, I., Kesselman, C., Tsudik, G., Tuecke, S.: A security architecture for computational grids. In: Proceedings of the 5th ACM Conference on Computer and Communications Security Conference. (1998) 83–92 8. Ferrari, A., Knabe, F., Humphrey, M., Chapin, S., A.Grimshaw: A flexible security system for metacomputing environments. In: Proceedings of the High Performance Computing and Networking conference 99 (HPCN’99). (1999) 9. Nemeth, E., Snyder, G., Seebass, S., Hein, T.: UNIX System Administration Handbook (3rd Edition). Prentice Hall PTR (2000) 10. Lampson, B.: Protection. In: Proc. of the 5th Princeton Conf. on Information Sciences and Systems, Princeton, 1971. Reprinted in ACM Operating Systems Rev. Volume 8, 1. (1974) 18–24 11. Sandhu, R.: Role-based access control. In: Advances in Computers. Volume 46. Academic Press (1998) 12. Yao, W., Moody, K., Bacon, J.: A Model of OASIS Role-Based Access Control and its Support for Active Security. In: Proceedings of Sixth ACM Symposium on Access Control Models and Technologies, SACMAT. (2001) 171–181 13. Bacon, J., Moody, K., Yao, W.: Access Control and Trust in the use of Widely Distributed Services. In: Lecture Notes in Computer Science. Volume 2218. Springer (2001) 295–310 14. Condor Team of the University of Wisconsin: Condor, high troughput computing. http://www.cs.wisc.edu/condor/ (1988) 15. Legion Research Group of the University of Virginia: Legion, a worldwide virtual computer. http://legion.virginia.edu/ (1993) 16. Globus Project: Globus toolkit. http://www.globus.org/ (1998) 17. Pearlman, L., Welch, V., Foster, I., Kesselman, C., Tuecke, S.: A community authorization service for group collaboration. In: Proceedings of the 2002 IEEE Workshop on Policies for Distributed Systems and Networks. (2002) 18. Kohl, J., Neuman, C.: The Kerberos Network Authentication Service (V5). Technical report, The Internet Engineering Task Force IETF (1993) http://www.ietf.org/rfc/rfc1510.txt. 19. UMLS: Unified medical language system. (http://www.nlm.nih.gov/research/umls/) 20. Bester, J., Foster, I., Kesselman, C., Tedesco, J., Tuecke, S.: Gass: A data movement and access service for wide area computing systems. In: Proceedings of the Sixth Workshop on I/O in Parallel and Distributed Systems. (1999)
Automated Negotiation for Grid Notification Services Richard Lawley1 , Keith Decker2 , Michael Luck1 , Terry Payne1 , and Luc Moreau1 1
2
Department of Electronics and Computer Science, University of Southampton, UK {ral01r,mml,trp,L.Moreau}@ecs.soton.ac.uk Computer and Information Sciences Department, University of Delaware, Newark, DE 19716
[email protected] Abstract. Notification Services mediate between information publishers and consumers that wish to subscribe to periodic updates. In many cases, however, there is a mismatch between the dissemination of these updates and the delivery preferences of the consumer, often in terms of frequency of delivery, quality, etc. In this paper, we present an automated negotiation engine that identifies mutually acceptable terms; we study its performance, and discuss its application to a Grid Notification Service. We also demonstrate how the negotiation engine enables users to control the Quality of Service levels they require.
1
Introduction
Notification services play an important role within distributed systems, by acting as intermediaries responsible for the asynchronous delivery of messages between publishers and consumers. Publishers (such as information services) provide information which is then filtered and delivered to subscribed consumers [1,2,3] based on a specification of topic and delivery parameters. Notification service features include persistent, reliable delivery and prioritisation of messages. Notifications may include announcements of changes in the content of databases [4], new releases of tools or services, and the termination of workflow execution. Notification services can also be used for replicating service directory contents [5]. As such, the Grid community has recognised the essential nature of notification services such as the Grid Monitoring Architecture [6], the Grid Notification Framework [7], the logging interface of the Open Grid Services Architecture [8] and peer-to-peer high performance messaging systems like NaradaBrokering [9]. They are also core architectural elements within the MyGrid [10] project. While the mechanisms for asynchronous notifications are well understood and robust implementations can be found, some issues still remain open. For example, providers hosting databases in the bioinformatics domain prefer to control the frequency at which notifications are published (such as daily digests), and discourage clients from continually polling for changes. However, clients have their own preferences about the frequency, format, quality or accuracy of the information being propagated. Similarly, many services within this domain are hosted by public institutions and are free to the community, but there are also paying customers expecting a certain quality of service from providers. The prices charged for notifications will affect the type (e.g. quality and frequency) of messages sent. As these examples suggest, both providers and consumers have preferences about the way notifications should be published, yet current notification H. Kosch, L. B¨osz¨orm´enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 384–393, 2003. c Springer-Verlag Berlin Heidelberg 2003
Automated Negotiation for Grid Notification Services
385
service technologies provide no support for determining a set of parameters that would be acceptable to both parties. Automatically finding mutually acceptable notification parameters for a set of terms (e.g. price, bandwidth, etc) can be viewed as a search for an optimum in a multidimensional space. A cooperative approach requires both parties to make valuation functions and preferences available to a search component. Here, preferences and utility functions are shared, enabling the optimal values to be found. However, it is not always possible to share preferences and valuation functions freely — businesses may view preferences as private, and valuations may be linked to a locally sensed environment. In these situations, an automatic search is not possible, as an optimum cannot be calculated. These situations require competitive approaches using mechanisms such as negotiation. Various approaches exist; in this paper we present a bilateral negotiation framework [11] that is applicable to the context of notification service negotiation. Our contributions are a practical implementation of an automatic negotiation engine, a study in terms of performance of bilateral negotiation and a study of negotiation in the specific context of notification services. This paper is organised as follows. Section 2 discusses negotiation in general. Section 3 describes the design of our system. In Section 4 we study the negotiation engine in general and more specifically in Section 5. We discuss related work in Section 6 and conclude in Section 7.
2
Negotiation
Negotiation is the process by which two or more parties exchange proposals in order to reach a mutually acceptable agreement on a particular matter. Parties in negotiation exchange proposals [12] that are either accepted or rejected. Rejection involves either turning down the proposal and allowing another to be sent or submitting a counterproposal, so that both parties converge towards an agreement. Utility functions (used to evaluate a proposal) and preferences (defining an acceptable range of values for each term) remain private and, because they are stored locally, can be linked to external conditions such as resource levels. For this reason, it is not practical to use cooperative searching in a Grid environment, where limited system resources need to be allocated. One solution is Faratin’s negotiation decision functions [11] algorithm, which is a bilateral negotiation model that allows external resource functions to be used to evaluate proposals and generate counter-proposals. In Faratin’s algorithm, the methods for generating proposals and counter-proposals are based on tactics and strategies. Tactics are functions that generate the value for a single negotiation term for inclusion in a proposal, and come in different flavours: timedependent tactics use the amount of time remaining in the negotiation thread to concede, whereas resource-dependent tactics use a resource function to determine how much of a particular resource is consumed. This resource may be the number of negotiations currently taking place or the load on the system, and may involve callbacks to monitor external resources. To combine different tactics during counter-proposal generation, strategies are used. These modify the weightings given to each tactic, which can be changed during a negotiation, for example to honour a resource-dependent tactic at the start of a negotiation and a time-dependent one nearer the deadline. Utility functions
386
R. Lawley et al.
evaluate the utility of a single negotiation term in a proposal. These can be simple linear functions or more complex callback functions to link the utility to resource conditions. The utility of a proposal is a weighted summation of the utility of the elements within the proposal. Proposals become acceptable when the counter-proposal generated has lower utility than the incoming proposal. Faratin’s algorithm is the basis for the rest of this paper.
3
Negotiation Engine Description
Our system is intended to allow applications to be programmed without specific knowledge of negotiation protocols. The actual negotiation mechanism is based on the work of Faratin [11], which allows a host to easily supply external conditions, such as resource levels, as inputs into the negotiation process without knowing how to negotiate. Negotiations take place between a requester and a requestee, and we assume that there is always an item that is the subject of the negotiation. The conditions being negotiated over are called negotiation terms, and can be such things as cost or duration. A conversation between a requester and requestee, where proposals are exchanged, is called a negotiation thread. Each party has a set of preferences, which consist of two values: an ideal value and a reservation value. The ideal value represents the initial negotiating value (i.e. the value the party would like to get in an ideal world) while the reservation value is the limiting point of concession. Values beyond the reservation value are unacceptable, and the negotiation will fail if it is not possible to find a value that satisfies all preferences. The negotiations in this system work to deadlines that are measured in terms of the number of messages exchanged between the two parties — the negotiation thread length. We refer to the application containing the negotiation component as a host and to the complete system as the negotiation engine. Although we conceptually regard the negotiation engine as an entity shared by the requester and requestee, it has been implemented as negotiation components distributed between each party. This approach maintains privacy of preferences and utility functions. A description of the negotiation process is depicted in Figure 1. Before a negotiation starts, both parties initialise their negotiation components with their preferences. Then, negotiation process proper is initiated with the requester sending a proposal to the requestee. The communication mechanism is left for the host to implement. As discussed in the previous section, when a proposal is received by party p, a counter-proposal is generated. Using p’s utility functions, p’s negotiation component determines if the utility of the counter-proposal is higher than the incoming proposal. If so, the counter-proposal is sent. This cycle continues until the incoming proposal has a higher utility, at which point an acceptance message is sent. Both negotiation components then give their hosts a successful proposal. Note that acceptance of a proposal is not a commitment — it is left to the host to commit, allowing negotiations with many parties. If the deadline passes before a successful proposal is found, a failure message is sent to the other party, and the hosts are notified. Negotiations can also be terminated by means of a failure message for reasons such as a system shutting down or a deal being made with another party.
Automated Negotiation for Grid Notification Services Requester
Requester NC
Requestee NC
registerPreferences(item)
387
Requestee
registerPreferences(item)
negotiateFor(item, requestees)
propose(proposal) callback
callback
callback generateCounterProposal() propose(proposal)
generateCounterProposal()
evaluateProposal(proposal)
evaluateProposal(proposal)
callback
accept
accept
accept(proposal)
accept(proposal) negotiationSuccess(acceptableProposal)
negotiationSuccess(acceptableProposal) makeCommitment(acceptableProposal)
Fig. 1. Sequence diagram showing message flow during a negotiation
Θe
nei nrr
Φ Optn
nei = requestee ideal value
ner Θr
nri
-
nri = requester ideal value ner = requestee reservation value nrr = requester reservation value
Fig. 2. Effect of environment parameters on preferences, also showing optimal value
4
Experimental Evaluation of Negotiation Engine
To verify the suitability of our negotiation engine for a Grid notification service, we must check that it scales up predictably to handle more negotiation terms, and longer negotiations, without any adverse performance. To determine that this component is suitable for our purposes, we performed a number of experiments. 4.1
Experiment Setup
The set of varying factors such as acceptable ranges and deadlines are grouped into an environment. Running two negotiations in the same environment produces identical results. As there is an infinitely large space of environments we generated a range of random environments using the methods from Faratin [11]. For each term, a fixed value minr was chosen, representing the requester’s reservation value. Random values within predefined ranges were assigned to the parameters Θr , Θe and Φ, where Θr represents the size of the acceptable region for the requester r, Θe represents the size of the acceptable region for requestee e, and Φ represents the degree of overlap between the acceptable ranges, with 0.99 indicating almost no overlap and 0 indicating complete overlap. These parameters are illustrated in Figure 2.
388
R. Lawley et al.
We used six tactics — three from the time-dependent and three from the resourcedependent families as in [11]. Utility functions are linear functions based on the preferences. The experiments all had the same basic structure — each tactic was played against each of the tactics (including itself) in each of the generated environments. This allows us to build up some average values demonstrating the sort of results we should expect from a real-world implementation. As the design of the system is independent of any communication mechanism we coupled the negotiation components together directly using method calls. Experiments measuring time only examined execution time. 4.2
Hypotheses and Results
The experiments described in this section were intended to determine the effects of varying both the deadline for negotiations and the number of negotiation terms. We consider the number of messages exchanged in a negotiation to be the primary component of time, as the dominant factor for message exchange in a real system is the transmission time. By contrast, transmission time in our experiments is very low, since the components are coupled by method calls, but we also measured execution time at the end of this section. Variable Deadline. Sometimes, negotiations should be completed within short deadlines, but this yields worse results than long deadlines [11]. To examine how the utility varies with the deadline we varied the deadline between 1 and 100 messages, using a single negotiation term. Hypothesis: With short deadlines the utility to both parties is poor. As deadlines increase, utility also increases, but at a decreasing rate, since a utility of 1 would indicate that no concessions were made, and is unlikely. The percentage of successful deals made increases as the deadline increases. Figure 3A shows that the utility (Ur and Ue ) for both parties is low for short deadlines. As the deadline increases, the utility also increases. The average optimal utilities (Optr and Opte ) are plotted on the graph — these are time-independent and therefore constant. They appear to be asymptotes to the utility curve. Figure 3B shows that the percentage of successful negotiations has a similar curve, fitting our assumption that there is a predictable curve that can be used to determine the effect of limiting the deadlines of negotiations. This can be used to determine what sort of values should be used to limit the length of a negotiation thread without trading-off too much utility. To determine how much of the available time is used, we examined a subset of the data using negotiations between time-dependent linear tactics. We plotted the utilities of the outcomes and the time taken for the negotiations for each environment, sorted by increasing client utility and provider utility. Figure 3C shows that the amount of time used in a negotiation ranges between 30% and 100% of the available time. There appears to be an interesting trend where the utility increased as the number of messages decreased. Our explanation for this is that the negotiations taking less time and giving greater utility have a better environment to negotiate in. The parameter of the environment that has the
Automated Negotiation for Grid Notification Services A
B
0.3 Success Rate
Utility
0.25 0.2 0.15 0.1
Ur Ue OptUr OptUe
0.05 0 0
10
20
30
40
50
60
70
80
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
90 100
C
30
40
50
60
70
80
90
100
D
25
0.8
Messages Phi
20
0.6 0.5 0.4 Ur Ue OptUr OptUe Messages
0
Messages
Messages
0.7 Utility
20
30
0.9
0.1
10
Deadline
1
0.2
SuccessRate 0
Deadline
0.3
389
15 10 5 0 -5
Negotiation Number
Negotiation Number
Fig. 3. A) Utilities, B) Success Rate, C) Utility vs. time used, D) Time vs. Φ
most significance is Φ, controlling the amount of overlap between the acceptable regions. This is plotted against the number of messages exchanged in Figure 3D, confirming this theory — negotiation finishes quicker with better negotiation environments and gives a greater overall utility. Multiple Negotiation Terms. If this negotiation engine were to be deployed in the notification service, it would not be negotiating over a single negotiation term. There would be many terms, so we must ensure that the system scales up as the number of terms increases. In consequence, the negotiations were evaluated with the number of negotiation terms set at every value between 1 and 25. The experiments used deadlines of between 30 and 60 messages. Hypothesis: As the number of negotiation terms increases, the number of messages exchanged during a negotiation remains constant, as each negotiation term is independent and the component concedes on all terms at the same time. Thus the length of the negotiation is constrained by the most limiting negotiation term. The utility of the outcome remains constant since the utility is limited by the most constraining negotiation term. As the number of terms increases, the time taken to perform the negotiations increases linearly, assuming that all the terms are evaluated using the same linear utility functions and tactics. Figure 4A shows that the average utility achieved with a varying number of negotiation terms remains fairly constant, and does not begin to drop with respect to the number of terms. Similarly, Figure 4B shows that the time appears to be increasing linearly. While there are a few deviations upwards of the linear trend, these can be explained by garbage collection being triggered in the execution environment.
390
R. Lawley et al. A
B 14
0.25
12
0.2
10
Time (ms)
Utility
0.3
0.15 0.1
Ur Ue OptUr OptUe
0.05
8 6 4 2
0
0 0
5
10
15
20
25
0
5
10
Number of Terms
15
20
25
Number of Terms
Fig. 4. Variable number of negotiation terms: A) Utilities of outcome, B) Time Taken
Execution Time. To confirm that the time taken for negotiation increases linearly with the number of messages exchanged, we measured the execution time for the negotiation to complete, averaged over 100 times to reduce inaccuracies. For a given number of messages exchanged, we recorded the average, minimum and maximum times taken for negotiations exchanging that quantity of messages. The deadlines in this experiment were between 30 and 60 messages. Hypothesis: As the number of messages exchanged between negotiations takes place, the corresponding increase in real time taken will be linear. As shown in Figure 5 there is a wide range of results for each number of messages exchanged. However, the average line is close to the minimum, indicating that few results are significantly higher than average. (We have explained the higher results as garbage collection in Java.) Failed negotiations are plotted as 0 messages, although they always take up to their deadline to complete, because reaching the deadline implies failure. Since the time taken for failed negotiations is approximately the same as with high numbers of messages, we conclude that the time taken is linearly related to the number of messages exchanged.
35 Mean Time Min Time Max Time
30
Time (ms)
25 20 15 10 5 0 0
10
20 30 40 Number of Messages
50
60
Fig. 5. Graph of time taken for negotiations
Automated Negotiation for Grid Notification Services
391
0.9 0.8 Provider Utility
0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 -0.1 0
50000 100000 150000 200000 250000 300000 Expected number of queries
Fig. 6. Provider utility against expected number of calculations
5
Evaluation for the Notification Service
We chose a specific case from the bioinformatics field as an example use of the notification service. The SWISS-PROT Protein Knowledgebase is a curated protein sequence database providing a high level of annotation, minimal redundancy and high integration with other databases [13]. Over the past 14 months it has grown by 20% with approximately 200 changes per day. As an example, we assume 1000 subscribers are interested in anything that matches 100 different sequences, and a particular similarity search takes 1 second and can be run iteratively to refine results. To provide the notifications, the provider runs the similarity search after data has been added or annotated. Although subscribers are happy to receive updates every few hours, this would place an unacceptable load on the provider. For example, daily searches iterated 5 times require five million searches per day. We use two negotiation terms for this experiment. Frequency represents the maximum number of hours between notifications: for the provider, this is between 24 and 168 hours, whereas for the consumer it is between 5 and 120 hours. The second term is the number of iterations of the search. The provider prefers this to be between 1 and 3, the client between 1 and 5. The preferences for the provider are kept constant and a random variation is introduced into the client preferences to simulate different clients. Negotiation deadlines are between 30 and 60 messages. After running the negotiations, the average value for the frequency was 67.6 hours and the number of iterations was 2.31 iterations. This works out as 651,000 searches per day, a reduction of 87%. The average utilities over the experiments were 0.39 for the consumer and 0.30 for the provider. Figure 6 shows that the provider utility decreases as the number of computations increases. The curve would have been smoother if we had weighted the utility of each term differently, as they influence the number of calculations in different ways. While these figures are based on a hypothetical situation, it demonstrates that the consumer’s requirements can be satisfied while reducing the number of searches the provider has to carry out to do so, indicating that a better Quality of Service can be achieved by using negotiation to establish QoS terms.
392
6
R. Lawley et al.
Related Work
There has been much research in automated negotiation. Bartolini et al. [14] produced a framework for negotiation allowing different types of negotiations to be specified using rules. We chose not to use this, as we wanted the two parties to be able to communicate directly rather than to use a third party to carry out the negotiation. Jennings et al. [15] give some details on a framework for automated negotiation, which focuses on rules of negotiation, and allowing many different types of negotiation to be carried out within the same framework. The RFC1782 [16] describes a simple extension to Trivial File Transfer Protocol (TFTP) to allow option negotiation prior to the file transfer. Although the idea is similar in principle, there is no economically sound model of negotiation used. A number of examples of subscription services have been identified in the Grid community, allowing the information to be used for monitoring or rescheduling of allocations [17]. The Grid monitoring architecture [6] is a distributed architecture allowing monitoring data to be collected by distributed components. The Grid Notification Framework [7] allows information about the existence of a grid entity, as well as properties about its state, to be propagated to other grid entities. NaradaBrokering [9] is a peer-to-peer event brokering system supporting asynchronous delivery of events to sensors, high end performance computers and handheld devices. The Open Grid Services architecture [8] has also identified a logging service as an essential component: it also relies on producer and consumer interfaces. None of these notification services support negotiation and might benefit from our work.
7
Conclusion and Future Work
This paper has presented our design for a negotiation engine for inclusion in a Notification Service. We have presented our reasons for choosing a competitive negotiation method and shown how our negotiation engine works. We have also shown that the performance of the system is predictable and does not have any adverse effects when used with many negotiation terms. Further development of this work is ongoing, and we have identified several areas we would like to proceed with. In our current system all negotiation terms are independent and negotiations concede on all of them. We would like to investigate introducing dependencies between negotiation terms and the possibilities of trading off one term against another. Negotiations currently take place between a requester and a notification service. We envisage a system where negotiations are chained between consumer, notification service and provider, and will examine negotiation in this situation. Finally, it is worth noting that this negotiation component will be deployed in the Notification Service in a real Grid environment using MyGrid as a testbed.
Acknowledgements. This research is funded in part by EPSRC myGrid project (ref. GR/R67743/01).
Automated Negotiation for Grid Notification Services
393
References 1. 2. 3. 4. 5. 6. 7. 8. 9.
10. 11. 12.
13. 14.
15.
16. 17.
Java Message Service API. http://java.sun.com/products/jms/ (1999) Object Management Group: Event service specification. www.omg.org (2001) Object Management Group: Notification service specification. www.omg.org (2002) Oinn, T.: Change events and propagation in mygrid. Technical report, European Bioinformatics Institute (2002) UDDI version 3 features list. www.uddi.org (2001) Tierney, B., Aydt, R., Gunter, D., Smith, W., Swany, M., Taylor, V., Wolski, R.: A grid monitoring architecture. Technical report, GGF Performance Working Group (2002) Gullapalli, S., Czajkowski, K., Kesselman, C.: Grid notification framework. Technical Report GWD-GIS-019-01, Global Grid Forum (2001) Foster, I., Gannon, D., Nick, J.: Open grid services architecure: A roadmap. Technical report, Open Grid Services Architecture Working Group (2002) Fox, G., Pallickara, S.: The narada event brokering system: Overview and extensions. In Arabnia, H., ed.: 2002 International Conference on Parallel and Distributed Processing Techniques and Applications (PDPTA’02). Volume 1., Las Vegas, Nevada, CSREA Press (2002) 353–359 myGrid Project: mygrid website. http://www.mygrid.org.uk/ (2003) Faratin, P., Sierra, C., Jennings, N.R.: Negotiation decision functions for autonomous agents. International Journal of Robotics and Autonomous Systems 24 (1998) 159–182 Sierra, C., Jennings, N.R., Noriega, P., Parsons, S.: A framework for argumentation-based negotiation. In: Intelligent Agents IV: 4th International Workshop on Agent Theories Architectures and Languages, (Springer) EBI: Swiss-prot website. http://www.ebi.ac.uk/swissprot/ (2003) Bartolini, C., Preist, C., Jennings, N.R.: Architecting for reuse: A software framework for automated negotiation. In: 3rd International Workshop on Agent-Oriented Software Engineering, Bologna, Italy (2002) 87–98 Jennings, N.R., Parsons, S., Sierra, C., Faratin, P.: Automated negotiation. In: 5th International Conference on the Practical Application of Intelligent Agents and Multi-Agent Systems, Manchester, UK (2000) 23–30 Malkin, G., Harkin, A.: TFTP option extension (rfc 1782). Technical report, Network Working Group (1995) Schwiegelshohn, Yahyapour: Attributes for communication between scheduling instances. Technical report, GGF, Scheduling Attributes Working Group (2001)
GrADSolve – RPC for High Performance Computing on the Grid Sathish Vadhiyar, Jack Dongarra, and Asim YarKhan Computer Science Department University of Tennessee, Knoxville {vss, dongarra, yarkhan}@cs.utk.edu
Abstract. Although high performance computing has been achieved over computational Grids using various techniques, the support for high performance computing on the Grids using Remote Procedure Call (RPC) mechanisms is fairly limited. In this paper, we discuss a RPC system called GrADSolve that supports execution of parallel applications over Grid resources. GrADSolve employs powerful scheduling techniques for dynamically choosing the resources used for the execution of parallel applications and also uses robust data staging mechanisms based on the data distribution used by the end application. Experiments and results are presented to prove that GrADSolve’s data staging mechanisms can significantly reduce the overhead associated with data movement in current RPC systems.
1
Introduction
The role of Remote Procedure Call (RPC) [2,15,18,4,9,3,12] mechanisms in Computational Grids [7] has been the subject of several recent studies [14,6,13]. Although traditionally RPCs have been viewed as communication mechanisms, recent RPC systems [3,12] perform a wide range of services for problem solving on remote resources. Computational Grids consist of large number of machines ranging from workstations to supercomputers and strive to provide transparency to the end users and high performance for end applications. While high performance is achieved by the parallel execution of applications on large number of Grid resources, user transparency can be achieving by employing RPC mechanisms. Though there are a large number of RPC systems that adequately support the remote invocation of sequential software from sequential environments, the number of RPC systems for supporting invocation of parallel software are relatively few [3,12,13,10,5]. Some of these parallel RPC systems [13,10] require invocation of remote parallel services from only parallel clients. Some of the RPC systems [3,12] support only master-slave or task farming models of parallelism. A few RPC systems [12,3] fix the amount of parallelism at the time when the services are uploaded into the RPC system and hence are not adaptive to
This work is supported in part by the National Science Foundation contract GRANT #EIA-9975020, SC #R36505-29200099 and GRANT #EIA-9975015
H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 394–403, 2003. c Springer-Verlag Berlin Heidelberg 2003
GrADSolve – RPC for High Performance Computing on the Grid
395
the load dynamics of the Grid resources. A few RPC systems [10,5] supporting invocation of parallel software are implemented on top of object oriented frameworks like CORBA and JavaRMI and may not be suitable for high performance computing according to a previous study [16]. In this paper, we propose a Grid-based RPC system called GrADSolve1 that enables the users to invoke MPI applications on remote Grid resources from a sequential environment. In addition to providing easy-to-use interfaces for the service providers to upload the parallel applications into the system and for the end users to remotely invoke the parallel applications, GrADSolve performs application-level scheduling and dynamically chooses the resources for the execution of the parallel applications based on the load dynamics of the Grid resources. GrADSolve also uses data distribution information provided by the library writers to partition the users’ data and stage to the different resources used for the application execution. Our experiments show that the data staging mechanisms in GrADSolve helps reduce the data staging times in RPC systems by 20-50%. In addition to the above features, GrADSolve also enables users to store execution traces for a problem run and use the execution traces for subsequent problem runs. Thus, the contributions of our research are: 1. design and development of an RPC system that utilizes standard Grid Computing mechanisms including Globus [8] for invocation of remote parallel applications from a sequential environment. 2. selection of resources for parallel application execution based on load conditions of the resources and application characteristics. 3. communication of data between the user’s address space and the Grid resources based on the data distribution used in the application and 4. maintenance of execution traces for problem runs. The current implementation of GrADSolve is only suitable for invoking remote MPI-based parallel applications from sequential applications and not from parallel applications. Section 2 presents a detailed description of the framework of the GrADSolve system. The support in the GrADSolve system for maintaining execution traces is explained in Section 3. In Section 4, the experiments conducted in GrADSolve are explained and results are presented to demonstrate the usefulness of the data staging mechanisms and execution traces in GrADSolve. Section 5 looks at related efforts in the development of RPC systems. Section 6 presents conclusions and future work.
2
Overview of GrADSolve
Figure 1 illustrates the overview of the GrADSolve system. 1
The system is called GrADSolve since it is derived from the experiences of the GrADS [1] and NetSolve [3] projects.
396
S. Vadhiyar, J. Dongarra, and A. YarKhan
lem
sp
ec
if
as ob tab pr da es L v e M ri ret the X nt lie rom C f 1.
Client
ti ica
on
XML Database
e
Machine 1
ote machines, ges data to rem 4. The client sta ote machines and rem the on lication launches the app al machines tput from the fin retrieves the ou
Machine 2
Machine 3
2. The client matches input data with the problem parameters 3. The client determines the final schedule based on the execution model of the application
Fig. 1. Overview of GrADSolve system
At the core of the GrADSolve system is a XML database. This database maintains four kinds of tables - users, resources, applications and problems. The users, resources and the applications tables contain information about the different users, machines and applications on the Grid system respectively. The problems table maintains information about the individual problem runs. There are three human entities involved in GrADSolve - administrators, library writers and end users. The role of these entities in GrADSolve and the functions performed by the GrADSolve system for these entities are explained below. Administrators. The GrADSolve administrator is responsible for managing the users and resources of the GrADSolve system. The administrator creates entities for different users and resources in the XML database by specifying configuration files that contains information for different users and resources, namely the user account names for different resources, the location of the home directories on different resources in the GrADSolve system, the names of the different machines, their computational capacities, the number of processors in the machines and other machine specifications. Library Writers. The library writer uploads his application into the GrADSolve system by specifying an Interface Definition Language (IDL) file for the application. In the IDL file, the library writer specifies the programming language in which the function is written, the name of the function, the set of input and output arguments, the description of the function, the names of the object files and libraries needed for linking the function with other functions, if the function is sequential or parallel, etc. For each input and output argument, the library writer specifies the name of the argument, if the argument is an input
GrADSolve – RPC for High Performance Computing on the Grid
397
or output argument, the datatype of the argument, the number of elements if the argument is a vector, the number of rows and columns if the argument is a matrix etc. An example of a IDL file written for a ScaLAPACK QR factorization routine is given in Figure 2.
PROBLEM qrwrapper C FUNCTION qrwrapper(IN int N, IN int NB, INOUT double A[N][N], INOUT double B[N][1]) ‘‘a version of qr factorization that works with square matrices.’’ LIBS = ‘‘/home/grads23/GrADSolve/ScaLAPACK/pdgeqrf_instr.o \ /home/grads23/GrADSolve/ScaLAPACK/pdscaex_instrQR.o \ ...’’ TYPE = parallel Fig. 2. An example GrADSolve IDL for a ScaLAPACK QR problem
After the library writer submits the IDL file to the GrADSolve system, GrADSolve translates the IDL file to a XML document. The GrADSolve translation system also generates a wrapper program and compiles the wrapper program with the appropriate libraries and stages the executable file to the remote machines in the GrADSolve system. Also, stored in the XML database for the application is the information regarding the location of the executable files on the remote resources. If the library writer wants to add an performance model for his application, he executes the getperfmodel template utility specifying the name of the application. The utility retrieves the problem description of the application from the XML database and generates a performance model template file. The template file contains the definitions of the performance model routines. The library writer fills in the performance model routines with the appropriate code for specifying if the given set of resources have adequate capacity to solve the problem, the predicted execution cost of the application and the data distribution used in the application. The library writer uploads his performance model by executing the add perfmodel utility which stores the location of the wrapper program and the performance model to the XML database corresponding to the entry for the application. End Users. The end users solve problems over remote GrADSolve resources by writing a client program in C or Fortran. The client program includes an invocation of a routine called gradsolve() passing to the function, the name of the end application and the input and output parameters needed by the end application. The invocation of the gradsolve() routine triggers the execution of the GrADSolve Application Manager. GrADSolve uses Globus Grid Security Infrastructure (GSI) for the authentication and authorization of users. The Application Manager then retrieves the problem description from the XML database
398
S. Vadhiyar, J. Dongarra, and A. YarKhan
and matches the user’s data with the input and output parameters required by the end application. If an performance model exists for the end application, the Application Manager downloads the performance model from the remote location where the library writer had previously stored it. The Application Manager then retrieves the list of machines in the GrADSolve system from the resources table in the XML database, and retrieves resource characteristics of the machines from the Network Weather Service (NWS) [17]. The Application Manager uses the list of resources with resource characteristics, the performance model and scheduling heuristics [19] to determine a final schedule for application execution and stores the status of the problem run and the final schedule in the problems table of the XML database corresponding to the entry for the problem run. The Application Manager then creates working directories on the scheduled remote machines for end application execution and enters the Application Launching phase. The Application Launcher stores the input data to files and stages these files to the corresponding remote machines chosen for application execution. An input data may be associated with data distribution information that was previously uploaded by the library writer. The data distribution information contains the kind of data distribution (e.g., block, block-cyclic, cyclic, user-defined etc.) used for the data. If data distribution information for an input data does not exist, the Application Launcher stages the entire input data to all the machines involved in end application execution. If the information regarding data distribution exists, the Application Launcher stages only the appropriate portions of the input data to the corresponding machines. For example, for data with block distribution, only the 2nd block has to be staged to the 2nd machine used for problem solving. This kind of selective data staging significantly reduces the time needed for the staging of entire data especially if large amount of data is involved. After the staging of input data, the Application Launcher launches the end application on the remote machines chosen for the final schedule using the Globus MPICH-G mechanisms. The end application reads the input data that were previously staged by the Application Launcher, solves the problem and stores the output data to the corresponding files on the machines in the final schedule. When the end application finishes execution, the Application Launcher copies the output data from the remote machines to the user’s address space. The staging in of the output data from the remote locations is a reverse operation of the staging out of the input data to the remote locations. The GrADSolve Application Manager finally returns success state to the user client program.
3
Execution Traces in GrADSolve – Storage, Management, and Usage
One of the unique features in the GrADSolve system is the ability provided to the users to store and use execution traces of problem runs. There are many applications in which the outputs of the problem depend on the exact number and configuration of the machines used for problem solving. Ill-conditioned problems
GrADSolve – RPC for High Performance Computing on the Grid
399
or unstable algorithms can give rise to vast changes in output results due to small changes in input conditions. For these kinds of applications, the user may desire to use the same initial environment for all problem runs. To guarantee reproducibility of numerical results in the above situations, GrADSolve provides capability to the users to store execution traces of problem runs and use the execution traces during subsequent executions of the same problem with the same input data. For storing the execution trace of the current problem run, the user executes his GrADSolve program with a configuration file called input.config that contains a TRACE FLAG variable that is either 0 or 1. During the registration of the problem run with the XML database, the value of the TRACE FLAG variable is stored. After the end application completes its execution and the output data are copied from the remote machines to the user’s address space, the Application Manager removes the remote files containing the input data for the end application if the TRACE FLAG is 0. But if the TRACE FLAG is set to 1, the Application Manager retains the input data in the remote machines. At the end of the problem run, the Application Manager generates an output configuration file that contains a TRACE KEY corresponding to the execution trace. When the user wants to execute the problem with a previously stored execution trace, he executes his client program specifying the TRACE KEY variable in the input.config file. The TRACE KEY variable is set with the key that corresponds to the execution trace. During the Schedule Generation phase, the Application Manager, instead of generating a schedule for the execution of the end application, retrieves the schedule used for the previous problem run corresponding to the TRACE KEY, from the problems table in the XML database . The Application Manager then checks if the capacities of the resources in the schedule at the time of trace generation are comparable to the current capacities of the resources. If the capacities are comparable, the Application Manager proceeds to the rest of the phases of its execution. During the Application Launching phase, the Application Manager, instead of staging the input data to remote working directories, copies the input data and the data distribution information, used in the previous problem run corresponding to the TRACE KEY, to the remote working directories. Thus GrADSolve guarantees the use of the same execution environment used in the previous problem run for the current problem run, and hence guarantees reproducibility of numerical results.
4
Experiments and Results
The GrADS testbed consists of about 40 machines from University of Tennessee (UT), University of Illinois, Urbana-Champaign (UIUC) and University of California, San Diego (UCSD). For the sake of clarity, our experimental testbed consists of 4 machines: – a 933 MHz Pentium III machine with 512 MBytes of memory located in UT, – a 450 MHz Pentium II machine with 256 MBytes of memory located in UIUC and
400
S. Vadhiyar, J. Dongarra, and A. YarKhan
– 2 450 MHz Pentium III machines with 256 MBytes of memory located in UCSD connected to each other by 100 Mb switched Ethernet. Machines from different locations are connected by Internet. In the experiments, GrADSolve was used to remotely invoke ScaLAPACK QR factorization. Since some of the unique features of GrADSolve include the data distribution mechanisms and the usage of execution traces, the experiments focus only on the times for staging data and not on the communication and total execution times. Block cyclic distribution was used for the matrix A. GrADSolve was operated in 3 modes. In the first mode, the performance model did not contain information about the data distribution used in the ScaLAPACK driver. In this case, GrADSolve transported the entire data to each of the locations used for the execution of the end application. This mode of operation is practiced in RPC systems that do not support the information regarding data distribution. In the second mode, the performance model contained information about the data distribution used in the end application. In this case, GrADSolve transported only the appropriate portions of the data to the locations used for the execution of end application. In the third mode, GrADSolve was used with an execution trace corresponding to a previous run of the same problem. In this case, data is not staged from the user’s address space to the remote machines, but temporary copies of the input data used in the previous run are made for the current problem run.
Data Staging and GrADSolve Overhead 1400
Full data staging Data staging with distribution Data staging with execution traces Overhead with full data staging Overhead with distribution Overhead with execution traces
1200
Time [secs.]
1000 800 600 400 200 0 1000
2000
3000
4000 5000 Matrix Size
6000
7000
8000
Fig. 3. Data staging and other GrADSolve overhead
Figure 3 shows the times taken for data staging and other GrADSolve overhead for different matrix sizes and for the three modes of GrADSolve operation. The machines that were chosen by the GrADSolve application-level scheduler for the execution of end application for different matrix sizes are shown in Table
GrADSolve – RPC for High Performance Computing on the Grid
401
1. The decisions regarding the selection of machines for problem execution were automatically made by the GrADSolve system taking into account the size of the problems and the resource characteristics at the time of the experiments. Table 1. Machines chosen for application execution Matrix size Machines 1000 1 UT machine 2000 1 UT machine 3000 1 UT machine 4000 1 UT machine 5000 1 UT, 1 UIUC machines 6000 1 UIUC, 1 UCSD machines 7000 1 UIUC, 1 UCSD machines 8000 1 UT, 1 UIUC, 2 UCSD machines
Comparing the first two modes in Figure 3, we find that for smaller problem sizes, the times taken for data staging in both the modes are the same. This is because only one machine was used for problem execution and the same amount of data are staged in both the modes when only one machine is involved for problem execution. For larger problem sizes, the times for data staging using the data distribution is less than 20-55% of the times taken for staging the entire data to remote resources. Thus the use of data distribution information in GrADSolve can give significant performance benefits when compared to staging the entire data that is practiced in some of the RPC systems. Data staging in the third mode is basically the time taken for creating temporary copies of data used in the previous problem runs in remote resources. We find this time to be negligible when compared to the first two modes. Thus execution traces can be used as caching mechanisms to use the previously staged data for problem solving. The GrADSolve overheads for all the three modes are found to be the same. This is because of the small number of machines used in the experiments. For experiments when large number of machines are used, we predict that the overheads will be higher in the first two modes than in the third mode. This is because in the first two modes, the application-level scheduling will explore large number of candidate schedules to determine the machines used for the end application while in the third mode, a previous application-level schedule will be retrieved from the database and used.
5
Related Work
Few RPC systems contain mechanisms for the parallel execution of remote software. The work by Maassen et. al [10] extends Java RMI for efficient communications in solving high performance computing problems. The framework requires the end user’s programs to be parallel programs. NetSolve [3] and Ninf [12] support task parallelism by the asynchronous execution of number of remote
402
S. Vadhiyar, J. Dongarra, and A. YarKhan
sequential applications. OmniRPC [13] is an extension of Ninf and supports asynchronous RPC calls made from OpenMP programs. But similar to the approaches in NetSolve and Ninf, OmniRPC supports only master-worker models of parallelism. NetSolve, and Ninf also supports remote invocation of MPI applications, but the amount of parallelism and the locations of the resources to be used for the execution are fixed at the time when the applications are uploaded to the systems and hence are not adaptive to dynamic loads in the Grid environments. The efforts that are very closely related to GrADSolve are PaCO [11] and PaCO++ [6,5] from the PARIS project in France. The PaCO systems are implemented within the CORBA [4] framework to encapsulate MPI applications in RPC systems. The data distribution and redistribution mechanisms in PaCO are much more robust than in GrADSolve and support invocation of remote parallel applications either from sequential or parallel client programs. The PaCO projects do not support dynamic selection of resources for application execution as in GrADSolve. Also, GrADSolve supports Grid related security models by employing Globus mechanisms. And finally, GrADSolve is unique in maintaining execution traces that can help bypass the resource selection and data staging phases.
6
Conclusions and Future Work
In this paper, an RPC system for efficient execution of remote parallel software was discussed. The efficiency is achieved by dynamically choosing the machines used for parallel execution and staging the data to remote machines based on data distribution information. The GrADSolve RPC system also supports maintaining and utilizing execution traces for problem solving. Our experiments showed that the GrADSolve system is able to adapt to various problem sizes and the resource characteristics and yielded significant performance benefits with its data staging and execution trace mechanisms. Interfaces for the library writers for expressing more capabilities of the end application are currently being designed. These capabilities include the ability of the application to be preempted and continued later with different processor configuration. These capabilities will allow GrADSolve to adapt to changing Grid scenarios. Remote execution of non-MPI parallel programs, applications with different modes of parallelism and irregular applications are also being considered.
References 1. F. Berman, A. Chien, K. Cooper, J. Dongarra, I. Foster, D. Gannon, L. Johnsson, K. Kennedy, C. Kesselman, J. Mellor-Crummey, D. Reed, L. Torczon, and R. Wolski. The GrADS Project: Software Support for High-Level Grid Application Development. International Journal of High Performance Applications and Supercomputing, 15(4):327–344, Winter 2001.
GrADSolve – RPC for High Performance Computing on the Grid
403
2. A.D. Birrell and B.J. Nelson. Implementing Remote Procedure Calls. ACM Transactions on Computer Systems, 2(1):39–59, February 1984. 3. H. Casanova and J. Dongarra. NetSolve: A Network Server for Solving Computational Science Problems. The International Journal of Supercomputer Applications and High Performance Computing, 11(3):212–223, Fall 1997. 4. CORBA http://www.corba.org. 5. A. Denis, C. P´erez, and T. Priol. Achieving Portable and Efficient Parallel CORBA Objects. Concurrency and Computation: Practice and Experience, 2002. 6. A. Denis, C. P´erez, and T. Priol. Portable Parallel CORBA Objects: an Approach to Combine Parallel and Distributed Programming for Grid Computing. In Proc. of the 7th International Euro-Par’01 Conference (EuroPar’01), pages 835–844. Springer, August 2001. 7. I. Foster and C. Kesselman eds. The Grid: Blueprint for a New Computing Infrastructure. Morgan Kaufmann, ISBN 1-55860-475-8, 1999. 8. I. Foster and C. Kesselman. Globus: A Metacomputing Infrastructure Toolkit. Intl J. Supercomputer Applications, 11(2):115–128, 1997. 9. Java Remote Method Invocation (Java RMI) java.sun.com/products/jdk/rmi. 10. J. Maassen, R. van Nieuwpoort, R. Veldema, H. Bal, T. Kielmann, C. Jacobs, and R. Hofman. Efficient Java RMI for Parallel Programming. ACM Transactions on Programming Languages and Systems (TOPLAS), 23(6):747–775, November 2001. 11. C. Ren´e and T. Priol. MPI Code Encapsulating using Parallel CORBA Object. Cluster Computing, 3(4):255–263, 2000. 12. H. Nakada M. Sato and S. Sekiguchi. Design and Implementations of Ninf: towards a Global Computing Infrastructure. Future Generation Computing Systems, Metascomputing Issue, 15(5-6):649–658, 1999. 13. M. Sato, M. Hirano, Y. Tanaka, and S. Sekiguchi. OmniRPC: A Grid RPC Facility for Cluster and Global Computing in OpenMP. In In Workshop on OpenMP Applications and Tools (WOMPAT2001), July 2001. 14. K. Seymour, H. Nakada, S. Matsuoka, J. Dongarra, C. Lee, and H. Casanova. Overview of GridRPC: A Remote Procedure Call API for Grid Computing. In M. Parashar, editor, Lecture notes in computer science 2536 Grid Computing GRID 2002, volume Third International Workshop, pages 274–278, Baltimore, MD, USA, November 2002. Springer Verlag. 15. Simple Object Access Protocol (SOAP) http://www.w3.org/TR/SOAP. 16. T. Suzumura, T. Nakagawa, S. Matsuoka, H. Nakada, and S. Sekiguchi. Are Global Computing Systems Useful? - Comparison of Client-Server Global Computing Systems Ninf, Netsolve versus CORBA. In In Proceedings of the 14th International Parallel and Distributed Processing Symposium, IPDPS ’00, pages 547–559, May 2000. 17. R. Wolski, N. Spring, and J. Hayes. The Network Weather Service: A Distributed Resource Performance Forecasting Service for Metacomputing. Journal of Future Generation Computing Systems, 15(5-6):757–768, October 1999. 18. XML-RPC http://www.xmlrpc.com. 19. A. YarKhan and J. Dongarra. Experiments with Scheduling Using Simulated Annealing in a Grid Environment. In M. Parashar, editor, Lecture notes in computer science 2536 Grid Computing – GRID 2002, Third International Workshop, pages 232–242, Baltimore, MD, USA, November 2002. Springer Verlag.
Resource and Job Monitoring in the Grid Zolt´an Balaton and G´abor Gomb´as MTA SZTAKI Computer and Automation Research Institute P.O. Box 63., H-1518 Hungary {balaton, gombasg}@sztaki.hu
Abstract. In a complex system like the grid monitoring is essential for understanding its operation, debugging, failure detection and for performance optimisation. In this paper a flexible monitoring architecture is introduced that provides advanced functions like actuators and guaranteed data delivery. The application of the monitoring system for grid job and resource monitoring, and the integration of the monitoring system with other grid services is also described.
1
Introduction
The Grid [8] is a vast array of distributed resources. Monitoring is essential for understanding the operation of the system, debugging, failure detection and for performance optimisation. To achieve this, data about the grid must be gathered and processed to reveal important information. Then, according to the results, the system may need to be controlled. It is the task of the monitoring system to provide facilities for this. Traditionally monitoring is considered as part of grid information systems such as, Globus MDS [5] and R-GMA [7]. This approach however has difficulties in supporting all monitoring scenarios, e.g. application monitoring. To overcome these limitations a number of application monitoring solutions have been proposed for grids. Systems such as NetLogger [10], OCM-G [4] and Autopilot [12] all focus on application monitoring independent of the information system. We proposed a division of the information and monitoring systems in [2] based on the properties of information they must handle. The monitoring system must be able to provide information about the current state of various grid entities, such as grid resources and running jobs, as well as to provide notifications when certain events (e.g. system failures, performance problems) occur. In the following sections a flexible and efficient monitoring system designed and implemented as part of the GridLab project [9] is introduced and its application for resource and job monitoring and connection to other grid services is presented.
2 The Monitoring System Architecture Grids are very large-scale and diverse systems thus, gathering all information in a central location is not feasible. Therefore, centralised solutions are not adequate, only distributed systems can be considered. Monitoring also has to be specific, i.e. only data that is actually needed has to be collected. It is also important that data should only be sent where it is needed and only when requested. The Grid Monitoring Architecture (GMA, [13]) proposed by the Global Grid Forum (GGF) satisfies these requirements. The architecture of H. Kosch, L. B¨osz¨orm´enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 404–411, 2003. c Springer-Verlag Berlin Heidelberg 2003
Resource and Job Monitoring in the Grid
405
the monitoring system is therefore based on the GMA. The input of the monitoring system consists of measurements generated by sensors. Sensors are controlled by producers that can transfer measurements to consumers when requested. Sensors implement the measurement of one or more measurable quantities that are represented in the monitoring system as metrics defined by a unique name, a list of formal parameters and a data type. The metric name identifies the metric definition (e.g. host.cpu.user). Many metrics can be measured at different places simultaneously. E.g., CPU utilisation can be measured on several hosts or grid resources. The parameters in the metric definition help to distinguish between these different metric instances. The entity to be monitored is determined by providing actual values for the formal parameters. E.g., the metric “CPU time used by a process” must have process ID as a formal parameter. A metric instance is then formed by supplying a PID. A measurement corresponding to a metric instance is called metric value. Metric values contain a timestamp and the measured data according to the data type of the metric definition. The data type is the definition of the structure used for representing measurement data. The monitoring system supports basic types (such as, 32 and 64 bit, signed and unsigned integers, double precision floating point numbers), strings and arbitrary binary data, which is uninterpreted by the monitoring system. The monitoring system also supports two complex types, arrays of identical types with either fixed or variable length, and a record type, which is a composite of other types with a predefined layout. These types allow the definition of metrics with arbitrarily complex data types however, monitoring data can typically be represented with simple types. Metrics can either be event-like, i.e. an external event is needed to produce a measurement (a typical example is a notification of a job state change), or continuously measurable, i.e. a measurement is possible whenever a consumer requests it (such as the CPU temperature in a host). Our monitoring system supports both measurement types. Continuously measurable metrics can be made event-like by requesting automatic periodic measurements. In addition to the components in the GMA, actuators (similar to actuators inAutopilot [12] or manipulation services in OMIS [11]) are also part of the system. Actuators are analogous to sensors but instead of taking measurements of metrics they implement controls that represent interactions with either the monitored entities or the monitoring system itself. Just like metrics a control is defined by a unique name, a list of formal parameters and a data type. In contrast to metrics that can generate multiple results however, controls are always single-shot, i.e. every invocation of a control produces exactly one result. The functional difference between metrics and controls is that metrics only provide data while controls do not provide data except for a status report but they influence the state or behaviour of the monitoring system or the monitored entity. The use of metrics and controls makes the monitoring system very general and flexible. Producers are able to handle diverse sensors and actuators without detailed knowledge of their inner workings. At the same time, sensors and actuators are free to exploit all the implementation possibilities. Sensors do not have to take all measurements themselves, they could also contact external processes to get data. For example, a sensor can query the local accounting system if available or it is also possible to implement persistent sensors (such as network monitors) that are running independently of the
406
Z. Balaton and G. Gomb´as
monitoring system and gather statistics over time. The producer can manage (start, stop and control) sensors, initiate measurements and invoke controls on a user’s request. The flexibility of metrics and controls is also exploited within the monitoring system. There are some metrics built in the producer that provide information about the monitoring system, such as producer capabilities and metric definitions. Influencing the monitoring system is also possible via controls that act on the monitoring system itself. Apart from very simple built-in sensors and actuators most sensors and actuators are loadable modules that are dynamically linked in the producer depending on configuration. This modularity makes the monitoring system very flexible and easily extensible. The GMA proposal defines two methods of information retrieval: query and subscribe. Our monitoring system supports further producer–consumer interaction functions. Consumers can request buffering of measurements in the producer and multiple logical connections between a producer and a consumer (called channels) are also available. Channels can either be consumer-initiated or producer-initiated, but producerinitiated channels must be explicitly requested previously by a consumer. Consumerinitiated channels are used mainly for interactive monitoring or control execution while producer-initiated channels can be used for event reporting and data archiving in a storage service. The two possible channel directions can also be used for getting through firewalls which block communication in one direction. Permanently storing data is outside of the responsibilities of the monitoring system thus, if a consumer needs to preserve monitoring data it must either save the data itself or supply a contact point for a storage service and appropriate credentials for authentication. Generally if the consumer fails to process some data due to a software or hardware failure the information may be lost. There are cases however (e.g. critical event notifications) when this behaviour is not acceptable. For these cases the monitoring system supports guaranteed delivery of measurements. Consumers can enable guaranteed delivery for a metric by executing a control. After guaranteed delivery is enabled, the producer awaits an acknowledgement of receiving each metric value from the consumer before deleting it from its buffer. Handling of guaranteed delivery is expensive as it consumes resources of the producer. Therefore, the producer might impose resource limits on the number of transactions being active at any given time.
3
Resource Monitoring
The monitoring system components used for resource monitoring is shown in Figure 1. drawn with solid lines. The figure depicts a grid resource consisting of three nodes. A Local Monitor (LM) service is running on each node and collects information from processes (P) running on the node as well as the node itself. The collected information is sent to a Main Monitor (MM) service. The MM is a central access point for local users (i.e. site administrators and non-grid users). Grid users can access information via the Monitoring Service (MS) which is also a client of the MM. In large sized grid resources there may be more than one MM to balance network load. The modularity of the monitoring system also allows that on grid resources where an MM is not needed (e.g. on a supercomputer) it can be omitted and the MS can talk directly to LMs. As the Grid itself consists of several layers of services the monitoring system also follows this layout. The LM – MM – MS triplet demonstrates how a multi-level moni-
Resource and Job Monitoring in the Grid
407
job state
control
Submitter
GJID Grid Resource Jobmanager
MS
LJID LRMS
LM PID P
Grid User Storage
MM
LM
LM P
P
P
Fig. 1. Resource and job monitoring
toring system is built. The compound producer–consumer entities described in the GMA proposal is exploited here: each level acts as a consumer for the lower level and as a producer for the higher level. This setup has several advantages. Complex compound metrics are easily introduced by sensors at intermediate levels which get raw data from lower levels and provide processed data for higher levels. Transparent access to preprocessed data or data from multiple sources can be supported in the same way. In cases when there is no direct network connection (e.g., because of a firewall) between the consumer and a target host, an intermediary producer can be installed on a host that has connectivity to both sides. This producer can act as a proxy between the consumer and the target host. With proper authentication and authorization policies at this proxy, this setup is more secure and more manageable than opening the firewall. The above described possibilities are exploited in the MS. The MS acts as a proxy between the grid resource and grid users. This is the component where grid security rules are enforced and mapped to local rules. As the Grid is a heterogenous environment it is very likely that different grid resources (or even different nodes of the same resource) provide different kinds of information. In order to make it possible for data from different sources to be interpreted and analysed in a uniform way, the raw data must be converted to a well-defined form. Such conversion is also done in the MS which takes resource specific metrics from the MM and converts them to grid metrics that are independent of the physical characteristics of the resource.
408
4
Z. Balaton and G. Gomb´as
Job Monitoring
Monitoring of grid jobs is a complicated task that requires interoperation between the monitoring system and other grid services. Figure 1. depicts the situation where an application started as a grid job is running on a grid resource. The running application consists of processes (labelled P in the figure) running on hosts constituting the grid resource. Processes are identified locally by the operating system by process identifiers (PIDs). The local resource management system (LRMS) controls jobs running on hosts belonging to a grid resource. It allocates hosts to jobs, starts and stops jobs on user request and possibly restarts jobs in case of an error. It may also checkpoint and migrate jobs between hosts of the resource which can be considered as a special case of job startup. The LRMS identifies the job it manages by a local job identifier (LJID). To monitor a job the monitoring system has to know the relation between LJIDs and PIDs. This is required because some metrics (e.g. CPU usage and low level performance metrics provided by hardware performance counters) are only available for processes, while other metrics (such as high level job status) can only be get from the LRMS by the LJID. The LRMS is the only component that has all the information about which processes belong to a job, thus it should have an interface for the monitoring system to provide the mapping between PIDs and LJIDs to support monitoring. In practice however, current LRMS implementations usually do not provide such an interface thus another way to get the mapping between PIDs and LJIDs is necessary. The most simple solution would be to require processes to register and identify themselves to the monitoring system by calling a function at startup. This would however require modification to the application and assumes that the application processes have access to their LJID which may not be the case for some LRMS implementations. This solution would also rely on the application to honestly register all its processes thus, ill behaving applications could prevent correct monitoring. Problems may arise for cooperating applications as well. For example, if the LRMS transparently checkpoints and migrates application processes they may be lost for the monitoring system until they re-register themselves. Another possible solution is to require the LRMS to set the name of the processes it starts in such a way that the LJID of the job the process belongs to can be inferred from it. This way, the monitoring system could identify processes being part of a grid job by looking at their process name. However, processes are able to change their process name thus, it is possible for applications to hinder monitoring. This solution also requires modification to the LRMS which may not be possible for some implementations. Moreover, both solutions described above are insecure because they rely on an untrusted component (the application) to correctly identify itself to the monitoring system and not to interfere with the mechanisms used for identification. Thus, in situations where precision of monitoring is important (e.g. when monitoring data may be used for accounting purposes) these solutions are inadequate. The jobmanager in Figure 1. represents the grid service, which allows grid users to start jobs on a grid resource (an example for the jobmanager is the Globus GRAM [6]). The user hands to the jobmanager a job manifest, a document that contains both the description of the job and the specification of the requested local resources. After successful authentication and authorization, the jobmanager translates the job manifest
Resource and Job Monitoring in the Grid
409
into a form understood by the local resource management system and starts the job under a local user account (identified by a user ID). The easiest and most secure way to identify processes belonging to a job is to start each job under a different user account. In this way, processes of a job can be identified by the user ID. For this, the jobmanager must ensure that each job is started under a user account that is distinct from accounts used by other currently running grid jobs. A free user account can be taken from a pool of accounts that are reserved for running grid jobs. This solution does not have the problems of the other two mechanisms described above. Moreover, this method has other advantages as well: – It does not require any modifications to the job. – The only support required from the LRMS is to start the processes of the job with the user account under which the job was submitted, that is supported by all common implementations. – Operating system mechanisms ensure that grid jobs cannot interfere with each other either accidentally or intentionally. File system permissions guarantee that different jobs cannot overwrite each others’ files. Different process ownership prevents sending of unexpected signals to other jobs. – Operating system facilities can be used for authorisation and enforcing local policies. Resource usage limits can be set and enforced for each job. File system permissions can be used to implement shared use of cached resources (e.g. if multiple jobs process the same read-only data file, it can be shared between the jobs without having to be downloaded multiple times but still preventing accidental overwrites by any of the jobs). – Highly accurate accounting is possible if the operating system supports process accounting. If the same grid user runs several jobs on the same (multiprocessor) machine under a single user ID, separating accounting records for different jobs can be difficult. The system-level accounting records are clearly separated however, if each job is mapped to a different local user ID. – Users cannot interfere with or circumvent monitoring. A process created by a grid job (e.g. by forking) cannot hide from the monitoring system nor can it pretend to belong to a different job and/or user. – Local account management of the resource can be simplified as there is no need to create local accounts for each grid user. Instead a pool of user accounts big enough for the maximum number of simultaneously running jobs is sufficient. On platforms having dynamic name services (NSS for Linux, Solaris and HP-UX, Loadable Security Modules for AIX, etc.) usernames can be generated on the fly. – Left-over processes can be easily identified and killed even if the same grid user has several other jobs running. Left-over files created by a job can also be identified by owner and destroyed when the corresponding job has finished. The jobmanager also allows the user to control the execution of the job. To reference jobs started by the jobmanager yet another identifier the grid job ID (GJID) is used. The jobmanager maintains the mapping between GJIDs and LJIDs and must provide an interface for the monitoring system to get this information together with the user ID that is assigned to this job. The multi-level setup of the monitoring system is useful for
410
Z. Balaton and G. Gomb´as
making this mapping transparent. When a grid user connects to the MS and requests monitoring of a grid job by a GJID, the MS can convert the GJID to a LJID and pass it to the MM. The MM then converts the local job ID to a user ID and passes it to the appropriate LMs that take the actual measurements. To support jobs running on or migrating between multiple grid resources (such as [1]) another grid service labelled Submitter in Figure 1. is required that keeps track on which sites processes belonging to the job are actually running. The Submitter represents a grid entity which starts a grid job and keeps track of the job status. The Submitter always gets notifications about changes in the state of the job and thus it always knows which grid resources the application is running on currently. The user can use this information to find the monitoring service contact points at the appropriate resources in order to control the monitoring of the job and get measurements. The Submitter can also act as a single concentrator element for jobs that run on multiple resources at the same time. This way information about the job can be requested at one well-known point even if the job is constantly moving from one resource to another. This solution provides a distributed registry for jobs running on the Grid.
5
Conclusion
In this paper we presented a flexible and efficient monitoring system designed and implemented as part of the GridLab project. The architecture of the monitoring system is based on the GMA and exploits its distributed design, compound producer–consumer entities and generality. In addition to the features described in the GMA document our monitoring system also supports actuators and additional producer–consumer interaction functions such as, buffering of measurements in the producer, multiple channels and guaranteed delivery of critical data. The application of the monitoring system for resource and job monitoring was also presented, and the interoperation with and support necessary from other grid services was discussed. We proposed a simple and secure solution for keeping track of processes and different identifiers belonging to a job that simplifies both the monitoring system and local management of a grid resource. An implementation of the monitoring system is available and being used in the GridLab project. An alpha version with the basic features implemented was released to the GridLab community in August 2002, and the first official prototype is released in January 2003 for the general public. The prototype contains the implementation of the producer and consumer APIs and implementation of the LM and the MM. It also contains sensors providing host monitoring, support for application monitoring using the GRM [3] instrumentation library, and several test consumers. Sensors, actuators and even the protocol encoding are implemented as loadable modules. This modular design allows flexibility and easy extensibility. To test the performance of the monitoring system implementation we measured the time required to send GRM events from an instrumented application via the GRM sensor, the LM and the MM to a consumer. These tests showed that the monitoring system is capable to handle more than 13000 GRM events per second. The time needed to transfer data was a linear function of the number of events. However, during development so far
Resource and Job Monitoring in the Grid
411
the top priority was correctness and completeness and not performance, thus there are several areas for further improvements. Development is continued, actuators, guaranteed delivery and security functions are being implemented during 2003. Acknowledgements. The work described in this paper has been sponsored by the European Commission under contract number IST-2001-32133, the Hungarian Scientific Research Fund (OTKA) under grant number T032226 and the Research and Development Division of the Hungarian Ministry of Education under contract number OMFB01549/2001.
References 1. G. Allen, D. Angulo, I. Foster, G. Lanfermann, C. Liu, T. Radke, E. Seidel, J. Shalf: The Cactus Worm: Experiments with Dynamic Resource Discovery and Allocation in a Grid Environment. International Journal of High Performance Computing Applications, Volume 15, Number 4, 2001. 2. Z. Balaton, G. Gomb´as, Zs. N´emeth: Information System Architecture for Brokering in Large Scale Grids. Proc. 4th Austrian-Hungarian Workshop on Distributed and Parallel Systems, Linz, Austria, September/October 2002. 3. Z. Balaton, P. Kacsuk, N. Podhorszki, F. Vajda: From Cluster monitoring to Grid Monitoring Based on GRM. Proc. Euro-Par 2001 International Conference on Parallel and Distributed Computing, Manchester, United Kingdom, August 2001. 4. B. Bali´s, M. Bubak, W. Funika, T. Szepieniec, R. Wism¨uller: An Infrastructure for Grid Application Monitoring. Proc. 9th European PVM/MPI Users’ Group Meeting, Linz, Austria, September/October 2002. 5. K. Czajkowski, S. Fitzgerald, I. Foster, C. Kesselman: Grid Information Services for Distributed Resource Sharing. Proc. 10th IEEE International Symposium on High Performance Distributed Computing, San Francisco, California, August 2001. 6. K. Czajkowski, I. Foster, N. Karonis, C. Kesselman, S. Martin, W. Smith, S. Tuecke: A Resource Management Architecture for Metacomputing Systems. Proc. IPPS/SPDP ’98 Workshop on Job Scheduling Strategies for Parallel Processing, 1998. 7. S. Fisher et al. R-GMA: A Relational Grid Information and Monitoring System 2nd Cracow Grid Workshop, Cracow, Poland, 2003. 8. I. Foster and C. Kesselman, editors: The Grid: Blueprint for a Future Computing Infrastructure, 1999. 9. The GridLab Project 10. D. Gunter, B. Tierney, K. Jackson, J. Lee, M. Stoufer: Dynamic Monitoring of HighPerformance Distributed Applications. Proc. 11th IEEE International Symposium on High Performance Distributed Computing, Edinburgh, Scotland, July 2002. 11. T. Ludwig, R. Wism¨uller: OMIS 2.0 – A Universal Interface for Monitoring Systems. Proc. 4th European PVM/MPI Users’ Group Meeting, Cracow, Poland, November 1997. 12. R. Ribler, J. Vetter, H. Simitci, D. Reed: Autopilot: Adaptive Control of Distributed Applications. Proc. 7th IEEE Symposium on High Performance Distributed Computing, Chicago, Illinois, July 1998. 13. B. Tierney, R. Aydt, D. Gunter, W. Smith, M. Swany, V. Taylor, R. Wolski: A Grid Monitoring Architecture. GGF Technical Report GFD-I.7, January 2002.
1
Delivering Data Management for Engineers on the Grid Jasmin Wason, Marc Molinari, Zhuoan Jiao, and Simon J. Cox School of Engineering Sciences, University of Southampton, UK {j.l.wason, m.molinari, z.jiao, sjc}@soton.ac.uk
Abstract. We describe the design and implementation of a database toolkit for engineers, which has been incorporated into the Matlab environment, to help manage the large amount of data created in distributed applications. The toolkit is built using Grid and Web services technologies, and exchanges XML metadata between heterogeneous Web services, databases and clients using open standards. We show an application exemplar of how this toolkit may be used in a grid-enabled Computational Electromagnetics design search.
1 Introduction Engineering design search and optimization (EDSO) is the process whereby engineering modelling and analysis are exploited to yield improved designs. This may involve lengthy and repetitive calculations requiring access to significant computational resources. This requirement makes the problem domain well-suited to the applications of Grid technology which enable large-scale resource sharing and coordinated problem solving within a virtual organisation (VO) [1]. Grid technology provides scalable, secure, high-performance mechanisms for utilizing remote resources, such as computing power, data and software applications over the Internet. While compute power may be easily linked into applications using grid computing middleware, there has been less focus on database integration, and even less still on providing it in an environment familiar to engineers. Traditionally, data in many scientific and engineering disciplines have been organized in application-specific file structures, and a great deal of data accessed within current Grid environments still exists in this form [2]. When there are a large number of files it becomes difficult to find, compare and share the data. If database technologies are used to store additional information (metadata) describing the files, they can be located more easily using metadata queries. The Storage Resource Broker (SRB) [3] uses its Metadata Catalog (MCAT) for dataset access based on attributes rather than names or physical locations. However, MCAT has limited support for application specific metadata which is often essential in assisting engineers to locate data specific to their problems. In Geodise [4] our goal is to develop sophisticated but easy-to-use toolkits to help engineers make use of grid resources from environments they use daily, such as Matlab [5] and Jython [6]. These consist of the Geodise computational toolkit [7], XML toolbox [8] and a database toolkit, the focus of this paper, in which we adopt an open standards and service oriented approach to leverage existing database technologies and make them accessible to engineers and suitable for their problems. 1
The full paper is available at [4]. This work is supported by the Geodise e-Science pilot project (UK EPSRC GR/R67705/01) and the DTI-funded GEM project.
H. Kosch, L. Böszörményi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 412–416, 2003. © Springer-Verlag Berlin Heidelberg 2003
Delivering Data Management for Engineers on the Grid
413
2 Architecture A major aim of the Geodise data management architecture is to bring together flexible, modular components for managing data on the Grid which can be utilized by higher level applications. Another objective is to provide a simple, transparent way for engineering users to archive files in a repository along with metadata. Various technical and application specific metadata about files, their locations and access rights are stored in databases. Files are stored in file systems and transported using the Globus Toolkit [9] which provides middleware for building computational grids and their applications. We use the platform independent Java CoG kit [10] to utilize the Grid Security Infrastructure (GSI) [11] for authentication, and GridFTP [12] for secure file transfer. As shown in Fig. 1, client side tools initiate file transfer and call Web services [13] for metadata storage, query, authorisation and file location. Client
Grid Geodise Database Toolkit Matlab Functions
Globus Server Refers to
GridFTP
Java clients
.NET
Location Service
Location Database
Authorisation Service
Authorisation Database
CoG Apache SOAP
SOAP SOAP
Browser
Java
Metadata Archive & Query Services
Metadata Database
Fig. 1. A high level set of scripting functions sits on top of a client side Java API to provide an interface to data management Web service functionality and secure file transfer.
Access to databases is provided through Web services, which may be invoked using the Simple Object Access Protocol (SOAP) [14], to transfer data between programs on heterogeneous platforms. For example, our Java client code running on Linux or Windows can communicate with .NET Web services on a Windows server and Java Web services on a Linux server. A unique handle is all that is required to locate and retrieve a file, as the file location service keeps a record of where it is stored. The metadata archive service allows the storage of additional descriptive information detailing a combination of technical characteristics (e.g. size, format) and user defined application domain specific metadata. The metadata query service provides a facility for engineers to find the data required without the need to remember the file names and handles. We use relational tables for structured data and XML for more flexible storage of complex, deeply nested engineering specific metadata. We require a set of services that allow us to access and interrogate both types of data storage in a standard way. We currently provide APIs to specific databases with tailored Web services and will use these on top of implementations compliant with proposed standards from the Data Access and Integration Services Working Group [15] of the GGF [16]. The authorisation service uses a database of registered users, containing data access permissions mapping between VO user IDs and authenticated Globus
414
J. Wason et al.
Distinguished Names, globally unique identifiers representing individuals. Query results are filtered and only metadata about files the user has access to are returned.
3 Problem Solving Environment and Application Example The basic tasks (Fig. 2) for an engineer to undertake to manage and share their data are to (A) generate the data using standard engineering tools, (B) store it in the repository, (C) search for data of interest, and (D) retrieve results to the local file system. The wrapping of the core services enabling these tasks is straightforward as much of the logic of the client side components is written in Java, which can be directly exposed to Matlab or other high-level scripting environments, such as Jython [6]. Matlab contains a large number of toolboxes tailored to scientists and engineers, and makes it easy for them to quickly generate, analyse and visualize their data. We have implemented a range of Matlab functions on top of our core database services so that they can be incorporated programmatically into the user’s scripts in a way consistent with the behaviour and syntax of the Matlab environment. Matlab
(A) Generate file (B) Archive
File archive
local file path
structure
Data file XML
filehandle
(C) Query
structure filehandle
(D) Retrieve
Metadata database
query string
XML XML XML
filehandle local file path
Data file
Fig. 2. Data flow of files and metadata: (A) file generation, (B) archive of file and user metadata, (C) querying of metadata, and (D) file retrieval.
We have applied the Geodise database toolkit in a real world example of design search in Computational Electromagnetics (CEM). The GEM project [16] is developing software for industrial use to improve the design of optical components for next generation integrated photonic devices. A specific device is the photonic crystal with certain light transmission and reflection properties which depend on the size and distribution of structures drilled into a slab of dielectric material. To investigate the characteristic photonic bandgap (PBG) of the crystal, an engineer samples a range of parameters (e.g. radius radius and spacing d of the holes) with each sample point giving rise to a different value of the objective function (the bandgap). Initially a large number of designs are explored which yield many solutions and large amounts of data. All of these solutions, whether good or poor, may yield valuable information for future simulations or devices and need to be preserved. Fig. 3 shows two scripts, one to create data and archive it, the other for query and retrieval of data. The first stage (a) involves the user creating a certificate proxy so they can be authenticated and then generating data files and defining custom metadata about the geometry parameters and resulting bandgap as a Matlab structure, m. Then
Delivering Data Management for Engineers on the Grid
415
the spectrum results file is stored using the gd_archive function (b), along with the metadata structure, which is converted into XML by our XML Toolbox for Matlab [8]. gd_archive then transports the file to a server and the metadata to the metadata service for storage in an XML database. a
b c d
gd_createproxy; m.model = ’pgb_design’; m.param.d = […]; m.param.radius = […]; ... compute_pgb(m.param, infile, outfile); m.result.bandgap = postprocs_pbg(outfile); gd_archive(outfile, m); ... Q = gd_query('model = pbg_design & result.bandgap < 99.7'); Q: 4x1 struct array with fields standard, model, param, result gd_retrieve({Q.standard.fileID}, '/home/Eng007/pbg_files/' ); visualise_pbg_landscape ('/home/Eng007/pbg_files/*' );
Fig. 3. Example scripts to generate, archive, query, retrieve and post-process data.
This script is run a number of times with different parameters. After the computations have finished and the design results are available in the database, the Engineer can check the results with a simple query (c). The query can be formed using a combination of named metadata variables and comparison operators or alternatively through a graphical interface. Q is a vector of structures containing the metadata of all PBG designs with a bandgap less than 99.7. In this case, four designs match the query. For further investigation or visualization, the Engineer can retrieve files associated with the above four designs to the local file system (d). Fig. 4 shows typical data we can obtain for the various design parameters. The simulation results form an objective function landscape of the photonic bandgap from which a full design search and optimization may be performed. The storage of the results in a database as well as the transfer of files to a file store on the Grid additionally allows data re-use by engineers.
Fig. 4. Example of CEM design search using Geodise database technology. Shown are design geometries, the computed frequency spectrum with the bandgap, and representative data for the objective function landscape (dots indicate sample points in parameter space).
416
J. Wason et al.
5 Conclusions and Future Work We have described a framework which allows the use of databases on the Grid in an engineer-friendly environment. We have implemented a suite of services which combine a commercial PSE (Matlab) with a core framework of open standards and service oriented technologies. We have shown how design search in electromagnetics can be supported by the Geodise database toolkit. The transparent integration of database tools into the engineering software environment constitutes a starting point for database applications in EDSO, and is only one of many potential applications in the engineering domain (CFD, CEM, etc.). The functions we have implemented to extend the Matlab environment allow engineers to share and re-use data conveniently. The automatic generation of standard metadata and support for user-defined metadata allows queries to be formed that represent the Engineer’s view of the data. Future work will involve providing tools to generate, compare and merge XML Schemas describing users' custom metadata. We will also evolve our Web service based components to GGF DAIS-WG compliant Grid services.
References [1] [2]
[3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16]
I. Foster, C. Kesselman, J. Nick, and S. Tuecke. The Physiology of the Grid: An Open Grid Services Architecture for Distributed Systems Integration, Open Grid Service Infrastructure WG, Global Grid Forum, June 22, 2002. M.P. Atkinson, V. Dialani, L. Guy, I. Narang, N.W. Paton, D. Pearson, T. Storey and P. Watson. Grid Database Access and Integration: Requirements and Functionalities. UK eScience Programme Technical Report http://www.cs.man.ac.uk/grid-db/papers/DAIS:RF.pdf The Storage Resource Broker, http://www.npaci.edu/DICE/SRB/ Geodise Project http://www.geodise.org/ Matlab 6.5. http://www.mathworks.com J. Hugunin. Python and Java: The Best of Both Worlds. 6th International Python Conference. San Jose, California, USA, 1997. G. Pound, H. Eres, J. Wason, Z. Jiao, A. J. Keane, and S.J. Cox, A Grid-enabled Problem Solving Environment (PSE) For Design Optimization Within Matlab. IPDPS-2003, April 22–26, 2003, Nice, France. M. Molinari. XML Toolbox for Matlab. http://www.soton.ac.uk/~gridem The Globus Project. http://www.globus.org/ Commodity Grid Kits. http://www.globus.org/cog/ R. Butler, D. Engert, I. Foster, C. Kesselman, S. Tuecke, J. Volmer, and V. Welch. National-Scale Authentication Infrastructure, IEEE Computer, 33(12):60–66, 2000. GridFTP, http://www.globus.org/datagrid/gridftp.html Web Services Activity, http://www.w3.org/2002/ws M. Gudgin, M. Hadley, N. Mendelsohn, J. Moreau, and H.F. Nielsen. SOAP Version 1.2, W3C Candidate Recommendation, 2002 Global Grid Forum, http://www.ggf.org/ The GEM project, http://www.soton.ac.uk/~gridem
A Resource Accounting and Charging System in Condor Environment Csongor Somogyi, Zolt´ an L´ aszl´o, and Imre Szeber´enyi BUTE Dept. of Control Engineering and Information Technology, Budapest, Hungary {csongor, laszlo, szebi}@iit.bme.hu
Abstract. The authors’ aim is to create a resource accounting and charging system for the Grid environment that can be set up as part of the supercomputing Grid prototype, which is developed in an ongoing domestic project. Stiller, Gerke, Reichl and Flury [1,2] has introduced a distributed accounting and charging concept, in which they point out that it could be applied in Grid context as well. This model has been adopted and improved by the authors in the given context; they had created the specification of a proposed solution at algorithm level. The necessary data structures and its query interfaces have been defined. The metering, accounting and pricing processes have been implemented. The system has been adapted to the Condor workload management system. The resulting application has been deployed at the departmental cluster. The authors funnel is to continue developing the remaining modules of the system and bring out the completed version as the part of the professional services for the Grid. This paper introduces the applied model, the specification that was built upon it, the results of the implementation and the operating test environment. Keywords: Grid, Resource accounting and charging, Condor
1 1.1
Introduction and Motivation Professional Services for Commercial Grids
It is necessary to consider a kind of infrastructure, which can give access to exploit many types of resources in mass without either interfering with the legacy environment of resource providers or constraining the resource consumers in the method of access, and it has to match the security and interoperability requirements, which are recently becoming more important. System scalability and high grade of diversity of utilization are also very important factors. This is the Grid initiative. The final result of this achievement is a proposal of standard. The challenging requirements against the Grid resource management have been identified by the researchers of the Globus meta-computing system [3]. The collected experiences led to draw the initial concept of the Grid architecture, a four-layer “hourglass” model [4]. They have started standardizing the architecture through the Open Grid Services Architecture model [5] using Web Services. H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 417–420, 2003. c Springer-Verlag Berlin Heidelberg 2003
418
1.2
C. Somogyi, Z. L´ aszl´ o, and I. Szeber´enyi
Research Context
Base services of the Grid may include the resource representation abstractions, resource management, informational services, data connectivity (data repositories, application repositories, file transfer, etc.) and the underlying communication protocols. On the other hand, users or consumers can only exploit the benefits given by Grid providers if they have adequate access. We have to define what the most important accessibility features are: – application development environment (like P-GRADE [7]) – refined user interface for the management of resource usage – accounting and charging of resource usage and payment services – user support (documentation, help, downloads, etc.) The Hungarian Supercomputing Grid project [6] supported by the home Ministry of Education has been created to establish the first working prototype of an interior Grid. Their aim is to suit the above mentioned demands, set up against a professional Grid – except the user support, which is a very costly business. The authors of this article have been developing the accounting and charging infrastructure to be applied in the Hungarian SuperGrid. On the other hand, their objective is to provide a flexible, sophisticated toolkit, which can be easily aligned with the requirements of the evolving Grid architecture. The authors are also involved in the EASYCOMP1 – IST Future and Emerging Technologies program – project. The goal of the project is to develop the foundation of such composition technology, which should enable users to compose (standard) components powerfully and easily.
2
System Specification
We have been applying in the field of Grid computing the charging and accounting model described by Stiller, Gerke, Reichl and Flury [1]. They started their work in order to elaborate the basis of the commerce of network (in the so-called CATI project — Charging and Accounting Technologies for the Internet) but they pointed out that this architecture is capable of being implemented in Grid context as well [2] because of its generality and modularity, which can be exploited in such a distributed environment like the Grid. The work of the authors of this article was to improve this model to adapt it to the Grid context. The CATI researchers have identified and decomposed the terms related to accounting and charging. This decomposition contains the abstract definitions of the terms metering, mediating, accounting, charging and billing and the schema of their relations. 1
“Work partially supported by the European Community under the IST programme – Future and Emerging Technologies, contract IST-1999-14191 – EASYCOMP. The authors are solely responsible for the content of this paper. It does not represent the opinion of the European Community, and the European Community is not responsible for any use that might be made of data appearing therein.”
A Resource Accounting and Charging System in Condor Environment
419
The authors of this article examined the mentioned model in the context of Grid and proposed an escalation of the original model. The three main subsystems (accounting – involving metering and mediation processes –, charging and billing) have been decomposed into their atomic elements, which have been matched with software components. The Grid related processes, which pass information between the Grid and the accounting and charging process, have been identified. The necessary data abstractions have been defined. These include the representations of Grid related entities (e.g. resource, resource user ), the parameters that are exchanged between the atomic processes of the system and the data constructs that are stored or used during the entire process (like accounting record, pricing rule, charging record and receipt entries). For each of these abstractions, the corresponding data structures have been declared. The process flow has been summarized in a data-flow model, while the data abstractions are represented as an entity-relationship model. Finally, a textual description of the dynamic behavior of the accounting and charging processes has been created. Both the logical, which was briefly introduced above, and the physical specification of the system have been completed. The most important requirement, distributiveness, is fulfilled. The metering and mediation modules should only be placed locally at the resource, while the rest of the components can be placed independently anywhere. Surely, one component can serve an arbitrary set of resources or resource users, it does not matter how the information at various levels (accounting, charging or billing) are grouped together. Actually, this approach provides us balanced information processing and storage.
3
Implementation and Results
The implementation of the fundamental components (metering, accounting and pricing) has been completed. CPU time consumption has been chosen as sample measured parameter. SQL and HTTP/XML are used for interfacing the different components. Later, these can be easily changed into Web Services. Development work is done in UNIX environment using mostly Perl and partially C languages, Perl DBI and GTop libraries. Condor system [8] is the selected local job manager infrastructure. Condor can run a job wrapper each time a job is started on a given host. This wrapper was developed in order to establish binding between the job manager and the accounting subsystem. The developed parts of the system are deployed at the departmental Condor cluster. Test runs are performed demonstrating the operation of the accounting and charging system. While the structure and the behavior of the most accounting and charging components are clarified, metering still faces with different challenges. The various platforms treat resource consumption parameters, especially process accounting parameters, in different ways. On the other hand, there are no stan-
420
C. Somogyi, Z. L´ aszl´ o, and I. Szeber´enyi
dard(ized) units defined. Since our measuring method takes resource usage snapshots at equal time-intervals, sampling error can be perceived. Since we need to know the resource consumption at a given moment as well as the total amount of consumption at the resource usage completion, we have to examine what accuracy is necessary and may have to combine the usage of different operating system provided ways of resource usage accounting in order to achieve accuracy precisely. We are investigating these questions in the near future.
4
Summary and Conclusions
The initial accounting and charging model has been extended in order to fit the requirements against the Grid context. The authors have completed the specification of the proposed solution. Partial development of the system, the accounting and the pricing modules have been completed. The application has been merged with the Condor workload management environment and the test system has been set up at the departmental cluster. Test runs have been performed in order to demonstrate the operation of the system. Our funnel is to continue the development of the remaining parts of the system, integrate the solution with the mentioned domestic supercomputing Grid and investigate the metering related issues.
References 1. B. Stiller, J. Gerke, P. Reichl and P. Flury: A Generic and Modular Internet Charging System for the Cumulus Pricing Scheme. Journal of Network and Systems Management, 3(9) (September 2001) 293–325 2. B. Stiller, J. Gerke, P. Reichl, P. Flury and Hasan: Charging Distributed Services of a Computational Grid Architecture. IEEE International Symposium on Cluster Computing (CCGrid 2001), Workshop on Internet QoS for the Global Computing (IQ 2001), (May 2001) 596–601 3. K. Czajkowski, I. Foster, N. Karonis, C. Kesselman, S. Martin, W. Smith, S. Tuecke.: A Resource Management Architecture for Metacomputing Systems. Proc. IPPS/SPDP ’98 Workshop on Job Scheduling Strategies for Parallel Processing (1998) 4. I. Foster, C. Kesselman and S. Tuecke.: The Anatomy of the Grid: Enabling Scalable Virtual Organizations. International J. Supercomputer Applications, 15(3) (2001) 5. I. Foster, C. Kesselman, J. Nick and S. Tuecke: The Physiology of the Grid: An Open Grid Services Architecture for Distributed Systems Integration. Open Grid Service Infrastructure WG, Global Grid Forum (June 22, 2002) 6. IKTA-00075/2001, Hungarian Supercomputing Grid. Tender for Information and Communication Technology Applications by the Hungarian Ministry of Education, Budapest (November 2002) 7. P. Kacsuk: Parallel Program Development and Execution in the Grid. Proc. of PARELEC 2002, Warsaw (2002) 8. Douglas Thain and Miron Livny: Condor and the Grid. in Fran Berman, Anthony J.G. Hey, Geoffrey Fox, editors, Grid Computing: Making the Global Infrastructure a Reality, John Wiley (2003)
Secure Web Services with Globus GSI and gSOAP Giovanni Aloisio1 , Massimo Cafaro1 , Daniele Lezzi1 , and Robert van Engelen2 1
High Performance Computing Center University of Lecce/ISUFI, Italy
[email protected] [email protected] [email protected] 2 Computer Science Department Florida State University, USA
[email protected] Abstract. In this paper we describe a plug-in for the gSOAP Toolkit that allows development of Web Services exploiting the Globus Security Infrastructure (GSI). Our plug-in allows the development of GSI enabled Web Services and clients, with full support for mutual authentication/authorization, delegation of credentials and connection caching. The software provides automatic, transparent transport-level security for Web Services and is freely available.
1
Introduction
Recently, the Web Services framework [1] has gained considerable attention. Based on XML (Extensible Markup Language) [2] technologies like SOAP (Simple Object Access Protocol) [3], WSDL (Web Services Description Language)[4], WSFL (Web Services Flow Language) [5] and UDDI (Universal Description, Discovery and Integration) [6], the Web Services approach to distributed computing represents the latest evolution supporting the creation, deployment and dynamic discovery of distributed applications. As a matter of fact, the Internet and the Web allow to publish and retrieve documents easily, and to access a number of commercial, public and e-government services. However, the focus on the usage of such services is now been shifted from people to software applications. The Web Services framework makes this shift possible, due to the convergence of two key technologies: the Web, with its well known and universally accepted set of standard protocols for communication, and Service-Oriented computing where both data and business logic is exposed through a programmable interface e.g. CORBA (Common Object Request Broker Architecture), Java RMI (Remote Method Invocation), DCE RPC (Remote Procedure Call). Web Services can be accessed through the HTTP (Hyper Text Transfer Protocol) and HTTPS (Secure Hyper Text Transfer Protocol) protocols and utilize XML to exchange data. This implies that Web Services are independent of platform, programming language and network infrastructure. H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 421–426, 2003. c Springer-Verlag Berlin Heidelberg 2003
422
G. Aloisio et al.
Even in the Grid community, the focus is now shifted from protocols [7] to Grid Services [8], as envisioned by the Open Grid Services Architecture (OGSA) [9]. Grid Services extend the Web Services framework, and the Grid itself becomes an extensible set of Grid Services that may be aggregated to provide new capabilities. However, a Grid Service is ”a (potentially transient) stateful service instance supporting reliable and secure invocation (when required), lifetime management, notification, policy management, credential management, and virtualization” [8]. So, Grid Services leverage both WSDL and SOAP but additional interfaces able to manage service lifetime, policies and credentials, and to provide support for notification are mandated by the OGSA specification. Since the OGSA specification is not yet completed and the Globus Toolkit v3 is not yet available to develop production software (only an alpha version has been released as of this writing), we decided to adopt the Web Services framework jointly with the Globus Toolkit v2 as our middleware/computing infrastructure in the GridLab project [10]. The adoption of Globus GSI [11] as the security infrastructure and of the gSOAP Toolkit [12] for the development of Web Services led us to write the GSI plug-in for gSOAP, needed to secure the Web Services developed in the context of the GridLab project. The plug-in provides automatic, transparent transport-level security for Web Services and is freely available [13]. The paper is organized as follows. Section 2 describes the gSOAP Toolkit. We present our GSI plug-in in Section 3, recall related work in section 4 and conclude the paper in Section 5.
2
The gSOAP Toolkit
The gSOAP toolkit is a platform-independent development environment for C and C++ Web services. The toolkit provides an easy-to-use RPC compiler that produces the stub and skeleton routines to integrate (existing) C or C++ applications into SOAP/XML Web services. A unique aspect of the gSOAP toolkit is that it automatically maps native C/C++ application data types to semantically equivalent XML types and vice versa. This enables direct SOAP/XML messaging by C/C++ applications on the Web. As a result, full SOAP interoperability can be achieved with a simple API relieving the user from the burden of SOAP details, thus enabling him or her to concentrate on the applicationessential logic. The toolkit uses the industry-standard SOAP 1.1/1.2 and WSDL 1.1 protocols and offers an extensive set of features that are competative to commercial implementations, including stand-alone HTTP server capabilities, Zlib compression, SSL encryption, and streaming direct internet message encapsulation (DIME) attachments. For many companies, gSOAP has proven an excellent strategy for developing Web services based on C and C++ applications. For example, gSOAP is integrated in the IBM alphaWorks Web Services Tool Kit for Mobile Devices (WSTKMD) [14] The gSOAP toolkit was designed with ease-of-use in mind. It exploits a novel schema-driven XML parsing technique to deserialize C/C++ application data
Secure Web Services with Globus GSI and gSOAP
423
from SOAP/XML messages in one sweep, thereby eliminating the overhead that is incurred by SOAP/XML software with multi-tier communication stacks. gSOAP is available for download from SourceForge [15] and is licensed under the open source Mozilla Public License 1.1 (MPL1.1).
3
The GSI Plug-in for gSOAP
Our plug-in exploits the modular architecture of the gSOAP Toolkit that enables a simple extension mechanism of gSOAP capabilities. To take advantage of a plug-in, a developer must register it with gSOAP, so that full access to run-time settings and function callbacks is granted. The registration associates the plug-in’s local data with gSOAP run-time and is done using the gSOAP soap register plugin function, supplying as one of the arguments the plug-in initialization function. In our case, we perform the necessary initialization steps inside the globus gsi function : we activate the Globus Toolkit I/O module, set-up local data and extend gSOAP capabilities overriding gSOAP function callbacks. The last initialization step is to provide two callbacks that will be used by the gSOAP environment respectively to copy and delete the plug-in’s local data (when de-registering the plug-in). The plug-in’s local data can be accessed through the soap lookup plugin function. Currently, as local data we have Globus I/O related variables (connection attribute, handle, etc), the distinguished names that identify a client or a server when mutual authentication is performed, and the pathname where on the local file system the proxy that a client sends when performing delegation of credentials is written by our plug-in. Finally, we have a boolean variable that distinguishes a client from a server. This mechanism is exploited for instance in the Globus I/O authorization callback that a developer must provide to perform authorization upon authentication: if the software acts as a client, then authorization is based on the server’s identity (distinguished name as found in the X509v3 certificate) and vice-versa. We now briefly describe how the plug-in functions utilize the Globus Toolkit I/O API to provide gSOAP with automatic, transparent transport-level security. The gsi connect function sets the client mode and calls globus io tcp connect to establish a connection to the remote Web Service; the gsi disconnect function checks the state of the connected handle and closes the connection calling globus io close; the variables related to the server’s or client’s identity that have been dynamically allocated, are then freed. The gsi send function is in charge of actually sending data on the Globus I/O connected handle; this is done calling in a loop until needed globus io write. Symmetrically, the gsi recv function reads data from a Globus I/O connected handle calling globus io read. The gsi listener, gsi listen and gsi accept functions are all needed to develop a server; gsi listener sets server mode and calls globus io tcp create listener to create a Globus I/O listening handle, while gsi listen calls globus io tcp listen and blocks waiting for incoming connections on the listening handle. Thus, the
424
G. Aloisio et al.
Globus function behaves differently from the traditional TCP sockets API listen call. Finally, the gsi accept function calls globus io tcp accept and in case of success creates the connection handle and calls globus io tcp get remote address to retrieve the peer’s IP address and port. The gsi connection caching and gsi reset connection caching are used respectively to setup and reset connection caching. This is achieved using the gSOAP internal mechanism for keep-alive connections and calling the Globus function globus io attr set socket keepalive as needed. The other functions we provide are needed to setup properly the GSI channel: the developer is allowed to setup the TCP socket reuse addr option, which is useful server side, and to setup authentication, channel mode, protection mode, authorization mode and delegation mode. Both clients and servers developed using our plug-in can use the Globus I/O authorization callback mechanism; this entails writing a related Globus function called globus io secure authorization callback to enforce the developer’s policy. The plug-in software requires GNU autotools (autoconf v2.57, automake v1.7.2), the Globus Toolkit v2.x and the gSOAP Toolkit v2.2.3d. We provide in our distribution example servers and their related clients, that show how to write a simple threaded server, a pre-threaded server, a simple fork server and a pre-forked server. The threaded servers provide a good example of how to do mutual authentication/authorization and how to setup and use connection caching; the fork servers in addition show how to do delegation of credentials: the server receives from the client the delegated credentials and uses them to submit a job to a remote Globus gatekeeper. Our software is licensed under the open source GNU General Public License.
4
Related Work
A similar GSI plug-in for gSOAP has been developed in the context of the Globus project for a C based preliminary version of OGSA. This software provides basically the same functionalities described here, however, our plug-in has been made available to the Grid community before and it is widely used in several academic institutions and software companies, while the Globus plug-in will not be considered production-level software until the final official release of the Globus Toolkit v3. Moreover, the new version of the Globus gSOAP plug-in, to be included in the C based alpha version of OGSA, will support in addition to transport-level security a new GSI mechanism for securing grid services, using message-level security. The issues that arise using transport-level security are the high computational cost (the entire message must be encrypted and/or signed), the problem related to firewalls (non privileged ports must be opened), and the problem of intermediate hops. A transport-level security mechanism provides two-way encryption and one-way or two-way authentication between a predefined pair of SOAP message endpoints, but SOAP request and response messages may be required to traverse multiple hops between the Web service and the consuming
Secure Web Services with Globus GSI and gSOAP
425
application. In this case, message security and user identity can be compromised at the intermediate hops. A message-level security mechanism is safer because there is no exposure at intermediaries (hop-to-hop vs end-to-end) and flexible because building only on proven Web infrastructure (HTTP etc). Another advantage is that firewalls are not an issue here (Web protocols are usually allowed through a firewall) and the associated computational cost may be relatively low, due to the possibility of selective encryption and/or signature of SOAP messages (only some XML elements actually need encryption because they contain sensitive information). Web Services security is an active research field, and many specifications are now available. We are now developing a complementary version of our plug-in to support message-level security. Both our software and the Globus based plug-in will harness the following specifications: – – – –
WS Secure Conversation [16]; WS Security [17]; XML Encryption [18]; XML Signature [19].
SOAP messaging is enhanced using message integrity, confidentiality and authentication. These specifications allow exchanging a security context between a Web Service and a client, establishing and deriving session keys, and provide a general purpose mechanism for associating security tokens with messages. An important property of these specifications is that they are extensible, i.e., they support multiple security token formats.
5
Conclusions
In this paper we have reported about a GSI plug-in for the gSOAP Toolkit. Our plug-in allows the development of GSI enabled Web Services and clients, with full support for mutual authentication/authorization, delegation of credentials and connection caching. The software is being used in several academic institutions for grid projects and testbeds including the GridLab project, moreover, some software companies have demonstrated an interest for it. The plug-in provides automatic, transparent transport-level security for Web Services and is freely available. We will continue to support the GSI plug-in to provide transport-level security for Web Services; in addition our focus is currently on the development of a complementary version of the software to provide message-level security.
Acknowledgements. We gratefully acknowledge support of the European Commission 5th Framework program, grant IST-2001-32133, which is the primary source of funding for the GridLab project.
426
G. Aloisio et al.
References 1. 2. 3. 4. 5. 6. 7.
8. 9.
10. 11.
12.
13. 14. 15. 16. 17. 18. 19.
Kreger, H.: Web Services Conceptual Architecture WSCA 1.0. IBM, 2001 XML specification. http://www.w3.org/XML/Core/ SOAP specification. http://www.w3.org/TR/SOAP/ WSDL specification. http://www.w3.org/TR/wsdl WSFL specification. http://www.ibm.com/software/solutions/webservices/pdf/WSFL.pdf UDDI specification. http://www.uddi.org/specification.html Foster, I., Kesselmann, C., Tuecke, S.: The Anatomy of the Grid: Enabling Scalable Virtual Organizations. International Journal Supercomputer Applications, Vol.15, 2001, No. 3, pp. 200–222 Foster, I., Kesselmann, C., Nick, J., Tuecke, S.: Grid Services for Distributed System Integration. Computer, Vol. 35, 2002, No. 6, pp. 37–46 Foster, I., Kesselmann, C., Nick, J., Tuecke, S.: The Physiology of the Grid: An Open Grid Services Architecture for Distributed System Integration. Technical Report for the Globus project. http://www.globus.org/research/papers/ogsa.pdf The GridLab project. http://www.gridlab.org Foster, I., Kesselmann, C., Tsudik G., Tuecke, S.: A security Architecture for Computational Grids. Proceedings of 5th ACM Conference on Computer and Communications Security Conference, pp. 83-92, 1998. Van Engelen, R.A., Gallivan, K.A.: The gSOAP Toolkit for Web Services and Peer-To-Peer Computing Networks. Proceedings of IEEE CCGrid Conference, May 2002, Berlin, pp. 128–135 Cafaro, M., Lezzi, D., Van Engelen, R.A.: The GSI plugin for gSOAP. http://sara.unile.it/∼cafaro/gsi-plugin.html IBM alphaWorks, Web Services Tool Kit for Mobile Devices, http://www.alphaworks.ibm.com/tech/wstkMD The gSOAP Toolkit. http://gsoap2.sourceforge.net WS Secure Conversation specification. http://www-106.ibm.com/developerworks/library/ws-secon/ WS Security specification. http://www.oasis-open.org/committees/wss/ XML Encryption. http://www.w3.org/Encryption/2001/ XML Signature. http://www.w3.org/Signature/
Future-Based RMI: Optimizing Compositions of Remote Method Calls on the Grid Martin Alt and Sergei Gorlatch Technische Universit¨ at Berlin, Germany Abstract. We argue that the traditional RMI (remote method invocation) mechanism causes much unnecessary communication overhead in Grid applications, which run on clients and outsource time-intensive method calls to high-performance servers. We propose future-based RMI, an optimization to speed up compositions of remote methods, where one method uses the result of another method as an argument. We report experimental results that confirm the performance improvement due to the future-based RMI on a prototypical Grid system on top of Java.
1
Introduction and Motivation
Grid computing aims to combine different kinds of computational resources connected by the Internet to make them easily available to a wide user community. One popular approach to developing applications for Grid-like environments is to provide libraries on high-performance servers, which can be accessed by clients using some remote invocation mechanism, e. g. RPC/RMI. Our work is motivated by many important applications where remote method calls are composed with each other. For illustration, we use a very simple Java code fragment, where the result of method1 is used as an argument by method2: result1 = server1 . method1 (); result2 = server2 . method2 ( result1 ); The execution of the methods can be distributed across different servers of the Grid, i. e. with different RMI references assigned to server1 and server2. The timing diagram for such a distributed composition of methods using standard RMI is shown in Fig. 1 (left): the result of method1 is first sent to the client, and from there (as a parameter of method2) to the second server. In larger applications, this leads to much unnecessary communication between the client and servers, thus increasing execution time of Grid programs.
2
Optimizing RMI Using Future References
The aim of future-based RMI is to optimize the remote execution of compositions of methods. The idea is that a method invocation immediately returns a remote reference to the result (a future reference), which is sent to the client and can immediately be passed on to the next method. Attempts to retrieve the referenced data are blocked until the data becomes available. For details, see [1]. H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 427–430, 2003. c Springer-Verlag Berlin Heidelberg 2003
428
M. Alt and S. Gorlatch
The timing diagram for our future-based RMI is shown in Fig. 1 (right): the reference to result1 is returned immediately and passed on to method2. The second server requests result1 from the first one, while the first method is still executing. When the result is available it is sent directly to the second server.
Server2
Server1
Client
method1
Server2
Server1
Client
method1 reference to result1 method2 (reference to result1) request result1
result1
result1 result2
method2 ( result1 ) result2
Fig. 1. Distributed composition using plain RMI (left) and future-based RMI (right)
The future-based RMI has two advantageous properties in the Grid context: Laziness: the amount of data sent from the server to the client upon method completion is reduced substantially, because only a remote reference is returned. This reference is passed to the next server, and used to request the result. The result itself is sent directly from one server to the next one. Asynchronity: communication between client and server overlaps with computations, because future references are returned immediately. Thus, network latencies can be hidden.
2.1
Implementation and Exception Handling
Under our optimized RMI, methods do not return their results directly. Instead, they return a remote reference to an object of the new class RemoteReference which is instantiated on the server side. It provides two methods: public Object getValue () ...; // remotely accessible public void setValue ( Object o ) ...; The setValue() method is called by the application method’s implementation when the result of the method is available, passing the result as a parameter. Method getValue() is used to retrieve this result and may be called remotely. Calls to getValue() block until a value has been assigned using setValue(). If getValue() is called remotely, the result is sent over the network to the caller. To execute a method asynchronously after returning a future reference to the caller, remote methods spawn a new thread to carry out computations and then return immediately. To avoid thread-creation overhead, a pool of threads is created when the server starts; threads are taken from the pool on demand. From the user’s point of view, a distributed composition of methods under the future-based RMI is expressed in much the same way as with plain RMI.
Future-Based RMI: Optimizing Compositions
429
The pseudocode for a method taking a RemoteReference as parameter and returning another one as result is as follows: public RemoteReference method ( RemoteReference ref ) { Object parameter = ref . getValue (); RemoteReference ret = new RemoteReference (); /* execute method1 in a new thread , that will call ret . setValue () upon completion */ Thread t = new Method1Thread ( parameter , ret ); return ret ;} An important mechanism for error handling in Java are exceptions, which should thus also be present in a distributed setting. With plain RMI, exception handling resembles local method invocation: an exception thrown in the remote method is caught on the server, sent to the client and rethrown there. In the future-based RMI, a remote method invocation returns to the calling client before the method on the server is actually completed; thus, an exception thrown on the server cannot be thrown immediately on the client. In our implementation, the exception is caught on the server and stored in the object, that the future reference returned to the client points to. When the getvalue() method of this object is called, the exception is rethrown. If the caller of getvalue() was another server-sided method, the exception is caught again and wrapped in the next object referenced by a future reference until it finally reaches the client.
3
Experimental Results
Our testbed environment consists of two university LANs, one at TU Berlin and the other at the University of Erlangen, at a distance of approx. 500 km. We used a SunFire6800 as server in Berlin and an UltraSparc IIi 360 MHz as the client in Erlangen, both with SUN JDK1.4.1 (HotSpot Client VM, mixed mode).
1800
plain RMI improved RMI lower bound
6000
plain RMI improved RMI lower bound
5000
1400
Time [ms]
Time [ms]
1600
1200 1000
4000 3000 2000
800 1000
600
0 200KB
400KB 600KB 800KB Parameter Size [byte]
1MB
200 250 300 350 400 450 500 550 600 Matrix Size
Fig. 2. Runtimes for simple method composition (left) and linear system solver (right).
We measured the performance of the small code from Sect. 1, with method1 and method2 both taking 500 ms, and the amount of data sent ranging between 100 KB and 1000 KB. Our second case study is a linear equations system solver using a server-side matrix library; it consists of composed RMI calls for solving a minimalization problem, matrix multiplication and subtraction.
430
M. Alt and S. Gorlatch
Fig. 2 (left) shows the runtimes for three different versions of the small code: (1) two method calls with plain RMI, (2) two calls with future-based RMI, and (3) one method call that takes twice as much time as the original call. We regard the one-call version as providing ideal runtime (“lower bound”) for a composition of remote methods. The figure presents five measurements for each version of the program, with the average runtimes for each parameter size connected by lines. For both the simple method composition and the linear system solver, the figure shows that the future-based RMI is between 100 ms and 350 ms faster than the plain RMI. The composition under future-based RMI is only 10-15 ms slower than the “lower-bound” version, which means that our optimizations eliminated between 85% and 97% of the original overhead.
4
Conclusions and Related Work
We have proposed future-based RMI, which greatly reduces communication overhead and network dataflow when executing compositions of remote methods. The concept of hiding latencies using futures was studied e. g. in [2] and implemented in Java (e. g. [3,4,5]). However, these approaches lack the ability to send futures to remote servers and can thus not realize server/server communication. This feature can be found in [6], where RMI calls are optimized using callaggregation and where a server can invoke methods on another server directly. While this approach optimizes RMI calls by reducing the amount of data, the method invocations are not asynchronous as in our implementation, instead they are delayed to find as much optimization possibilities as possible. The prototypical implementation using Sun’s Java RMI confirmed the advantages of our future-based solution, which can be built on top of an arbitrary RMI mechanism, including optimized versions like KaRMI or Manta. Acknowledgements. We wish to thank the anonymous referees and also Thilo Kielmann and Robert V. van Nieuwpoort for their very helpful comments on the preliminary version of this paper.
References 1. Alt, M., Gorlatch, S.: Optimizing the use of Java RMI for grid application programming. Technical Report 2003/08, TU Berlin (2003) ISSN 1436-9915. 2. Walker, E.F., Floyd, R., Neves, P.: Asynchronous remote operation execution in distributed systems. 10th Int. Conference on Distributed Computing Systems. (1990) 3. Raje, R., Williams, J., Boyles, M.: An asynchronous remote method invocation (ARMI) mechanism in Java. Concurrency: Practice and Experience 9 (1997) 4. Falkner, K.K., Coddington, P., Oudshoorn, M.: Implementing asynchronous remote method invocation in Java. In: Proc. of Parallel and Real Time Systems (1999) 5. Baduel, L., Baude, F., Caromel, D.: Efficient, flexible, and typed group communications in Java. In: JGI’02, Seattle, ACM (2002) 6. Yeung, K.C., Kelly, P.H.J.: Optimising Java RMI programs by communication restructuring. In: Middleware 2003, Springer (2003)
Topic 7 Applications on High-Performance Computers Jacek Kitowski, Andrzej M. Goscinski, Boleslaw K. Szymanski, and Peter Luksch Topic Chairs
The emergence of low-cost PC clusters together with the standardization of programming models (MPI and OpenMP) have paved the way for parallel computing to come into production use. In all domains of high performance computing, parallel execution routinely is considered as one of the major sources of performance. In some domains, like computational fluid dynamics, commercial codes already offer parallel execution as an option. In new domains, like bioinformatics, parallel execution is considered early in the design of algorithms and software. Besides clusters, grid computing is receiving increasing attention. This year, 24 papers were submitted, 13 of which have been accepted (acceptance rate: 54.2%), eight as regular papers, three as short papers. The topic is subdivided into three sections. Numerical Analysis. Algorithms from linear algebra are considered in “Parallel Linear System Solution and its Application to Railway Power Network Simulation” and “Improving Performance of Hypermatrix Cholesky Factorization”. “Implementation of Adaptive Control in Algorithms in Robot Manipulators using Parallel Computing” deals with robot manipulators whose dynamic parameters are not completely known. The paper “Cache Performance Optimizations for Parallel Lattice Boltzmann Codes” shows that efficient utilization of the memory hierarchy is essential to application performance. Algorithms from other domains. Two papers address image processing: “Low Level Parallelization of Nonlinear Diffusion Filtering Algorithms for Cluster Computing Environments” and “Interactive Ray Tracing on Commodity PC ”. The paper “Comparing two Long Biological Sequences Using a DSM System” describes parallel implementations of the Smith-Waterman algorithm on a distributed shared memory system. “Parallel Agent-Based Simulation on a Cluster of workstation” presents a discrete time simulation implemented in a dynamically typed language. “Effectiveness of Parallelizing the ILOG-CPLEX Mixed Integer Optimizer in the PUBB2 Framework ” describes the parallelization of branch and cut algorithms that solve mixed integer optimization problems on clusters of PCs. Management of parameter studies and automatic optimization is addressed in three papers. “CAD-Grid: Corporate Wide Resource Sharing for Parameter Studies” describes a framework for corporate-wide shared use of simulation software and HPC resources that is being deployed in an industrial environment. A tool that manages execution of a large number of experiments, H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 431–432, 2003. c Springer-Verlag Berlin Heidelberg 2003
432
J. Kitowski et al.
including resubmission of failed jobs, is presented in “Towards Automatic Management of Embarrassingly Parallel Applications”. “Two Dimensional Airfoil Optimization using CFD in a Grid Computing Environment” describes a tool for automatic CFD optimization that assembles different industrial software components.
CAD Grid: Corporate-Wide Resource Sharing for Parameter Studies Ed Wheelhouse1 , Carsten Trinitis2 , and Martin Schulz3 1
Department of Computer Science, University of Newcastle, Newcastle, UK
[email protected] 2 Technische Universit¨ at M¨ unchen (TUM), Institut f¨ ur Informatik Lehrstuhl f¨ ur Rechnertechnik und Rechnerorganisation, Boltzmannstr. 3, 85748 Garching bei M¨ unchen, Germany
[email protected] 3 School of Electrical and Computer Engineering, Cornell University, Ithaca, NY, 14853, USA
[email protected] Abstract. The optimization process for most modern engineering problems involves a repeated modeling of the target system, simulating its properties, and refining the model based on the results. This process is both time and resource consuming and therefore needs to rely on a distributed resource sharing framework in order to optimally exploit the existing resources and minimize the response time for the design engineers. We have implemented such a framework for the design process of high voltage components and have shown its applicability to a real industrial environment. First results are very encouraging and show a high acceptance rate with the end-users. In addition, experiments with various different models show the profound impact of the optimization on the design of high-voltage components.
1
Motivation
The geometric shape of transformers and other high voltage gear has a profound impact on their electrical properties and on their performance. Suboptimal designs can lead to overheating, flashovers between close parts, higher energy loss rates, etc. . It is therefore necessary to optimize the geometric shape already in the initial design phase using a detailed simulation of the electric properties of the designs. This enables the engineers to detect critical regions and to change the respective CAD geometry in order to obtain an optimal design. This can be automated using an iterative optimization approach. During each iteration, a new CAD model is generated and used as the input for the following simulation of the electric field. The result of the simulation is then used to refine the model. This process continues until an optimal design has been found. Each iteration requires repeated access to the CAD modeling tool and to the compute server for the simulation. The requests for these resources are interleaved and the usage of each resource potentially requires a significant amount H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 433–440, 2003. c Springer-Verlag Berlin Heidelberg 2003
434
E. Wheelhouse, C. Trinitis, and M. Schulz
of time since both model generation and simulation are non-trivial operations. In addition, the resources for each step are strictly limited by the number of available parallel systems for the simulation and by appropriate licenses for the CAD modeling. Therefore, a single optimization request should not block a compute or a CAD modeling resource for its entire runtime and neither should each engineer require his or her own, dedicated set of resources. Instead, a corporatewide resource sharing infrastructure is needed, which dynamically assigns jobs to available resources. We have implemented such a resource sharing environment in cooperation with ABB corporate research. Design engineers can access the necessary resources for the complete optimization process from their desktop. The overall process has been integrated seamlessly into the design workflow. In addition, the infrastructure enables an efficient resource sharing across all corporate–wide resources, which eliminates long response times for optimization requests, allows for a higher system utilization, and significantly reduces the number of required software licenses and hence cost.
2
Optimization Process
In the area of numerical optimization, several software packages are available that enable the user to apply different algorithms to a specific simulation problem, most notably Optimus [7], the DAKOTA iterator toolkit [6], and Nimrod/O [5]. These tools, however, are primarily designed as universal optimization tools requiring significant changes for the use in electric field specific simulation. In addition, these system are unable to cope with limited resources during the model generation phase, as it is given here due to the CAD modeling. We have therefore decided to design our own optimization environment, which we will describe below. 2.1
Optimization Loop
Our framework consists of three primary components: a parametric CAD modeling system based on a commercially available system, a model evaluation component using field simulation, and the numerical optimization algorithm. The latter is designed to restrict the search space to a small subset for an optimal set of parameters. The workflow between these components is as follows: Starting with an initial set of parameters, the CAD program generates the first instance of the model that is to be calculated. The model is prepared for the field calculation, i.e. the boundary conditions and dielectrics are assigned and the model is discretized. Using simulation, the quality of the generated model is computed and passed to the optimizer. Combined with the result of all previous simulation runs within this optimization invocation, the optimization algorithm defines a new set of design parameters. These design parameters are again read by the parametric CAD modeler, which creates a new instance of the model to be optimized.
CAD Grid: Corporate-Wide Resource Sharing for Parameter Studies
435
In [4] several minimization algorithms have been investigated for a two dimensional field optimization problem. Based on the results of this study, the Hooke-Jeeves Method [8], the Nelder-Mead Method [10], and the Fletcher-Reeves Algorithm [12] have been used for the optimization process in this work. This optimization loop is repeated until a termination criterion is reached. The exact criterion depends on the optimization algorithm in use. In any case, it indicates that a local minimum has been found. Due the physical properties of this particular problem, this will be either equivalent to the global minimum or sufficiently close to it. 2.2
Electrical Simulation
In order to facilitate the evaluation of generated models, a simulation environment for three dimensional electric fields in real-world high voltage apparatus is required. From a mathematical point of view, the calculation of fields can be described as the calculation of the potential Φ(x, y, z) and the electric field strength E(x, y, z) = −∇Φ. This can be achieved by solving Laplace’s differential equation ∆Φ = 0. For electrostatic fields this differential equation can be solved by ρ(r q ) 1 dV (1) Φ(r p ) = 4π |r p − r q | with r p being the radius vector in the respective point of interest and r q being the radius vector in the integration point. To solve these problems, Boundary Element Methods [3] have proven to be very efficient. They reduce the problem to a system of algebraic equations by discretizing the model’s surfaces with well selected, small curvilinear patches (boundary elements) on the interfaces between media with different material characteristics. Over a boundary element, the field is expressed as an analytical interpolation function between the field values at the nodes (element vertices). Based on these principles, a parallel simulation environment named POLOPT [2] has been developed in cooperation with ABB corporate research and is in production for the design of high-voltage equipment. It uses a master/slave approach for its computation and is implemented using MPI [9] as the parallel programming model. Encouraging results showing a high efficiency have been achieved under production conditions using large-scale input sets [11]. 2.3
Implications
The overall optimization process requires repeated access to two kind of resources: compute servers to execute the electric field simulation and CAD servers to perform the model generation. The requests for these two are always interleaved and require both substantial execution time. It is therefore inefficient to allocate both resources throughout the whole runtime of the optimization process; instead both resources should be allocated on demand in order to allow a more efficient execution of concurrent optimization requests.
436
E. Wheelhouse, C. Trinitis, and M. Schulz
In addition, in a corporate environment these resources should ideally be shared across the whole company. Each of them is associated with a non-trivial amount of money (the compute servers are high–end clusters and CAD servers are associated with expensive CAD licenses). Such resource sharing allows an optimal utilization of each resource and hence the reduction of the number of required resources. This cuts cost and reduces the number of sites hosting the resources, which allows more centralized system management with a lower total cost of ownership.
3
Software Architecture and Implementation
This kind of corporate-wide resource sharing can be achieved in the form of a grid-like infrastructure. This grid splits into two main components: a) the grid for the compute servers and b) a grid for the CAD servers. As a result, the final system, which we call CAD grid, has to deal with two orthogonal sets of resource restrictions: the availability of a set of specific machines required for the simulation and the availability of CAD licenses (the CAD programs itself run on standard workstations). 3.1
System Structure
To achieve such flexible resource management we have designed a modular system. Its structure is depicted in Figure 1 and consists of the following five major components: 1. Clients, from which jobs get submitted. 2. A set of compute clusters to execute the field simulation, forming the simulation grid. 3. A set of CAD workstations to generate the models depending on varying input parameters, forming the CAD grid. 4. The optimization module executing the optimization algorithm. 5. A central coordinator, which retrieves jobs from the clients and is responsible for the execution. The latter two components will usually be combined within one system, as the optimization algorithm does not require major compute times. Hence it is not necessary to offload this work to remote server farms analog to the simulation or the CAD grid. Note, that this central component, which forms the cornerstone of our framework, does not impose a major bottleneck, since the execution times of the model generation and the electric field simulations are several orders of magnitude higher than the one consumed by the coordinator. This coordinator is implemented as a server, which acts as the interface for communication involving the clients. An important consequence of this design is that the client needs no knowledge of the overall architecture of the system. In particular, it need not know the location of the CAD hosts or the simulation clusters. Therefore, changes may be made to the rest of the system without
CAD Grid: Corporate-Wide Resource Sharing for Parameter Studies
437
altering the client’s view of it. This ensures the easy-of-use of the overall system, as end-users generally should not be influenced by such configuration changes, and at the same time significantly increases the manageability and the ability to tolerate system faults. 3.2
Workflow
When a job is submitted by a client to the coordinator, all of the CAD files, which specify the model to be analyzed, are sent, along with any design constraints and the initial parameter values as specified by the user. The optimization program is then started and loops until it reaches an acceptable value. The loop consists of the following stages: 1. An available CAD server is selected depending on the availability of a (potentially floating) software license and the capabilities of the target machine. The CAD files and parameter values needed to generate the model are then sent to the chosen CAD server. 2. The CAD server generates the specified model. For this task it uses special API software allowing to connect to the CAD program. In our work, we used the CAD package Pro/Engineer together with the Pro/Toolkit API [1]. 3. A file encapsulating the required details of the CAD model is returned to the coordinator. 4. The coordinator selects an available compute cluster within the simulation grid depending on the compute requirements for the chosen task and then sends the file containing the CAD to this system and initiates POLOPT. 5. POLOPT computes the electric field of the given model and computes the objective value used for the optimization (in our case the maximal field value). 6. The objective value is passed back to the coordinator. 7. The design parameters of interest and the objective value are handed to the optimization algorithm. It analyzes the value and, if appropriate, generates a new set of parameters and sends its result to the coordinator. 8. Depending on the result received from the optimization algorithm, the coordinator starts again with step 1) or aborts the loop and returns the objective value together with the final set of parameters and the last CAD model to the client. After the completion of the iterative optimization process, the client receives the final result in the form of an optimized CAD model. This can then be displayed together with the final field distribution for a final investigation by the design engineer. 3.3
User Interface
In order to ease the use of the system, we have implemented a graphical user interface which provides a straightforward access to the optimization framework. Users can upload their initial files and specify the design parameters, which
438
E. Wheelhouse, C. Trinitis, and M. Schulz
Simulation Cluster Node
Node
Node
Node
Simulation Cluster Node
Node
Node
Node
Node
Node
Node
Node
Optimization Server Optimization Algorithm
Model file
CAD Server CAD files & parameters
Parameters
Objective Value & Constraints
Model file
Coordinator Objective Value
Final CAD file and parameters
Client with GUI
CAD Server
CAD Server
Initial CAD file and parameters
Simulation Grid
CAD Server
CAD Grid
Fig. 1. System Structure
should be investigated by the optimization algorithm. After the job has been submitted, the user can monitor the progress of the optimization, and, after its completion, the final result is displayed. This integrates the optimization framework into the design workflow of the target users, the design engineers, and hence reduces the associated learning curve. This GUI has been implemented in Java as a standalone client. This decision was made to enable the client to access local data and use this data for uploads, which would have not been possible in applets. This approach, however, maintains the platform independence of the client enabling an easy deployment across a large number of platforms without porting efforts.
4
Technical Evaluation
We have implemented this resource sharing infrastructure in cooperation with ABB corporate research and have deployed it at their site for production use. It currently operates with two PC clusters, each eight nodes, and up to three CAD workstations. Within the compute clusters, the POLOPT environment [11] is used to execute the simulations, while a set of CAD workstations with Pro/Engineer [1] is deployed to generate the model files. First experience shows a positive feedback and high acceptance rate among the design engineers. The unified GUI effectively hides the complexity of the optimization from the user and provides a clean interface, significantly lowering the learning curve. In addition, by its implementation in Java, it can be used from arbitrary workstations, further simplifying its application and deployment. The availability of this unifying infrastructure provides the design engineers with a straightforward access to corporate-wide resources and enables the com-
CAD Grid: Corporate-Wide Resource Sharing for Parameter Studies
439
putation of complex optimization problems. This is illustrated in the following based on a sample model. Figure 2 shows eight design parameters of a transformer output lead shielding electrodes for which an optimal set with regard to the maximum field strength had to be found. The goal is the minimization of the overall space requirements by this component, while staying within a safe range in terms of possible flashovers. This example has been optimized using the optimization algorithms discussed above. The numerical changes of the eight design parameter during the optimization process can be seen in Table 1. x2 x5 x8
x4
x6
x7
Table 1. Change of design parameters during optimization
x3 x1
Parameter Initial Value Optimal Value x1 [mm] x2 [mm] x3 [mm] x4 [mm] x5 [mm] x6 [mm] x7 [mm] x8 [mm]
420.0 113.0 531.0 42.0 32.0 113.0 165.0 25.0
438.3 17.3 237.0 135.1 147.0 236.5 56.2 60.7
Fig. 2. Design parameters for the transformer
5
Business Perspective
The ability to compute these optimization processes is invaluable to companies like ABB. They allow a predictable assessment of properties of their products and enable significant design improvements without having to build expensive and time-consuming prototypes. This leads to more competitive products with respect to both price, due to reduced engineering cost and development time, and quality, due to highly optimized systems. The exact impact can hardly be quantified, but can be assumed to be substantial considering that most components in this product area (e.g., large scale transformer and switching units) are custom designs without the ability to mass produce and can be in price ranges well beyond millions of dollars.
6
Conclusions
The optimization of the geometry is an integral part of the design process of high voltage components. This is done iteratively by repeated simulations of the electric field and an adjustment of the geometric model based on the simulation results. Combined with optimization algorithms, which are capable of minimizing a chosen objective value for a given set of parameters, this can be used to automatically compute an optimal set of design parameters. We have presented a framework, which implements such an optimization process. It is designed to
440
E. Wheelhouse, C. Trinitis, and M. Schulz
efficiently leverage corporate wide resource and to allow an interleaving of several concurrent optimization requests. It thereby distinguishes between two different sets of resources — compute servers to run the electric field simulations and CAD servers to perform the model generation based on a given set of parameters — and ensures the efficient utilization of both. This framework has been implemented in a real industrial environment and is already in production use. It is highly accepted by the end users, the design engineers, due to its easy-to-learn interface and clean integration into their overall workflow. It has already successfully been deployed to optimize the design of high voltage components. In the example shown in this work, it has lead to significant optimizations in the geometric design and to a reduction of the electric field by over 25%.
References 1. Web Page – URL: http://www.ptc.com. 2. Z. Andjelic. POLOPT 5.3 User’s Guide. Internal document, Asea Brown Boveri Corporate Research, Heidelberg, 1999. 3. R. Bausinger and G. Kuhn. Die Boundary-Element Methode (In German). Expert Verlag, Ehingen, 1987. 4. C. Trinitis, H. Steinbigler, M. Spasojevic, P. Levin, and Z. Andjelic. Accelerated 3-D optimization of High Voltage Apparatus. Conference Proceedings of the 9th Int. Symposium on High Voltage Engineering, paper 8867, Graz, 1995. 5. L. Kotler D. Abramson, J. Giddy. High performace parametric modelling with nimrod/g:ller application for the global grid? International Parallel and Distributed Processing Symposium (IPDPS), May 2000. pp 520–528, Cancun, Mexico. 6. M.S. Eldred and W.E. Hart. Design and implementation of multilevel parallel optimization on the intel teraflops. Seventh AIAA/USAF/NASA/ISSMO Symposium on Multidisciplinary Analysis and Optimization, September 1998. pages 44-54, St. Louis, MO, AIAA-98-4707. 7. P. Guisset and N. Tzannetakis. Numerical methods for modeling and optimization of noise emission applications. ASME Symposium in Acoustics and Noise Control Software, November 1997. ASME International Mechanical Engineering Congress and Exposition, Dallas, TX. 8. R. Hooke and T.A. Jeeves. Direct Search Solution of numerical and statistical Problems. In Journal of Ass. of Comp., vol. 8, pages 212–229, 1961. 9. Message Passing Interface Forum (MPIF). MPI: A Message-Passing Interface Standard. Technical Report, University of Tennessee, Knoxville, June 1995. http://www.mpi-forum.org/. 10. J.A. Nelder and T. Mead. A simplex method for function minimization. In Computer Journal 7, pages 308–313, 1965. 11. C. Trinitis, M. Schulz, and W. Karl. A Comprehensive Electric Field Simulation Environment on Top of SCI. Lecture notes in Computer Science 2474, EuroPVM’2002, Springer Verlag, pp. 114–121, 2002. 12. G.N. Vanderplaaats. Numerical Optimization Techniques for Engineering Design. Mc-Gaw Hill Book Company, New York, 1994.
Cache Performance Optimizations for Parallel Lattice Boltzmann Codes Jens Wilke, Thomas Pohl, Markus Kowarschik, and Ulrich R¨ ude Lehrstuhl f¨ ur Systemsimulation (Informatik 10) Institut f¨ ur Informatik Friedrich–Alexander–Universit¨ at Erlangen–N¨ urnberg, Germany {Jens.Wilke,Thomas.Pohl,Markus.Kowarschik,Ulrich.Ruede}@cs.fau.de
Abstract. When designing and implementing highly efficient scientific applications for parallel computers such as clusters of workstations, it is inevitable to consider and to optimize the single–CPU performance of the codes. For this purpose, it is particularly important that the codes respect the hierarchical memory designs that computer architects employ in order to hide the effects of the growing gap between CPU performance and main memory speed. In this paper, we present techniques to enhance the single–CPU efficiency of lattice Boltzmann methods which are commonly used in computational fluid dynamics. We show various performance results to emphasize the effectiveness of our optimization techniques.
1
Introduction
In order to enhance the performance of any parallel scientific application, it is important to focus on two related optimization issues. Firstly, it is necessary to minimize the parallelization overhead itself. These efforts commonly target the choice of appropriate load balancing strategies as well as the minimization of communication overhead by hiding network latency and bandwidth. Secondly, it is necessary to exploit the individual parallel resources as efficiently as possible; e.g., by achieving as much performance as possible on each CPU in the parallel environment. This is especially true for distributed memory systems found in computer clusters based on off–the–shelf workstations communicating via fast networks. Our current research focuses on this second optimization issue. In order to mitigate the effects of the growing gap between theoretically available processor speed and main memory performance, today’s computer architectures are typically based on hierarchical memory designs, involving CPU registers, several levels of cache memories (caches), and main memory [10]. Remote main memory and external memory (e.g., hard disk drives) can be considered as the slowest components in any memory hierarchy. Fig. 1 illustrates the memory architecture of a current high performance workstation [12]. Efficient execution in terms of work units per second can only be obtained if the codes exploit the underlying memory design. This is particularly true for numerically intensive codes. Unfortunately, current compilers cannot perform H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 441–450, 2003. c Springer-Verlag Berlin Heidelberg 2003
442
J. Wilke et al.
ALU Registers
Capacity Bandwidth
Latency
L1D cache
16 KB
16 GB/s
1 cycle
L2 cache
256 KB
32 GB/s
5+ cycles
L3 cache
1.5/3 MB
32 GB/s
12+ cycles
Main memory
2 GB
6.4 GB/s >100 cycles
Ext. mem., remote mem.
Fig. 1. Memory architecture of a workstation based on an Intel Itanium2 CPU with three levels of on–chip cache [12].
highly sophisticated code transformations automatically. Much of this optimization effort is therefore left to the programmer [8,14]. Generally speaking, efficient parallelization and cache performance tuning can both be interpreted as data locality optimizations. The underlying idea is to keep the data to be processed as close as possible to the corresponding ALU. From this viewpoint, cache optimizations form an extension of classical parallelization efforts. Research has shown that the cache utilization of iterative algorithms for the numerical solution of linear systems can be improved significantly by applying suitable combinations of data layout optimizations and data access optimizations [3,6]. The idea behind these techniques is to enhance the spatial locality as well as the temporal locality of the code [11]. Similar work focuses on other algorithms of numerical linear algebra [16] and hardware–oriented FFT implementations [7]. An overview of cache optimization techniques for numerical algorithms can be found in [13]. Our current work concentrates on improving the cache utilization of a parallel implementation of the lattice Boltzmann method (LBM), which represents a particle–based approach towards the numerical simulation of problems in computational fluid dynamics (CFD) [5,17]. This paper is structured as follows. Section 2 contains a brief introduction to the LBM. Section 3 presents code transformation techniques to enhance the single–CPU performance of the LBM and introduces a compressed grid storage technique, which almost halves its data set size. The performance results of LBM implementations on various platforms are shown in Section 4. We conclude in Section 5.
2
The Lattice Boltzmann Method
It is important to mention up front that we only give a brief description of the LBM in this section, since the actual physics behind this approach are not essential for the application of our optimization techniques.
Cache Performance Optimizations for Parallel Lattice Boltzmann Codes
443
The usual approach towards solving CFD problems is based on the numerical solution of the governing partial differential equations, particularly the Navier– Stokes equations. The idea behind this approach is to discretize the computational domain using finite differences, finite elements or finite volumes, to derive algebraic systems of equations and to solve these systems numerically. In contrast, the LBM is a particle–oriented technique, which is based on a microscopic model of the moving fluid particles. Using x to denote the position in space, u to denote the particle velocity, and t to denote the time parameter, the so–called particle distribution function f (x, u, t) is discretized in space, velocity, and time. This results in the computational domain being regularly divided into cells (so–called lattice sites), where the current state of each lattice site is defined by an array of floating–point numbers that represent the distribution functions w.r.t. to the discrete directions of velocity. The LBM then works as follows. In each time step, the entire grid (lattice) is traversed, and the distribution function values at each site are updated according to the states of its neighboring sites in the grid at the previous discrete point in time. This update step consists of a stream operation, where the corresponding data from the neighboring sites are retrieved, and a collide operation, where the new distribution function values at the current site are computed according to a suitable model of the microscopic behavior of the fluid particles, preserving hydrodynamic quantities such as mass density, momentum density, and energy. Usually, this model is derived from the Boltzmann equation ∂f 1 + u, ∇f = f − f (0) , ∂t λ which describes the time–dependent behavior of the particle distribution function f . In this equation, f (0) denotes the equilibrium distribution function, λ is the relaxation time, ., . denotes the standard inner product, and ∇f denotes the gradient of f w.r.t. the spatial dimensions [17].
Fig. 2. Representation of the LBM updating a single lattice site by a stream operation (left) and a subsequent collide operation (right).
Fig. 2 shows the update of an individual lattice site in 2D. The grid on the left illustrates the source grid corresponding to the previous point in time, the
444
J. Wilke et al.
darker arrows represent the particle distribution function values being read. The grid on the right illustrates the destination grid corresponding to the current point in time, the dark arrows represent the new particle distribution function values; i.e., after the collide operation. Note that, in every time step, the lattice may be traversed in any order since the LBM only accesses data corresponding to the previous point in time in order to compute new distribution function values. From an abstract point of view, the structure of the LBM thus parallels the structure of an implementation of Jacobi’s method for the iterative solution of linear systems on structured meshes. The time loop in the LBM corresponds to the iteration loop in Jacobi’s method. In contrast to the lattice gas approach [17] where hexagonal grids are more common, we focus on orthogonal grids, since they are almost exclusively used for the LBM.
3 3.1
Optimization Techniques for Lattice Boltzmann Codes Fusing the Stream and the Collide Operation
A naive implementation of the LBM would perform two entire sweeps over the whole data set in every time step: one sweep for the stream operation, copying the distribution function values from each lattice site into its neighboring sites, and a subsequent sweep for the collide operation, calculating the new distribution function values at each site. A first step to improve performance is to combine the streaming and the collision step. The idea behind this so–called loop fusion technique is to enhance the temporal locality of the code and thus the utilization of the cache [1]. Instead of passing through the data set twice per time step, the fused version retrieves the required data from the neighboring cells and immediately calculates the new distribution function values at the current site, see again Fig. 2. Since this tuning step is both common and necessary for all subsequent transformations, we consider this fused version as our starting point for further cache optimizations. 3.2
Data Layout Optimizations
Accessing main memory is very costly compared to even the lowest cache level. Therefore, it is essential to choose a memory layout for the implementation of the LBM which allows the code to exploit the benefits of the hierarchical memory architecture. Since we need to maintain the grid data for any two successive points in time, our initial storage scheme is based on two arrays of records. Each of these records stores the distribution function values of an individual lattice site. Clustering the distribution function values is a reasonable approach since the smallest unit of data to be moved between main memory and cache is a cache block which typically contains several data items that are located adjacent in memory. In the following, we present two data layout transformations that aim at further enhancing the spatial locality of the LBM implementation; grid merging and grid compression.
Cache Performance Optimizations for Parallel Lattice Boltzmann Codes
Array t
Array t+1
445
Merged array
Fig. 3. The two arrays which store the grid data at time t and time t + 1, respectively, are merged into a single array.
Grid merging. During a fused stream–and–collide step it is necessary to load data from the grid (array) corresponding to time t, to calculate new distribution function values, and to store these results into the other grid corresponding to time t + 1. Due to our initial data layout (see above) the first two steps access data which are tightly clustered in memory. Storing the results, however, involves the access of memory locations that can be arbitrarily far away. In order to improve spatial locality, we introduce an interleaved data layout where the distribution function values of each individual site for two successive points in time are kept next to each other in memory. This transformation is commonly called array merging [11,13]. Fig. 3 illustrates the application of this technique for the 2D case. Grid compression. The idea behind this data layout is both to save memory and to increase spatial locality. We concentrate on the 2D case in this paper, while the extension of this technique to the 3D case is currently being implemented.
1.
2.
Fig. 4. Fused stream–and–collide step for the cell in the middle (left, 1 and 2), layout of the two overlayed grids after applying the grid compression technique (right).
In an implementation of the LBM in 2D, nearly half of the memory can be saved by exploiting the fact that only the data from the eight neighboring cells are required to calculate the new distribution function values at any regular site
446
J. Wilke et al.
of the grid1 . It is therefore possible to overlay the two grids for both points in time, introducing a diagonal shift of one row and one column of cells into the stream–and–collide operation. The direction of the shift then determines the update sequence in each time step since we may not yet overwrite those distribution function values that are still required henceforth. In Fig. 4, the light gray area contains the values for the current time t. The two pictures on the left illustrate the stream–and–collide operation for a single cell: the values for time t + 1 will be stored with a diagonal shift to the lower left. Therefore, the stream–and–collide sweep must also start with the lower left cell. Consequently, after one complete sweep, the new values (time t + 1) are shifted compared to the previous ones (time t). For the subsequent time step, the sweep must start in the upper right corner. The data for time t + 2 must then be stored with a shift to the upper right. After two successive sweeps, the memory locations of the distribution functions are the same as before. This alternating scheme is shown in the right picture of Fig. 4. 3.3
Data Access Optimizations
Data access optimizations change the order in which the data are referenced in the course of the computation, while respecting all data dependencies. Our access transformations for implementations of the LBM are based on the loop blocking (loop tiling) technique. The idea behind this general approach is to divide the iteration space of a loop or a loop nest into blocks and to perform as much computational work as possible on each individual block before moving on to the next block. If the size of the blocks is chosen according to the cache size, loop blocking can significantly enhance cache utilization and, as a consequence, yield significant performance speedups [1,8,13]. In the following, we present two blocking approaches in order to increase the temporal locality of the 2D LBM implementation. Both blocking techniques take advantage of the stencil memory access exhibited by the LBM. Because of this local operation, a grid site can be updated to time t + 1 as soon as the sites it depends on have been updated to time t. It is important to point out that each of these access transformations can be combined with either of the two layout transformations which we have introduced in Section 3.2. 1D blocking. Fig. 5 illustrates an example, where two successive time steps are blocked into a single pass through the grid. White cells have been updated to time t, light gray cells to time t + 1, and dark gray cells to time t + 2. It can be seen that in Grid 2, all data dependencies of the bottom row are fulfilled, and can therefore be updated to time t + 2, shown in Grid 3. This is performed repeatedly until the entire grid has been updated to time t + 2, see Grids 4 to 10. The downside to this method is that, even two rows may contain too much 1
For the sake of simplicity, we focus on regular sites and omit the description of how to treat boundary sites and obstacle sites separately.
Cache Performance Optimizations for Parallel Lattice Boltzmann Codes
447
data to fit into cache if the grid is too large. In this case, no performance gain will be observed.
1.
6.
2.
3.
4.
5.
7.
8.
9.
10.
Fig. 5. 1D blocking technique to enhance temporal locality.
2D blocking. Fig. 6 illustrates an example, in which a 4×4 block of cells is employed. This means that four successive time steps are performed during a single pass through the grid. Since the data contained in the 2D block is independent of the grid size, it will always (if the size is chosen appropriately) fit into the highest possible cache level, regardless of the grid size. In Fig. 6, Grids 1 to 4 demonstrate the handling of one 4×4 block, and Grids 5 to 8 a second 4×4 block. The block which can be processed moves diagonally down and left in order to avoid violating data dependencies. Obviously, special handling is required for those sites near grid edges which cannot form a complete 4×4 block.
1.
2.
3.
4.
5.
6.
7.
8.
Fig. 6. 2D blocking technique to enhance temporal locality.
448
4
J. Wilke et al.
Performance Results
In order to test and benchmark the various implementations of the LBM, a well known problem in fluid dynamics known as the lid–driven cavity has been used. It consists of a closed box, where the top of the box, the lid, is continually dragged across the fluid in the same direction. The fluid eventually forms a circular flow around the center of the box [9].
6.5
11.0 10.0
Two Grids Grid Compression GC with 2 Rows Blocked GC with 8 Rows Blocked GC with 16x16 Sites Blocked
6.0
9.0
5.5
MLSUD/s
MLSUD/s
8.0 7.0 6.0 5.0
5.0 4.5
4.0
4.0
3.0 2.0 2 0
2
2
2
2
2
2
2
2
2
2
100 200 300 400 500 600 700 800 900 1000 grid size
3.5 2 0
2
2
2
2
2
2
2
2
2
2
100 200 300 400 500 600 700 800 900 1000 grid size
Fig. 7. Performance in millions of lattice site updates per second (MLSUD/s) on machines based on an AMD Athlon XP 2400+ (left) and on an Intel Itanium2 (right), respectively, for various grid sizes.
Performance. Fig. 7 demonstrates the performance of five different implementations of the LBM in ANSI C++ on two different machines2 for various grid sizes. Note that problem size n means that the grid contains n2 cells. The simple implementation involving a source and destination grid is the slowest, the implementation using the layout based on grid compression is slightly faster. Two implementations of 1D blocking are shown, one with two time steps blocked, and one with eight. Initially, they show a significant increase in performance. For large grids, however, performance worsens since the required data cannot fit into even the lowest level of cache. This effect is more dramatic on the AMD processor since it has only 256 kB of L2 and no L3 cache, whereas the Intel CPU even has 1.5 MB of L3 cache on–chip. Finally, the 2D blocked implementation shows a significant increase in performance for all grid sizes. It should be noted that each implementation based on the grid merging layout performs worse than its counterpart based on grid compression. Therefore, no performance results for the grid merging technique are shown. We have obtained similar performance gains on several further platforms. For example, our results include speedup factors of 2–3 on machines based on DEC 2
We use an AMD Athlon XP 2400+ based PC (2 GHz) [2], Linux, gcc 3.2.1, as well as an Intel Itanium2 based HP zx6000 [12], Linux, Intel ecc V7.0. Aggressive compiler optimizations have been enabled in all experiments.
Cache Performance Optimizations for Parallel Lattice Boltzmann Codes
449
Alpha 21164 and DEC Alpha 21264 CPUs. Both of them particularly benefit from large off–chip caches of 4 MB.
3.0
L2 cache misses / lattice site update
L1 cache misses / lattice site update
5.5 5.0 4.5 4.0 3.5 3.0 2.5 2.0 1.5 1.0 0.5 0.0 2 0
2
2
2
2
2
2
2
2
2
2
100 200 300 400 500 600 700 800 900 1000 grid size
2.5
Grid Compression GC with 2 Rows Blocked GC with 8 Rows Blocked GC with 16x16 Sites Blocked
2.0 1.5 1.0 0.5 0.0 2 0
2
2
2
2
2
2
2
2
2
2
100 200 300 400 500 600 700 800 900 1000 grid size
Fig. 8. Cache behavior of the AMD Athlon XP measured with PAPI.
Cache Behavior. Fig. 8 demonstrates the behavior of the L1 and L2 cache of the AMD Athlon XP for the different implementations of the LBM. These results have been obtained by using the profiling tool PAPI [4]. When compared to Fig. 7 it can be seen that there is a strong correlation between the number of cache misses and the performance of the code. The correlation between the performance drop of the two 1D blocked implementations and their respective rise in L2 cache misses is especially dramatic. Additionally, Fig. 8 reveals the cause of the severe performance drop exhibited by the 2D blocked implementation at a grid size of 8002 . The high number of cache misses are conflict misses which are caused when large amounts data in memory are mapped to only a small number of cache lines [15].
5
Conclusions and Future Work
Due to the still widening gap between CPU and memory speed, hierarchical memory architectures will continue to be a promising optimization target. The CPU manufacturers have already announced new generations of CPUs with several megabytes of on–chip cache in order to hide the slow access to main memory. We have demonstrated the importance of considering the single–CPU performance before using parallel computing methods in the framework of a CFD code based on the LBM. By exploiting the benefits of hierarchical memory architectures of current CPUs, we were have obtained factors of 2–3 in performance on various machines. Unfortunately, the parallelization of cache–optimized codes is commonly tedious and error–prone due to their implementation complexity. We are currently extending our techniques to the 3D case. From our experience in cache performance optimization of iterative linear solvers, we expect that appropriate data layout transformations and data access optimizations, in particular loop blocking, can be applied to improve the performance.
450
J. Wilke et al.
Acknowledgments. We wish to thank the members of the High Performance Computing Group at the Computing Center in Erlangen (RRZE) for their support.
References 1. R. Allen and K. Kennedy. Optimizing Compilers for Modern Architectures. Morgan Kaufmann Publishers, San Francisco, CA, USA, 2001. 2. AMD Corporation. AMD Athlon XP Processor 8 Data Sheet, 2002. Publication #25175 Rev. F. 3. F. Bassetti, K. Davis, and D. Quinlan. Temporal Locality Optimizations for Stencil Operations within Parallel Object–Oriented Scientific Frameworks on Cache– Based Architectures. In Proc. of the Int. Conference on Parallel and Distributed Computing and Systems, pages 145–153, Las Vegas, NV, USA, 1998. 4. S. Browne, J. Dongarra, N. Garner, G. Ho, and P. Mucci. A Portable Programming Interface for Performance Evaluation on Modern Processors. Int. Journal of High Performance Computing Applications, 14(3):189–204, 2000. 5. S. Chen and G.D. Doolen. Lattice Boltzmann Method for Fluid Flow. Annual Reviews of Fluid Mechanics, 30:329–364, 1998. 6. C.C. Douglas, J. Hu, M. Kowarschik, U. R¨ ude, and C. Weiß. Cache Optimization for Structured and Unstructured Grid Multigrid. Electronic Transactions on Numerical Analysis, 10:21–40, 2000. 7. M. Frigo and S.G. Johnson. FFTW: An Adaptive Software Architecture for the FFT. In Proc. of the Int. Conference on Acoustics, Speech, and Signal Processing, volume 3, pages 1381–1384, Seattle, WA, USA, 1998. 8. S. Goedecker and A. Hoisie. Performance Optimization of Numerically Intensive Codes. SIAM, 2001. 9. M. Griebel, T. Dornseifer, and T. Neunhoeffer. Numerical Simulation in Fluid Dynamics. SIAM, 1998. 10. J. Handy. The Cache Memory Book. Academic Press, second edition, 1998. 11. J.L. Hennessy and D.A. Patterson. Computer Architecture: A Quantitative Approach. Morgan Kaufmann Publisher, Inc., San Francisco, CA, USA, second edition, 1996. 12. Intel Corporation. Intel Itanium2 Processor Reference Manual, 2002. Document Number: 251110–001. 13. M. Kowarschik and C. Weiß. An Overview of Cache Optimization Techniques and Cache–Aware Numerical Algorithms. In Algorithms for Memory Hierarchies, volume 2625 of LNCS. Springer, 2003. 14. D. Loshin. Efficient Memory Programming. McGraw–Hill, 1998. 15. G. Rivera and C.-W. Tseng. Data Transformations for Eliminating Conflict Misses. In Proc. of the ACM SIGPLAN Conference on Programming Language Design and Implementation, Montreal, Canada, 1998. 16. R.C. Whaley and J. Dongarra. Automatically Tuned Linear Algebra Software. In Proc. of the ACM/IEEE Supercomputing Conference, Orlando, FL, USA, 1998. 17. D.A. Wolf-Gladrow. Lattice–Gas Cellular Automata and Lattice Boltzmann Models. Springer, 2000.
Effectiveness of Parallelizing the ILOG-CPLEX Mixed Integer Optimizer in the PUBB2 Framework Yuji Shinano1 , Tetsuya Fujie2 , and Yuusuke Kounoike1 1
2
Department of Computer, Information and Communication Sciences, Tokyo University of Agriculture and Technology, 2-24-16, Naka-cho, Koganei-shi, Tokyo 184-8588, Japan
[email protected],
[email protected] Department of Management Science, Kobe University of Commerce, 8-2-1, Gakuen-nishimachi, Nishi-ku, Kobe 651-2197, Japan
[email protected] Abstract. In this paper, we introduce a new method of parallelizing a MIP (Mixed Integer Programming) solver. This method is different from a standard implementation that constructs a parallel branch-and-cut algorithm from scratch (except using an LP solver). The MIP solver we use is ILOG-CPLEX MIP Optimizer (Version 8.0), which is one of the most efficient implementations of branch-and-cut algorithms. The parallelization of the solver is performed by using the software tool PUBB2 developed by the authors. We report a part of our computational experience using up to 24 processors. In addition, we point out some problems that should be resolved for a more efficient parallelization.
1
Introduction
A mixed integer programming problem (MIP) is a linear programming problem (LP) with a restriction that some or all of the decision variables must be integer. MIP has a wide variety of applications in scheduling, planning and other practical problems. MIP has been considered as one of the most difficult discrete optimization problems. However, recent remarkable advancements of branchand-cut (B&C) algorithms, especially for traveling salesman problems[1] and other optimization problems, stimulate research interest on MIPs. The B&C algorithm is a branch-and-bound (B&B) algorithm in which each subproblem is evaluated by the cutting plane algorithm. We may point out several factors in the recent advancements of the B&C algorithm. The first is a development of fast and robust LP solvers. Fast LP solvers are important since most of computational time of the B&C algorithm is devoted to solving LP relaxation problems, and robust LP solvers enable us to implement involved strategies in the B&C algorithm. The second is that many classes of cutting planes are incorporated
This work was partially supported by MEXT in Japan through Grants-in-Aid (13680511).
H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 451–460, 2003. c Springer-Verlag Berlin Heidelberg 2003
452
Y. Shinano, T. Fujie, and Y. Kounoike
into the B&C algorithms as a consequence of the fast and robust LP solvers. Until recently, Gomory cuts, proposed in the 1950’s and the 1960’s, have been considered less effective in practice. However, the recent advancement of the LP solvers make them work in the B&C algorithm. The last is that many effective strategies are incorporated in many parts of the B&C algorithm. There are several parallel implementations of the B&C algorithm in the literature[2,3, etc.]. Recently, general frameworks of the parallel B&C algorithm have been developed, such as SYMPHONY[8], PICO[4] and COIN/BCP[9]. In this paper, we demonstrate a parallelization of a MIP solver itself, which is different from a construction of a parallel B&C algorithm from scratch (except using an LP solver). The MIP solver used in the study is ILOG-CPLEX MIP Optimizer Version 8.0[5], and is parallelized by using the software tool PUBB2 developed by the authors[11]. Motivation of this work comes from the following facts : (i) PUBB2 clearly separates the user specific parts from the common procedures of B&B algorithms, which means that a sequential implementation is enough for the users to execute both sequential and parallel B&B algorithms. PUBB2 further separates the computing platform, and it enables a parallelization, for example, via the computational GRID. The parallelization of a MIP solver is a part of our PUBB2 project, considering a MIP solver as the user specific implementation. (ii) More importantly, MIP solvers have been greatly improved so that middle and even large-scale problem instances become tractable. Hence, parallelizing MIP solvers can be very significant. In addition, the ILOG-CPLEX MIP Optimizer provides callback functions and classes, which enables us to parallelize such an efficient MIP solver in the PUBB2 framework. The purpose of the paper is to examine how practical our parallelization is and to find problems that should be resolved in order to achieve a more efficient parallelization. The remainder of the paper is organized as follows. Section 2 presents a brief review of the B&C algorithm. Section 3 gives an outline of PUBB2 and describes the parallelization of the ILOG-CPLEX MIP Optimizer using PUBB2. Computational results are reported in Section 4. Finally, we describe concluding remarks in Section 5.
2
Branch-and-Cut Algorithm for MIPs
In this section, we briefly review the B&C algorithm. See [6,7,12] for more comprehensive treatment of the algorithm. Let us consider the MIP of the following form : (MIP) Minimize cT x subject to Ax ≤ b, xj : integer (j = 1, . . . , p), where c is a n-dimensional vector, b is an m-dimensional vector, A is an m × n matrix and p ≤ n. If p = n then (MIP) is called a pure integer programming problem. A 0-1 MIP problem is referred to as (MIP) imposed on the additional
Effectiveness of Parallelizing the ILOG-CPLEX Mixed Integer Optimizer
453
constraints 0 ≤ xj ≤ 1 (j = 1, . . . , p). An LP relaxation of (MIP) is obtained by dropping the integrality constraint, i.e., (LP) Minimize cT x subject to Ax ≤ b. In a standard (LP-based) B&B algorithm, each subproblem is evaluated by solving its LP relaxation problem. A cutting plane algorithm is an another approach to the MIP. Let x be an optimal solution of (LP). Then, if an inequality aT x ≤ b were found, which does not delete any feasible solutions of (MIP) but delete x (i.e., aT x > b), then the LP relaxation (LP) can be tightened by adding that inequality as an additional constraint. Such an inequality is called a cut, and the cutting plane algorithm iteratively add cuts to (LP) and then solve the updated LP relaxation problem. A B&C algorithm is a B&B algorithm in which each subproblem is evaluated by the cutting plane algorithm. Cuts are classified into global and local. A cut that is generated by a subproblem evaluation is called global if it does not delete any feasible solutions of (MIP), and local otherwise. Global cuts are stored in a cut pool, and they are used in subsequent subproblem evaluations. In summary, a prototype of the B&C algorithm is described as in Figure 1, which leaves some implementation flexibility. Corresponding to items (a),. . . ,(e) in Figure 1, the following strategies are particularly important in implementing an efficient B&C algorithm : (a) Preprocessing. Before performing the B&C or the B&B algorithm, preprocessing may be applied, in which redundant variables and constraints are deleted, lower and upper bounds of variables are tightened, coefficients of the constraints are modified to tighten the LP relaxation problem, and so on. (b) Node selection. Node selection is a common topic of the B&B algorithms. In PUBB2, the depth-first search, the best bound search, and the search based on a user-defined priority are available. (c) Cutting planes. There are several cuts that can be applied to MIPs such as the mixed integer Gomory cut for general MIPs, the fractional Gomory cut for pure IPs, the lift-and-project cut for 0-1 MIPs, etc. In addition, if the constraints of the MIP contain a structured constraint, then cuts valid for the constraint is available. For example, the (lifted) cover inequality for the knapsack constraints and the clique inequalities for the set packing constraints are known. (d) Heuristics. In a standard B&C or B&B algorithm for the MIP, an improved solution is found when an optimal solution of an LP relaxation problem is feasible to the MIP. In addition to this, recent solvers involve heuristics based on LPs. (e) Variable selection. Let x be an optimal solution of an LP relaxation problem of a subproblem S. Then branching is done by finding a variable with a fractional value xj and creating two subproblems which are obtained from S by adding the new inequalities xj ≤ xj and xj ≥ xj , respectively.
454
Y. Shinano, T. Fujie, and Y. Kounoike
Branch-and-Cut Algorithm; begin {Initialization Phase: See an explanation of the class InitData} (a) Apply preprocessing to the problem (MIP); z ∗ := ∞; Put the root problem into the subproblem pool; Let the cut pool be empty; {Search Phase: See an explanation of the class SolverBase} while the subproblem pool is nonempty do begin (b) Choose a subproblem S from the subproblem pool; Delete S from the subproblem pool; Add (globally valid) inequalities in the cut pool to S; {Evaluation Phase} (c) Execute a cutting plane algorithm; (d) if improved solution has been found then Update x∗ and z ∗ ; (e) if new subproblems have been generated then Put them into the subproblem pool end; return x∗ end.
Fig. 1. A prototype of the branch-and-cut algorithm
There are several selection rules including the maximum infeasibility rule, the pseudo-cost rule and the strong branching rule. See [5] for the strategies incorporated in the ILOG-CPLEX MIP Optimizer Version 8.0.
3
Parallelization in the PUBB2 Framework
PUBB2[11] is a redesigned version of the software tool PUBB (Parallelization Utility for Branch-and-Bound algorithms, see [10]). In particular, the layer structure (see Figure 2) is improved so that the Provider layer for computing platforms is separated from the core layer. PUBB2 primarily consists of three layers : the User, the PUBB2 core, and the Provider layers. A B&B algorithm is completed by establishing connections between the User and the PUBB2 core layers and between the Provider and the PUBB2 core layers. PUBB2 is written in C++ and the connections are achieved by implementing derived classes of the abstract classes in the PUBB2 core layers. For a detailed description of PUBB2, see [11]. Our parallel implementation of the B&C algorithm is performed by embedding the ILOG-CPLEX MIP Optimizer into the User layer. For the interaction between the User and the PUBB2 core layers, four abstract classes are provided in the PUBB2 core building block :
Effectiveness of Parallelizing the ILOG-CPLEX Mixed Integer Optimizer
455
%
& ,
*
#$
)
&'
(
,
&'
)
)
( *
,
)
!"
( *+
Fig. 2. Layers in the PUBB2 Architecture (quoted from [11])
InitData: An abstraction of data for an initialization of a B&B algorithm. The Initialization Phase in Figure 1 is described in a class derived from InitData. SubproblemBase: An abstraction of a subproblem representation. SolutionBase: An abstraction of a solution representation. SolverBase: An abstraction of a search phase of the B&B algorithm. The Search Phase in Figure 1 is described in a class derived from SolverBase. In this paper, an implementation of a new and quite simple parallelization strategy is used, named MPMS (Master-Pool Master-Slave), which was proposed by the authors [11] and was implemented in the PUBB2 framework. MPMS concerns with an implementation of the Provider layer. The following two abstract classes are provided in the PUBB2 core building block for interaction between the Provider and the PUBB2 core layers : ProblemManager: An abstraction of the problem-solving manager. The B&B parallelization algorithm is encapsulated in this class, represented by the instances of the classes derived from this core class. These instances may be mapped over several tasks, processes or threads in parallelizations. Each task/process may have its own pool, which is a collection of unprocessed subproblems. IOManager: An abstract of I/O and encapsulated I/O operations on a specific computing platform. Figure 3 shows the construction of master and slave tasks, the pool allocation, the DerivedSolver linkage, the task mapping and the interaction between these tasks. The load-balancing strategy is basically the same as a hierarchical Master-Slave load balancing strategy. As long as a local pool in the MasterProblemManager is not empty and an idle DerivedSolver instance exists, a subproblem is assigned to that instance. However, in this paper, the parallelization strategy of MPMS
456
Y. Shinano, T. Fujie, and Y. Kounoike
!"
!"
!"
Fig. 3. Pool allocation, DerivedSolver linkage and task mapping
is not so important, because load balancing is controlled from the User layer. PUBB2 has such a mechanism. In the instance of the DerivedSolver, a subproblem is received from the ProblemManager and is solved by the ILOG-CPLEX MIP Optimizer. The Optimizer itself performs the B&C algorithm and it stores subproblems in its internal pool. Since the internal pool is maintained and updated efficiently, we are better off performing the algorithm in the Optimizer as long as no idle DerivedSolver instance exists. Load balancing is considered only when an idle solver exists in the system (existence of idle solvers can be recognized through the state of the ProblemManager) and a subproblem is transferred between DerivedSolvers. This transfer is made possible by using public functions void putSubproblem(SubproblemBase *) and void push() of ProblemManager. The former stores a subproblem to a local pool of the ProblemManager and the latter forces the ProblemManager to perform the transfer. These procedures are written in the callbacks of the ILOG-CPLEX MIP Optimizer. Finally, we describe some remarks concerning our specific implementation. 1. When tasks are loaded, (LP) is solved and its optimal basis is stored. The basis is used when a task receives a new “root” problem from another task, since it may help to reduce the time to solve an LP relaxation problem of the “root” problem. 2. Contrary to a standard Master-Slave, the master task executes the Search Phase too. Then, only when a local pool of the MasterProblemManager is empty, one of the two newly generated subproblems at the branching stage ((e) in Figure 1) is outputted to the local pool. 3. In order to avoid wasteful transfers, every instance of the DerivedSolver is forced not to output subproblems to the ProblemManager when [# of subproblems in the internal pool] × [average computing time per subproblem] < [computing time for an LP relaxation problem of the received “new” subproblem] + [average computing time per subproblem] × [run-time parame-
Effectiveness of Parallelizing the ILOG-CPLEX Mixed Integer Optimizer
457
Table 1. Problem statistics name constraints qiu 1192 swath2 884 seymour1 4944 markshare2 1 7 mod011 4480 fast0507 507
variables total 0-1 integer general integer 840 48 0 6805 2406 0 1372 451 0 74 54 0 10958 96 0 63009 63009 0
ter] is satisfied. For the results presented in the next section, this parameter was set equal to 10.
4
Computational Experience
A PC cluster of 16 PCs was used, each of which contains two Pentium III processors with 1GHz processing speed, 1Gbyte of memory and an Intel Pro/1000 MT LAN desktop adapter. All 16 PCs are connected to a NETGEAR GS 516T Gigabit Switch. PVM3.4.3 was used to implement Provider layer in Figure 2. H. Mittelmann’s webpage1 was referenced for problem instances including those from the MIPLIB library. At first, all the problem instances were tested by a default (sequential) MIP solver, and the moderately hard instances were selected for our parallel runs. A part of the results are reported here. In Table 1, statistics of the problem instances are shown. qiu, swath2, seymour1, markshare2 1 and mod011 are 0-1 MIP problems and fast0507 is a pure 0-1 integer programming problem arising from the set cover problem. Computational results are summarized in Table 2. In the table, time denotes the running time in seconds, speedup the ratio of the sequential run time to the corresponding parallel run time, opt. time the time it takes for an upper bound to reach an optimal solution value, updates the number that an incumbent solution (x∗ in Figure 1) is updated and nodes the total number of subproblems generated during the run. Each value in the table is an average of 5 runs. We set the relative mipgap tolerance (IloCplex::EpGap) to 1.0−5 and the absolute mipgap tolerance (IloCplex::EpAGap) to 1.0−6 . We first remark that preprocessing worked effectively in general. For most of the problem instances not listed in Table 1, preprocessing made the running time quite faster. Preprocessing also worked effectively for our parallel runs. As the table shows, satisfactory speedups cannot be obtained except a few problem instances. One major reason may be explained by an observation that speedup depends on opt. time. To explore an another reason for that, upper and lower bounds were drawn as functions of time (Figure 4). The problem instance is swath2 and the preprocessing is not applied. The running time for the parallel run of 4 tasks is almost the same as that of the sequential run : Comparing Figures 4(a) and 4(b), it may be seen that only one of the 4 tasks essentially 1
ftp://plato.asu.edu/pub/milpc.txt
458
Y. Shinano, T. Fujie, and Y. Kounoike
works. Vertical lines in Figure 4(b) indicate that the transferred “root” problem is solved with few branchings. Hence, many vertical lines reduce the effectiveness of parallelization. For the same reason, it can be seen that the parallel run of 8 tasks is executed effectively and, once again, the running time for the parallel run of 16 tasks is almost the same as that for the parallel run of 8 tasks. Therefore, it may be concluded that a variable selection rule ((e) in Figure 1) for parallel B&C algorithms is an important issue as well as a node selection rule which has been widely discussed for parallel B&B algorithms. Also, the frequency of the cut generation in subproblem evaluations is an issue to be reconsidered for parallel runs. Although reducing the frequency of the cutting plane phase will increase the number of subproblems, it may nevertheless enable efficient parallelizations. It must also be noted that the global information that the MIP solver maintains is lost in the parallelization. For example, as already described, global cuts are unavailable and information on psudo-costs is lost. Hence, some additional implementations of global cuts or pseudo-costs may be necessary in order to improve the performance of parallelizing MIP solvers.
420
objective function value
objective solution value
420
400
380
360
340
400
380
360
340 0
500 1000 1500 2000 2500 3000 3500 time (sec.)
0
500
(a) sequential
2500
(b) 4 tasks 420
objective function value
420
objective function value
1000 1500 2000 time (sec.)
400
380
360
340
400
380
360
340 0
100
200
300 400 time (sec.)
(c) 8 tasks
500
600
0
100
200
300 400 time (sec.)
500
600
(d) 16 tasks
Fig. 4. Upper and lower bounds for the swath2 instance (executed without preprocessing)
Effectiveness of Parallelizing the ILOG-CPLEX Mixed Integer Optimizer
459
Table 2. Computational Results ((p) means the preprocessing is applied.) name qiu (p)
qiu
swath2 (p)
swath2
seymour1 (p)
seymour1
markshare2 1 (p)
markshare2 1
mod011 (p)
mod011
fast0507 (p)
fast0507
time speedup opt. time updates nodes time speedup opt. time updates nodes time speedup opt. time updates nodes time speedup opt. time updates nodes time speedup opt. time updates nodes time speedup opt. time updates nodes time speedup opt. time updates nodes time speedup opt. time updates nodes time speedup opt. time updates nodes time speedup opt. time updates nodes time speedup opt. time updates nodes time speedup opt. time updates nodes
sequential parallel runs (# of tasks) run 2 4 8 16 24 1412.74 1521.86 916.83 633.12 510.93 1193.95 − 0.93 1.54 2.23 2.77 1.18 833.61 400.54 73.35 50.49 34.88 42.36 9.00 11.00 12.60 17.40 28.00 46.80 16004.00 25655.80 27282.20 31102.00 34430.40 64790.80 1211.84 1026.47 1044.98 824.36 57.69 31.33 − 1.18 1.16 1.47 21.01 38.68 8.69 6.80 6.83 6.86 6.90 6.84 2.00 2.00 2.00 2.00 2.00 2.00 12278.00 9101.00 9102.20 7281.80 292.20 48.00 5092.63 2905.16 1777.79 1444.39 1363.40 1397.26 − 1.75 2.86 3.53 3.74 3.64 4885.59 1643.92 310.47 221.99 143.40 144.70 19.00 21.40 15.60 26.00 31.60 54.00 258343.00 190872.40 111880.80 102903.60 91657.40 106970.40 2795.99 4781.72 2622.94 705.03 729.78 804.44 − 0.58 1.07 3.97 3.83 3.48 2584.40 4304.44 2432.69 623.50 658.06 739.37 9.00 13.40 25.40 14.40 34.60 78.60 67182.00 136040.40 92822.60 45171.80 52419.60 58211.00 8605.53 7609.23 4778.17 5517.70 4998.62 3981.05 − 1.13 1.80 1.56 1.72 2.16 2291.42 1430.03 1702.87 3066.39 361.95 1030.97 10.00 11.00 15.80 29.00 36.40 76.20 5757.00 9324.80 9324.00 11976.80 15278.20 14306.40 8501.69 6208.54 4188.94 3669.58 3213.46 3045.53 − 1.37 2.03 2.32 2.65 2.79 7854.38 2726.51 2806.45 2650.63 880.57 994.43 10.00 13.00 15.20 29.60 34.40 74.40 6127.00 6659.00 7963.80 8112.00 8670.20 9099.00 1982.47 2188.45 634.29 229.93 86.76 167.60 − 0.91 3.13 8.62 22.85 11.83 1982.20 2186.31 633.10 229.62 86.58 167.37 23.00 19.60 18.40 21.60 49.60 42.20 5848476.00 11975374.20 7107435.60 4739941.60 3815132.80 9556764.60 7954.91 1103.98 386.64 424.74 449.80 142.66 − 7.21 20.57 18.73 17.69 55.76 7940.82 1101.67 386.06 423.55 449.07 142.26 20.00 17.00 18.80 24.60 33.80 49.20 22370838.00 5792448.80 4058190.40 8719984.00 17794572.40 8639924.00 4886.01 3332.80 2143.70 1510.18 1279.63 1896.96 − 1.47 2.28 3.24 3.82 2.58 4115.98 2696.17 1812.10 1171.62 209.87 238.16 13.00 12.00 13.00 22.60 26.40 49.60 12806.00 15956.20 17960.60 18501.00 18842.00 22307.80 17314.76 10882.08 8076.57 5717.72 5571.91 5274.62 − 1.59 2.14 3.03 3.11 3.28 12031.80 7071.45 2573.09 2648.01 2536.73 2689.66 9.00 8.00 8.00 16.20 31.20 41.60 14398.00 15950.60 17183.80 18351.60 22268.00 22817.20 23108.70 24590.12 8079.49 7520.61 4904.10 5098.77 − 0.94 2.86 3.07 4.71 4.53 18488.30 22237.51 5242.34 4958.72 2331.63 2396.44 8.00 8.00 10.20 11.80 21.00 30.80 10938.00 19669.00 9418.80 14165.40 10423.60 15301.00 28592.92 17249.44 12339.97 11786.31 6174.30 5202.93 − 1.66 2.32 2.43 4.63 5.50 25042.98 12740.87 8752.64 5814.15 2420.45 2360.70 9.00 10.00 10.00 12.60 19.60 32.60 12685.00 12284.00 12775.80 17994.80 11224.20 13379.40
460
5
Y. Shinano, T. Fujie, and Y. Kounoike
Concluding Remarks
In this paper, a parallelization of the ILOG-CPLEX MIP Optimizer (Version 8.0) in the PUBB2 framework was demonstrated. It was observed that, in order to achieve an efficient parallelization, additional information, which is maintained by the MIP solver and is unavailable via the callback functions and classes, might be necessary. On the other hand, it must be also pointed out that, due to the new parallelization scheme of the B&C algorithm, the load balancing strategy, as well as other strategies of the parallel B&C algorithm, must be studied in more detail. For example, as far as load balancing is concerned, it would be interesting to compare the strategy proposed in this paper to a strategy based on the status of each subproblem in the internal pool, e.g. lower bounds, etc. (see Figure 3), which is currently unavailable via the callback functions and classes.
References 1. Applegate, D., Bixby, R., Chv´ atal, V. and Cook, W.: On the Solution of Traveling Salesman Problems. Doc. Math. J. DMV Extra Volume ICM III (1998) 645–656 2. Bixby, R. E., Cook, W., Cox, A., Lee, E. K.: Computational Experience with Parallel Mixed Integer Programming in a Distributed Environment. Ann. Oper. Res. 90 (1999) 19–43 3. Eckstein, J.: Parallel Branch-and-Bound Algorithms for General Mixed Integer Programming on the CM-5. SIAM J. Optim. 4 (1998) 794–814 4. Eckstein,J., Phillps, C.A., Hart, W.E.: PICO: An Object-Oriented Framework for Parallel Branch and Bound. RUTCOR Research Report 40-2000, Rutgers University (2000) 5. ILOG CPLEX 8.0 Reference Manual, ILOG (2002) 6. Martin, A.: General Mixed Integer Programming: Computational Issues for Branch-and-Cut Algorithms. In: J¨ unger, M., Naddef, D. (eds.) Computational Combinatorial Optimization. Lecture Notes in Computer Science, Vol. 2241. Springer-Verlag, Berlin Heidelberg New York (2001) 7. Nemhauser, G.L., Wolsey, L.A.: Integer Programming and Combinatorial Optimization: John Wiley & Sons, New York (1988) 8. Ralphs, T. K., Lad´ anyi, L., Es¨ o, M.: SYMPHONY 3.0 User’s Manual. available at http://www.branchandcut.org/SYMPHONY (2001) 9. Ralphs, T.K., Lad´ anyi, L.: COIN/BCP User’s Manual. available at http://www-124.ibm.com/developerworks/opensource/coin/documentation. html (2001) 10. Shinano, Y., Higaki, M., Hirabayashi, R.: A Generalized Utility for Parallel Branch and Bound Algorithms. Proceedings of the Seventh IEEE Symposium of Parallel and Distributed Processing (1995) 392–401 11. Shinano, Y., Kounoike, Y., Fujie, T.: PUBB2: A Redesigned Object-Oriented Software Tool for Implementing Parallel and Distributed Branch-and-Bound Algorithms. Working Paper, Department of Computer, Information and Communication Sciences, Tokyo University of Agriculture and Technology (2001) 12. Wolsey, L.A.: Integer Programming: John Wiley & Sons, New York (1998)
Improving Performance of Hypermatrix Cholesky Factorization Jos´e R. Herrero and Juan J. Navarro Computer Architecture Department, Universitat Polit`ecnica de Catalunya, Jordi Girona 1–3, M` odul D6, E-08034 Barcelona, (Spain) {josepr,juanjo}@ac.upc.es
Abstract. This paper shows how a sparse hypermatrix Cholesky factorization can be improved. This is accomplished by means of efficient codes which operate on very small dense matrices. Different matrix sizes or target platforms may require different codes to obtain good performance. We write a set of codes for each matrix operation using different loop orders and unroll factors. Then, for each matrix size, we automatically compile each code fixing matrix leading dimensions and loop sizes, run the resulting executable and keep its Mflops. The best combination is then used to produce the object introduced in a library. Thus, a routine for each desired matrix size is available from the library. The large overhead incurred by the hypermatrix Cholesky factorization of sparse matrices can therefore be lessened by reducing the block size when those routines are used. Using the routines, e.g. matrix multiplication, in our small matrix library produced important speed-ups in our sparse Cholesky code.
1
Introduction
The Cholesky factorization of a sparse matrix is an important operation in the numerical algorithms field. This paper presents our work on the optimization of the sequential algorithm when a hypermatrix data structure is used. 1.1
Hypermatrix Representation of a Sparse Matrix
Sparse matrices are mostly composed by zeros but often have small dense blocks which have traditionally been exploited in order to improve performance [1]. Our application uses a data structure based on a hypermatrix (HM) scheme [2]. The matrix is partitioned recursively into blocks of different sizes. The HM structure consists of several (N) levels of submatrices. The top N-1 levels hold pointer matrices which point to the next lower level submatrices. Only the last (bottom) level holds data matrices. Data matrices are stored as dense matrices and operated as such. Null pointers in pointer matrices indicate that the
This work was supported by the Ministerio de Ciencia y Tecnolog´ıa of Spain and the EU FEDER funds (TIC2001-0995-C02-01)
H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 461–469, 2003. c Springer-Verlag Berlin Heidelberg 2003
462
J.R. Herrero and J.J. Navarro
corresponding subblock does not have any non-zero elements and is therefore unnecessary, both for storage and computation. Figure 1 shows a sparse matrix and its corresponding hypermatrix with 2 levels of pointers. The main potential advantages of a HM structure w.r.t. 1D data structures, like the Compact Row Wise structure, are: the ease of use of multilevel blocks to adapt the computation to the underlying memory hierarchy; and the operation on dense matrices.
Fig. 1. A sparse matrix and its corresponding hypermatrix.
Choosing a block size for data submatrices is rather difficult. Large block sizes favour greater potential performance when operating on dense matrices. On the other hand, the larger the block is, the more likely it is to contain zeros. Since computation with zeros is useless, effective performance can therefore be low. Thus, a trade-off between performance on dense matrices and operation on non-zeros must be reached. The use of windows of non-zero elements within blocks allows for a larger default block size. When blocks are quite full operations performed on them can be rather efficient. However, in those cases where only a few non-zero elements are present in a block, only a subset of the total block is computed. Figure 2 shows a window of non-zero elements within a larger block. The window of non-zero elements is defined by it’s top-left and bottom right corners. Zeros stored outside those limits are not used in the computations. Null elements within the window are still computed. However, the overhead can be greatly reduced.
Fig. 2. A data submatrix and a window within it.
A commercial package known as PERMAS uses the hypermatrix structure [3]. It can solve very large systems out-of-core and can work in parallel.
Improving Performance of Hypermatrix Cholesky Factorization
463
However, the disadvantages of the hypermatrix structure mentioned above introduce a large overhead. Recently a variable size blocking was introduced to save storage and to speed the parallel execution [4]. In this way the HM was adapted to the sparse matrix being factored. The work presented in this paper is focused on the optimization of a hypermatrix Cholesky factorization based on the data structure presented above. We have developed some rather efficient routines which work on small matrices. The purpose of these routines is to get high performance while keeping low the overhead due to unnecessary computations on null elements. 1.2
Goals
This work focuses on obtaining high performance from a hypermatrix Cholesky factorization. We want to reduce the overhead introduced by operations on zeros when large blocks are used. This can be done by reducing the block size. However, in order to keep high performance, we need specialized routines which operate very efficiently on small matrices. 1.3
Motivation
Figure 3 shows the performance of different routines for matrix multiplication for several matrix sizes on an Alpha-21164 processor.
Fig. 3. Comparison of performance of different routines for matrix sizes 4x4 and 16x16.
The vendor’s BLAS routine, labeled as dgemm nts, fails to produce good performance for a very small matrix product (4x4) getting better results as matrix dimensions grow towards a size that fills the L1 cache (16x16). A simple matrix multiplication routine mxmts g which avoids any parameter checking and scaling of matrices (alpha and beta parameters in dgemm) can outperform the BLAS for very small matrix sizes. The matrix multiplication code mxmts fix in our library gets excellent performance for small matrices of sizes 4x4 and 16x16. There is actually one routine for each matrix size. In this paper we call all of them mxmts fix for convenience. Each mxmts fix routine is obtained by fixing
464
J.R. Herrero and J.J. Navarro
leading dimensions and loop limits at compilation time and trying a set of codes with different loop orders and unroll factors. The one producing the best result is then selected. 1.4
Related Work
The Cholesky factorization of a sparse matrix has been an active area of research for more than 30 years [1,5,6,7,8,9]. Iterative compilation [10] consists in a repetitive compilation of code using different parameters. Program transformations like loop tiling and loop unrolling are very effective techniques to exploit locality and expose instruction level parallelism. The authors claim that finding the optimal combination of tile size and unroll factor is difficult and machine dependent. Thus, they propose an optimization approach based on the creation of several versions of a program and decide upon the best by actually executing them and measuring their execution time. Our approach for obtaining high performance codes is similar with the difference that, while they apply a set of transformations, we use simple codes and let the compiler do its best. Several projects were targeted at producing efficient BLAS routines through automatic tuning [11,12,13]. The difference with our work is that they are not focused on operations on small matrices. A software for the parallel solution of sparse linear systems called BlockSolve95 [14] uses macros to put code inline for Level 2 BLAS routines GEMV and TRMV. They claim an improvement ratio between 1.2 and 2 for single processor codes working on small systems. Our approach however, is based on the improvement of Level 3 BLAS routines working on small matrices. The reminder of the paper is organized as follows: first we present our optimization of routines operating on small matrices, namely the matrix multiplication operation. Then we show the impact of its application to the HM Cholesky factorization.
2 2.1
The Small Matrix Library (SML) Generation of Efficient Code
Creation of efficient code has traditionally been done manually using assembly language and based on a great knowledge of the target architecture. Such an approach, however cannot be easily undertaken for many target architectures and algorithms. Alternatively, creation of efficient codes specific for a target computer can be written in a high level language [15,16]. This approach avoids the use of the assembly language but keeps the difficulty of manually tuning the code. It still requires a deep knowledge of the target architecture and produces a code that, although portable, will rarely be efficient on a different platform.
Improving Performance of Hypermatrix Cholesky Factorization
465
A cheaper approach relies on the quality of code produced by current compilers. The resulting code is usually less efficient than that written manually by an expert. However, its performance can still be extremely good and some times it can even yield better code. We have taken this approach for creating a Small Matrix Library (SML). For each desired operation, we have written a set of codes in Fortran. For instance, for a matrix multiplication we have codes with different loop orders (kji, ijk, etc.) and unroll factors. Using a Benchmarking Tool [17], we compile each of them using the native compiler trying several optimization options. For each resulting executable, we automatically execute it and register its highest performance. These results are kept in a database and finally employed to produce a library using the best combination of parameters. By fixing the leading dimensions of matrices and the loop trip counts we have managed to obtain very efficient codes for matrix multiplication on small matrices. Since several parameters are fixed at compilation time the resulting object code is only useful for matrix operations using these fixed values. Actual parameters of these routines are limited to the initial addresses of the matrices involved in the operation performed. Thus, there is one routine for each matrix size. For convenience, we call all of them mxmts fix in this paper, but each one has its own name in the library. We also tried feedback driven compilation using the Alpha native compiler but performance either remained the same or even decreased slightly. We conclude that, as long as a good compiler is available, fixing leading dimensions and loop limits is enough to produce high performance codes for very small dense matrix kernels. 2.2
Matrix Multiplication Performance
Figure 4a shows the performance of different routines for matrix multiplication for several matrix sizes on an Alpha-21164. The matrix multiplication performed in all routines benchmarked uses the first matrix without transposition (n), the second matrix transposed (t), and subtracts the result from the destination matrix (s). This is the reason why we call the BLAS routine dgemm nts. The vendor BLAS routine dgemm nts yields very poor performance for very small matrices getting better results as matrix dimensions grow towards a size that fills the L1 cache (8 Kbytes). This is due to the overhead of passing a large number of parameters, checking for their feasibility, and scaling the matrices (alpha and beta parameters in dgemm). This overhead is negligible when operation is performed on large matrices. However, it is notable when small matrices are multiplied. Also, since its code is prepared to deal with large matrices, further overhead can appear in the inner code by the use of techniques like strip mining. A simple matrix multiplication routine mxmts g which avoids any parameter checking and scaling of matrices can outperform the BLAS for very small matrix sizes. Finally, our matrix multiplication code mxmts fix with leading dimensions and loop limits fixed at compilation time gets excellent performance for all block
466
J.R. Herrero and J.J. Navarro
Fig. 4. Comparison of the performance of different routines for several matrix sizes: a) on an Alpha b) on an R10000.
sizes ranging from 4x4 to 16x16. The latter is the maximum value that allows for a good use of the L1 cache on the Alpha unless tiling techniques are used. Figure 4b shows the performance of different routines for several matrix sizes on the R10000 processor. Results are similar to those of the Alpha with the only differences that the L1 cache is larger (32 Kbytes) and mxmts g performs very well. This is due to the ability of the MIPSpro F77 compiler to produce software pipelined code, while the Alpha compiler hardly ever manages to do so. Though our SML codes were adapted to the underlying architecture, they were written in a high level programming language (FORTRAN). The resulting codes were very good for both the Alpha-21164 and the R10000 processors. We believe these results can be generalized for other current superscalar architectures.
3
Using SML Routines as Computational Kernels for HM Cholesky
We have used SML routines to improve our sparse matrix application based on hypermatrices. Matrices were ordered with METIS [18] and renumbered by an elimination tree postorder. Performance varies substantially from matrix to matrix due to the different sparsity patterns and density. However, we see that using our matrix multiplication in SML improves performance substantially for all the benchmarks and block sizes. Figure 5 shows results of the HM Cholesky factorization on an R100001 for matrix QAP15 from the Netlib set of sparse matrices corresponding to linear programming problems [19]; and problem pds40 from a Patient Distribution System (40 days) [20]. Ten submatrix sizes are shown: 4x4, 4x8, 4x16, . . . 32x32. Effective Mflops are presented. They refer to the number of useful floating point operations performed per second. This metrics excludes useless operations on 1
Results on the Alpha are similar.
Improving Performance of Hypermatrix Cholesky Factorization
467
zeros performed by the HM Cholesky algorithm when data submatrices contain zeros.
Fig. 5. Factorization of matrices QAP15 and pds40: Mflops obtained by different MxM codes in HM Cholesky on an R10000.
When dgemm nts is used, the best performance is usually obtained with data submatrices of size 16 × 16 or 16 × 32. Since the amount of zeros used can be large, the effective performance is quite low. Using mxmts fix however, smaller submatrix sizes usually produce better results than larger submatrix sizes. Particularly effective in this application is the use of rectangular matrices due to the fill-in produced by the Cholesky factorization (skyline). For instance, using 4×16 or 4 × 32 submatrix sizes the routine used yields very good performance. Since the number of operations on zeros is considerably lower, the effective Mflops obtained are much higher than those of any other combination of size and routine. The use of a fixed dimension matrix multiplication routine speeded up our Cholesky factorization between 20% and 100% depending on the input matrix. Table 1. Characteristics and performance of HM Cholesky on several LP problems Matrix Dimension Factor NZs Density TRIPART1 4.2 1.1 0.127 TRIPART2 19.7 5.9 0.030 TRIPART3 38.8 17.8 0.023 TRIPART4 56.8 76.8 0.047 pds40 76.7 27.6 0.009 pds50 95.9 36.3 0.007 pds80 149.5 64.1 0.005 pds90 164.9 70.1 0.005 QAP15 6.3 8.7 0.436
Mflops 223.0 226.9 237.4 278.2 236.6 249.9 254.4 263.3 303.1
Table 1 shows the characteristics of several matrices obtained from linear programming problems [19,20] and the performance obtained by our modified hypermatrix code. The factorization was performed on an R10000 processor with a theoretical peak performance of 500 Mflops. Dimensions are in thousands and
468
J.R. Herrero and J.J. Navarro
Factor non-zeros are in millions. Using SML routines our HM Cholesky often gets over half of the processor’s peak performance for medium size matrices factored in-core.
4
Conclusions
We have shown that fixing dimensions and loop limits is enough to produce high performance codes for very small dense matrix kernels. Since no single algorithm was the best for all matrix sizes in any platform, we conclude that an exhaustive search is necessary to get the best one for each matrix size. For this reason we have implemented a Benchmarking Tool which automates this process. We have generated a small matrix library (SML) on a couple of systems obtaining very efficient codes specialized on operations on small matrices. These routines outperform the vendor’s BLAS routine for small matrix sizes. Fast computations on small matrices are of great utility in sparse matrix computations. A direct application of our small matrix library can be found in the hypermatrix Cholesky factorization. High performance routines operating on small matrices allow for the election of small block sizes. This choice avoids operation of non-zeros while retaining good performance. In practice, we found that blocks of size 4x16 and 4x32 often produced the best results for the hypermatrix Cholesky factorization. We obtained important speed-ups in our Cholesky factorization application by using the SML routines. Therefore, we believe it is worthwhile to develop this sort of library.
References 1. Duff, I.S.: Full matrix techniques in sparse Gaussian elimination. In: Numerical analysis (Dundee, 1981). Volume 912 of Lecture Notes in Math. Springer, Berlin (1982) 71–84 2. Fuchs, G., Roy, J., Schrem, E.: Hypermatrix solution of large sets of symmetric positive-definite linear equations. Comp. Meth. Appl. Mech. Eng. 1 (1972) 197–216 3. Ast, M., Fischer, R., Manz, H., Schulz, U.: PERMAS: User’s reference manual, INTES publication no. 450, rev.d (1997) 4. Ast, M., Barrado, C., Cela, J., Fischer, R., Laborda, O., Manz, H., Schulz, U.: Sparse matrix structure for dynamic parallelisation efficiency. In: Euro-Par 2000,LNCS1900. (2000) 519–526 5. George, A., Liu, J.W.H.: Computer Solution of Large Sparse Positive-Definite Systems. Prentice-Hall, Englewood Cliffs, NJ (1981) 6. George, A., Gilbert, J.R., Liu, J.W., eds.: Graph Theory and Sparse Matrix Computation. Volume 56 of The IMA volumes in mathematics and its applications. Springer-Verlag, New York (1993) 7. Ng, E.G., Peyton, B.W.: Block sparse Cholesky algorithms on advanced uniprocessor computers. SIAM J. Sci. Comput. 14 (1993) 1034–1056 8. Ashcraft, C., Grimes, R.G.: The influence of relaxed supernode partitions on the multifrontal method. ACM Trans. Math. Software 15 (1989) 291–309
Improving Performance of Hypermatrix Cholesky Factorization
469
9. Rothberg, E., Gupta, A.: An efficient block-oriented approach to parallel sparse Cholesky factorization. SIAM J. Sci. Comput. 15 (1994) 1413–1439 10. Kisuki, T., Knijnenburg, P., O’Boyle, M.: Combined selection of tile sizes and unroll factors using iterative compilation. In: Parallel Architectures and Compilation Techniques. (2000) 237–246 11. Bilmes, J., Asanovic, K., Chin, C.W., Demmel, J.: Optimizing matrix multiply using PHiPAC: a portable, high-performance, ANSI C coding methodology. In: 11th ACM Int. Conf. on Supercomputing, ACM Press (1997) 340–347 12. Cuenca, J., Gimenez, D., Gonzalez, J.: Towards the design of an automatically tuned linear algebra library. In: Proceedings. 10th Euromicro Workshop on Parallel, Distributed and Network-based Processing. (2002) 201–208 13. Whaley, R.C., Dongarra, J.J.: Automatically tuned linear algebra software. In: Supercomputing ’98, IEEE Computer Society (1998) 211–217 14. Jones, M.T., Plassmann, P.E.: BlockSolve95 users manual: Scalable library software for the parallel solution of sparse linear systems. Technical report, Argonne National Laboratory (1995) 15. Kamath, C., Ho, R., Manley, D.: DXML: A high-performance scientific subroutine library. Digital Technical Journal 6 (1994) 44–56 16. Navarro, J.J., Garc´ıa, E., Herrero, J.R.: Data prefetching and multilevel blocking for linear algebra operations. In: Proceedings of the 10th international conference on Supercomputing, ACM Press (1996) 109–116 17. Herrero, J.R., Navarro, J.J.: Automatic benchmarking and optimization of codes: an experience with numerical kernels. In: Proceedings of the 2003 International Conference on Software Engineering Research and Practice, CSREA Press (2003) 18. Karypis, G., Kumar, V.: A fast and high quality multilevel scheme for partitioning irregular graphs. Technical Report TR95-035, Department of Computer Science, University of Minnesota (1995) 19. NetLib: (Linear programming problems) http://www.netlib.org/lp/. 20. Frangioni, A.: (Multicommodity Min Cost Flow problems) http://www.di.unipi.it/di/groups/optimize/Data/.
Parallel Agent-Based Simulation on a Cluster of Workstations Konstantin Popov1 , Vladimir Vlassov2 , Mahmoud Rafea1 , Fredrik Holmgren1 , Per Brand1 , and Seif Haridi2 1
2
SICS, Kista, Sweden. http://www.sics.se KTH/IMIT, Kista, Sweden. http://www.imit.kth.se
Abstract. We discuss a parallel implementation of an agent-based simulation. Our approach allows to adapt a sequential simulator for largescale simulation on a cluster of workstations. We target discrete-time simulation models that capture the behavior of WWW. The real-world phenomena of emerged aggregated behavior of the Internet population is studied. The system distributes data among workstations, which allows large-scale simulations infeasible on a stand-alone computer. The model properties cause traffic between workstations proportional to partition sizes. Network latency is hidden by concurrent simulation of multiple users. The system is implemented in Mozart that provides multithreading, dataflow variables, component-based software development, and network-transparency. Currently we can simulate up to 106 Web users on 104 Web sites using a cluster of 16 computers, which takes few seconds per simulation step, and for a problem of the same size, parallel simulation offers speedups between 11 and 14.
1
Introduction
This paper discusses a parallel implementation of discrete-time agent-based simulation models. Such models are defined bottom-up, i.e. by defining individual behavior of components called agents. Our approach allows to adapt a sequential simulator for large-scale simulation on a cluster of workstations. We target simulation models that capture the behavior of WWW. Web users are active and stateful agents connected with each other in a graph resembling the social network. Web sites are connected in a similar graph, forming a “landscape” where agents reside. At each time step, users exhibit certain behaviours such as visiting some of their bookmarked sites, exchanging information about Web sites in the “word-of-mouth” style, “web surfing” (i.e. visiting sites pointed by sites), and updating bookmarks. Once all users are simulated, the simulation proceeds to the next time step. In general, users access different sites at different time steps. Since users exchange information about sites, every user can acquire a reference and then access every site. The model parameters affect behavior of individual model components and their interaction. Running the simulation produces statistics which are used to study the aggregated behavior of the Internet population. In particular, the relationship between site popularity (i.e. number H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 470–480, 2003. c Springer-Verlag Berlin Heidelberg 2003
Parallel Agent-Based Simulation on a Cluster of Workstations
471
of site visits) and the ratio of sites with that popularity, as well as the formation of clusters of popular sites are studied. Large-scale simulations with millions of model users may aid the understanding of the evolution of the Internet [9]. The simulator distributes data among workstations because of the memory consumption. Every workstation simulates its portion of model users. Simulation of 105 model users with 104 sites using in current model requires approximately 250Mb of main memory, while we target at least 106 users that would require at least 2.5Gb. More refined simulation models (for instance, reflecting the evolution of sites in terms of the quality of service, the type of content and the connectivity with other sites) require significantly more memory. On the other hand, we would like to speedup the execution of todays’ simulation runs – with 105 users for thousands of steps. The sequential simulator requires approximately 1 minute for 104 users for 102 steps on a 1GHz computer, so we could expect the parallel execution to finish in “hours” instead of “days”. Model users can access arbitrary model sites, so the latter can be seen as shared data. This is different to simulation models such as computational fluid dynamics or neural networks, where model entities encapsulate their state entirely. In the worst case, distribution of shared data among computers causes network traffic proportional to the amount of that data. Moreover, users access different sites over time, as their bookmarks and bookmarks of their neighbours evolve, so that any distribution of sites aimed at minimizing the network traffic has to be dynamic. We opted for static partitioning because of the complexity and memory and run-time costs of a dynamic one. The graphs that represent the social network of users and links between sites are “small-world” graphs, where every two nodes are connected through a rather short chain of links, while neighbour nodes are tightly connected with each other [23]. Intuitively, in order to meet these defining properties, such a graph must have a sufficient number of “long-distance” links. In a typical simulation run of our system, 30% of links point to remote nodes which are uniformly distributed over the entire set. Our heuristic for graph partitioning tries to keep tightly connected neighbours on the same workstation. We are unaware of an algorithm that optimizes partitioning with respect to links to remote nodes. Randomness of links to remote nodes causes the majority of them to cross the partition boundaries and thus cause network traffic. The amount of the traffic is proportional to the number of nodes in the graph. Very recently, however, we noticed a work on extraction of larger sub-graphs from a small-world graph [12]; we are studying its applicability for our system. The system must be also maintainable, so that different web-simulation models could be captured within the same parallel simulation framework. Our colleagues who design and study models often change user behaviours and graph construction mechanisms. To meet this practical need, our system consists of a set of modules implementing behaviours of web users, constructing web sites, and also constructing users and sites graphs. The other part of the system is the infrastructure that holds collections of users and sites, and runs the simulation. Moreover, the interfaces between modules and the infrastructure part is
472
K. Popov et al.
the same as in the sequential simulator, so our colleagues develop modules using the sequential system and we use the same modules in the parallel one. Rapid prototyping of simulator modules is facilitated by our implementation platform, the Mozart system [13,21], available for both Un*x systems and Windows. Mozart implements Oz – a concurrent multi-paradigm programming language that offers symbolic computation with a variety of builtin data types, automatic memory management, and network-transparent distribution. In the parallel system, the implementations of abstract data types (ADTs) that hold collections of users and sites hide the network from the simulation code. The data is partitioned between Mozart processes, but remote data can also be cached as long as the simulation model allows. The ADTs’ implementations, in turn, are built atop of one single “distributed ADT” (DistADT) abstraction which completely encapsulates the operations over the network. When data is requested that does not belong to the local partition, a corresponding computer is contacted inside of the DistADT implementation. Since there are multiple simulation threads, network latency is automatically masked. The parallel simulator can utilize different workstations. Our current model of data partitioning assigns partition sizes proportionally to the capacities of corresponding computers. Capacities are reversely proportional to the execution times of a certain fixed sequential simulation. The number of threads per Mozart process is a system-wide constant, which turned out to be good enough. The performance of the simulation has been improved by avoiding explicit and implicit synchronization between processes: barriers between time steps have been replaced by a more relaxed synchronization, and garbage collection in Mozart processes is initiated synchronously by the manager process. We have already obtained promising results: on a cluster of 16 workstations simulating 106 users on 104 web sites, our simulator scales nearly perfectly if the shared data is stateless and replicated eagerly, nearly perfectly if it is replicated lazily, and deteriorates only by some 40% if data is stateful. If we interpolate the execution time to simulate 106 users on a single computer (the value we cannot directly obtain due to memory requirements), the simulation system offers the speedup of 14 with stateless replicated data, and 11 with stateful data.
2
Related Work
Our simulation models belong to the “discrete time system” class [25], which is a specialization of “discrete event system specification” (DEVS) class where time is continuous but only a finite number of events take place. Parallelization of discrete event models has received a lot of attention [25,8,6,16]. Our models are specified in terms of pro-active behaviour and interaction between entities rather than “events” and “response behaviour”. It is acknowledged that most of the work on parallel discrete event simulation is done with the focus on parallel speedups, while the scalability, software engineering, and model reuse are the real issues for the modeler [18,24]. All these issues are our goals. Relatively little effort is devoted for large-scale simulation
Parallel Agent-Based Simulation on a Cluster of Workstations
473
and corresponding tools, with the notable exceptions of specially designed DEVS implementations [3], DaSSF(Dartmouth SSF, Scalable Simulation Framework)related activities [4,17], as well as various attempts to build distributed simulators through federation of sequential ones [5]. We address the scalability issue by explicit distribution of data, and encapsulating the network operations within the implementations of collections of model users and sites. Parallelizing a simulation involves at least the use of a specially designed simulation platform (e.g. [24]) that hides parallelization details, yet usually an adaptation of the model and some annotations for the simulation run-time system are necessary too [15,5]. Our approach is radically different: only the implementation of collections of agents (web users) and landscape locations (web sites) changes, while the implementations of the agent behaviours are kept unchanged. Our work can be also assessed from the point of view of multi-agent simulation [10,20,1]. The difference to our work is twofold: while agents in multi-agent simulations appear to be more sophisticated than those in our case, we target orders of magnitude larger simulations. Another extreme are the so-called microsimulations [2,14], where numerous agents are as simple as finite automata. Our approach to distributed simulation is novel practically as it does not involve explicit message passing as in [19,11], but rather provides a distributed implementation of the same abstract data types as in the sequential simulator. The programmer’s obligation is to spawn the number of threads sufficient for hiding communication latency. Our approach is similar to the one of Nexus [7], where special “global pointers” are network-transparent entities.
3
The Simulation Model
The “word of mouth” (WoM) simulation model is a stochastic socio-economical model that addresses the real-world phenomena of the behavior of the Internet population [9]. The model of time is discrete. The model is defined in terms of Web users and Web sites. Users are modeled as stateful entities: in particular, they have short-term (“loyalty”) and long-term (“bookmarks”) memory. Sites have ranks representing their “goodness”. Users and sites are interconnected by two separate small world (SW) graphs [23]. The users graph represents a social network of interaction between users, and the sites graph represents the Web. The topology of graphs and the number of nodes are fixed. We use the Watts-Strogatz model for building SW-graphs, where a SW-graph is based on a 1-d lattice, which models interconnections between users/sites according to their physical proximity. A fraction of links of the lattice is then randomly rewired. At each step, each user exhibits a number of behaviours: (a) it visits a random subset of its bookmarked sites, (b) contacts a random set of its neighbour users and asks them about sites worth visiting (the “word of mouth” (WoM) mechanism), (c) surfs some links on some of the sites visited during behaviours a and b, and (d) evaluates sites visited during behaviours a,b and c, and updates the loyalty/bookmark lists. Users that are contacted during WoM behaviour must complete the previous simulation step, and must not complete the next step.
474
K. Popov et al.
Some of them may have completed the current step; thus, users can be simulated in any order within a time step. Note that information about “interesting” sites is propagated through the social network. Best visited sites are accumulated in the short-term memory, and can eventually be promoted to the bookmarks. Model parameters define the number of users/sites, constants for SW-graph construction, sizes of loyalty/bookmarks lists, and various constants controlling user behaviours. The simulation output is a sequence of statistic vectors containing numbers of visits per each site taken at every N th step. The simulation has shown that the distribution of users per sites follows a universal power law [9]. Note that the model is stochastic. The simulation results did not change once a more relaxed synchronization was introduced, where users contacted during the WoM behaviour were allowed to have completed the next simulation step.
4
Oz and Mozart
Oz is a dynamically typed language with a variety of basic data types, including atoms, numbers, records and arrays. Builtin dictionaries allow to store and retrieve (key, value) pairs. Oz is a lexically-scoped language. Procedures in Oz are first-class, and generally have environment which encapsulates values from the procedure’s lexical scope. Concurrent threads in Oz can synchronize by dataflow variables. A dataflow variable represents a value that can be defined only once. If a value is not defined yet, threads that depend on it are transparently blocked by the Mozart run-time system until a value is defined. Defining a variable’s value is referred to binding a variable. Threads can also communicate via asynchronous message passing through ports. A port is an abstract data type with the “send a value” operation that provides a partially ordered mailbox for messages: messages sent by the same thread are retrieved from the box in the same order as they are sent. Mozart provides for a network-transparent distribution of Oz programs: an application runs on a network of computers as it were one computer. This is supported by the Mozart’s distribution subsystem. Oz entities shared by threads running on different Mozart processes are called distributed. Stateless distributed entities are replicated, and stateful ones are kept consistent with the help of access structures that connect involved processes and are used by entity-type specific distribution protocols. When an operation on a distributed stateful entity is performed, distribution subsystems of involved processes exchange message(s) according to the entity’s protocol. Mozart differentiates between stationary and mobile entities. A ports, for example, is stationary: once created, it stays at a process where created; messages to the port are delivered to its “home” process.
5
The Simulator
Our parallel simulator is based on the sequential simulator which organization follows the one of the Worker on Figure 1. Simulation is controlled by the Manager component. Simulator executes behaviours for each user using data
Parallel Agent-Based Simulation on a Cluster of Workstations
475
from Users and Sites modules, and records statistics into Statistics. Model users and sites are represented as records, and are indexed. Users and Sites abstractions store representations under indices. Users’ neighbours, users’ bookmarks, and outgoing links from sites are represented as lists of indices. Parallel simulation is conducted by workers, each running several threads. Users and Sites modules hold now partitions of data, but provide transparent access to all data. Only locally stored users are simulated. Users and Sites are the same abstractions as in the sequential system but with different implementations, so the Simulator component is reused without changes. The Manager process constructs the WoM graphs for users and sites, and the representations of Fig. 1. The Simulator. users and sites themselves. This data is initialized by the sequential Users and Sites components, so these components are reused as well. Once constructed and initialized, the data is divided into partitions and partitions are distributed to corresponding workers. Workers construct the distribution-aware Sites and Users modules containing the given partition data. We have implemented both the strict and relaxed synchronization between workers, as described in Section 3. With the strict synchronization, there are barriers between simulation steps: the manager waits until all workers finish a step, then it initiates the next step. With the relaxed synchronisation, if a worker at a time step n contacts another worker at step m, then either n = m or n = m − 1, i.e. a user that is contacted because of the WoM behaviour is allowed to have completed one step ahead. It is implemented as follows. At the beginning of a step n, each worker waits until all other workers report that they completed step n − 2, then it by itself tells other workers that it completed n − 1, and proceeds with the step n. It can be shown that the time in any two workers can differ by at most 1. Since synchronization messages and accesses to remote data go over the same channels, this guarantees that when a worker at step m receives a request from a worker at step n, then n = m±1. The remaining case of n = m + 1 means the worker that received a request is lagging behind, so the request is queued until the next step.
6
Distributed Abstract Data Types
Users and Sites data abstractions of the parallel system contain local data and GetRemote procedures which provide access to data from remote partitions. GetRemote abstractions are provided by the generic DistADT abstract data type, hence the generic “GetRemote” name. Let us consider the Users
476
K. Popov et al.
abstraction; the Sites abstraction is implemented similarly. Users provides the GiveVote method that calculates a WoM recommendation from a specified user. GiveVote takes a user index and returns a user index. Assume that a requested user happens to be non-local (arrow 1 on Figure 2), in which case GetRemote is used. It sends a request to a server on the right partition (arrow 2). The server uses that partition’s GiveVoteLocalUser procedure (arrow 3). The output of GetVoteLocalUser is delivered backwards (arrows 4, 5 and 6) in a container sent by GetRemote along with the request. A container is implemented as a dataflow variable: it is declared by GetRemote, becomes distributed between the two processes when a request arrives to the server, and is bound to the result by the server. Eventually the value is delivered back (arrow 5) by the Mozart distribution subsystem. The GetRemote abstraction is provided for Users partition by the DistADT data abstraction (see Figure 3). DistADT modules are shared among all partition processes. In our example, the server Oz port and process logically belong to the DistADT module serving the Fig. 2. Distributed Users ADT. Users module. Although not shown on picture, ports and servers of other Users partitions belong to the same DistADT module too. There is exactly one port/server pair per partition. The DistADT module encapsulates a map from user indices to their partition indices, so for any user index GetRemote can choose the correct port and send a request to the correct partition.
Fig. 3. Users ADT using DistADT.
In order to use a DistADT module, Users partitions have to register. Registration gives a DistADT module access to local partition data. In our example, the GiveVoteLocalUser procedure is the access abstraction. During registration a (stationary) Oz port is constructed and a local server (a thread that processes requests sent to the port) is started. DistADT modules are created by the manager and passed over to all partitions, so they become shared between partitions.
Parallel Agent-Based Simulation on a Cluster of Workstations
477
Note that invocation of GetRemote transparently returns a value of whatever type the server’s “GetLocal” (GetVoteLocalUser in our example) does. In this way, two (distinct) instances of the very same DistADT data abstraction are used for both Users and Sites ADTs. Both the “data to computation” and “computation to data” approaches to parallel execution are supported by DistADT, and used in our system. In the GiveVote example, the vote is computed by the process holding the data. Computing a vote requires analysis of the bookmark list, while both the query and the response messages exchanged between processes contain only an index. Thus, it is beneficial to move the computation to data. On the other hand, the complex algorithm of user simulation requires representations of several sites visited by the user. Those sites can even belong to different partitions. Thus, site representations (data) are moved to the partition where simulation takes place.
7
Performance
We have evaluated our system on a cluster of 16 AMD Athlon 1900+ computers connected by the 100Mbit switched Ethernet. Scalability and speedup properties are presented on Figure 4. The “no caching/replication” case refers to the situation when model sites that reside on a remote worker are retrieved over the network every time they are needed. The “caching” version keeps retrieved sites for the duration of the model time step In the version“with replication”, all workers have their own copies of stateless model sites. While in the WoM model the sites are stateless, our experiments show how the simulator would behave if that data were stateful. Since the majority of data is remote even with 16 workers, the system should scale well also beyond that size. The system also delivers respectful speedups when it counts most – for larger problem sizes. scalability (10k model sites, 100 steps)
speedup with replication (10k sites, 100 steps) 16
600
125k users 250k users 500k users 1M users
14
500
12
300
200
100
0 1 (62.5k users) 2
no caching/replication caching lazy replication (eager) replication
12
10 8
4
8
16 (1M users)
10 8
6
6
4
4
2
2
0 workers
125k users 250k users 500k users 1M users
14
speedups
speedups
time (sec)
400
speedup with caching (10k sites, 100 steps) 16
0 1
2
4
8
16
workers
1
2
4
8
16
workers
Fig. 4. Scalability and Speedups.
Figure 5 shows the impact of multithreading. Given the properties of the simulation model and the best-case communication latency in Mozart, we have also estimated the number of threads required to hide communication latency using the (analytical) deterministic model of processor utilization in a multithreaded
478
K. Popov et al.
architecture (see e.g. [22])1 . The model predicts that approximately 100 threads is needed to hide the latency, which is in compliance with our experimental data. The next plot shows that sporadical load imbalance due to e.g. activities of other processes in OSes is a no big issue for our simulator. We emulated such kind of imbalance by deliberate over/underloading of one worker during initial partitioning. Note also that underloading a worker, which apparently causes its more frequent preemption by the OS, does not have a “cascading” effect on other workers. We are investigating this issue further. performance = f(#threads)
350
800
300
500
time(sec)
400
900
600
600
450
1000
700
1M users, caching 1M users, replication 250k users, caching 250k users, replication
250 200 150 100
400 1 2 4 9 19 39 78 156 312 1.25k 5k 20k 40k
400 300 200 C: caching R: replication
100
50
threads
Effect of synchronization relaxed, R strict, R 500 relaxed, C strict, C
Effect of under/overload of one of 16 workers
1M users/10k sites
time(sec)
time (seconds)
1100
0
0 -10
-5
0
5
10
250k
load imbalance (%) (< 0: underloaded, > 0: overloaded)
1M users
Fig. 5. Number of threads, load balance, and synchronization.
Finally, the effect of “relaxed” synchronization discussed in Sections 3 and 5 is presented: one can see quite a significant improvement, in particular for larger simulations. Given that a barrier in Mozart can require ca. 50msec, so on 100 steps ca. 5 seconds could be saved just by eliminating the barriers, the improvement is larger than we have expected for large simulations. Our hypothesis here is that relaxed synchronization allows to “smooth out” temporal imbalance of work between adjacent time steps.
8
Conclusions
We have shown a design of a parallel simulator for discrete-time agent-based simulation models targeted for studying the evolution of WWW. Distribution of data among computers is mandatory due to memory requirements, but efficient parallel simulation is not trivial because of the large amount of spatially and temporally irregular inter-process communication. At the same time, the simulator must be easily adapted for a set of different simulation models. The parallel simulator runs on both a network of workstations and a multiprocessor, and shows good scalability and speedup for up to 1M of model agents. The simulator is small and easy to maintain. It is implemented in Oz/Mozart that supports symbolic, concurrent and network-transparent programming, all necessary for this work. The evaluation of the simulation on a cluster of workstations is still going on, concurrently with integration of new simulation models. 1
Nthreads = L/(R + C) + 1, where L and R are the mean times for communication latency and thread run time between remote memory accesses, respectively, and C is the thread context switch time.
Parallel Agent-Based Simulation on a Cluster of Workstations
479
Acknowledgments. We are thankful to all members of our lab for the creative environment, ubiquitous support and constant encouragement. Special thanks go to Petros Kavassalis and Stelios Lelis from ICS-FORTH, Heraklion, Crete, Greece. The work is done within the EU project iCities (IST-1999-11337).
References 1. Anderson, J.: A Generic Distributed Simulation System for Intelligent Agent Design and Evaluation. In: Proc. of the 10th Int. Conf. on AI, Simulation, and Planning in High Autonomy Systems (2000) 36–44 2. Cetin, N., Nagel, K., Raney, B., Voellmy, A.: Large-Scale Multi-Agent Transportation Simulations. In: Proc. of the Workshop on Cellular Automata, Neural Networks and Multi-Agent Systems in Urban Planning, Milan, Italy (2001) 3. Chow, A.C.H., Zeigler, B.P.: Parallel DEVS: A Parallel, Hierarchical, Modular, Modeling Formalism. Proc. of the 26th Winter Simulation Conf. (1994) 716–722 4. Cowie, J., Liu, H., Liu, J., Nicol, D., Ogielski, A.: Towards Realistic Million-Node Internet Simulations. In: Proc. of PDPTA’99, Las Vegas, NV, USA (1999) 5. Ferenci, S.L., Perumalla, K.S., Fujimoto, R.M.: An Approach for Federating Parallel Simulators. In: Proc. of PADS’00, Bologna, Italy. IEEE CS Press (2000) 63–70 6. Ferscha, A.: Parallel and Distributed Simulation of Discrete Event Systems. In: Parallel and Distributed Computing Handbook. McGraw-Hill (1996) 1003–1041 7. Foster, I., Kesselman, C., Tuecke, S.: The Nexus Approach to Integrating Multithreading and Communication. J. of Paral. and Dist. Computing 37 (1996) 70–82 8. Fujimoto, R.M.: Parallel and Distributed Simulation Systems. Wiley (2000) 9. Lelis, S., Kavassalis, P., Sairamesh, J., Haridi, S., Holmgren, F., Rafea, M., Hatistamatiou, A.: Regularities in the Formation and Evolution of Information Cities. In: Second Kyoto Workshop on Digital Cities. LNCS 2362, Springer (2001) 41–55 10. Logan, B., Theodoropoulos, G.: The Distributed Simulation of Multiagent Systems. Proceedings of the IEEE 89 (2001) 174–185 11. Lowenthal, D.K., Freeh, V.W.: Architecture-Independent Parallelism for Both Shared- and Distributed-Memory Machines Using the Filaments Package. Parallel Computing 26 (2000) 1297–1323 12. Matsuo, Y.: Clustering using Small World Structure. Proc. of the Int. Conf. Knowledge-Based Intelligent Inform. and Engineer. Systems, Crema, Italy (2002) 13. Mozart Consortium: The Mozart Programming System. http://www.mozart-or.oz/ 14. Nagel, K., Rickert, M.: Parallel Implementation of the TRANSIMS MicroSimulation. Parallel Computing 27 (2001) 1611–1639 15. Nicol, D.M., Heidelberger, P.: Parallel Execution for Serial Simulators. ACM Transactions on Modeling and Computer Simulation (TOMACS) 6 (1996) 210–242 16. Nicol, D.M.: Principles of Conservative Parallel Simulation. In: Proc. of the 28th Winter Simulation Conf., Coronado, CA, USA. ACM Press (1996) 128–135 17. Nicol, D.M., Liu, J.: Composite Synchronization in Parallel Discrete-Event Simulation. IEEE Transactions on Parallel and Distributed Systems 13 (2002) 433–446 18. Page, E.H.: Beyond Speedup: PADS, the HLA and Web-Based Simulation. In: Proc. of PADS’99, Atlanta, GA, USA. IEEE Society Press Press (1999) 2–9 19. Strumpen, V.: Software-Based Communication Latency Hiding for Commodity Workstation Networks. In: Proc. of ICPP’96. CRC Press (1996) 146–153 20. Uhrmacher, A.M., Gugler, K.: Distributed, Parallel Simulation of Multiple, Deliberative Agents. Proc. of PADS’00, Bologna, Italy. IEEE CS Press (2000) 101–108
480
K. Popov et al.
21. Van Roy, P., Haridi, S.: Concepts, Techniques, and Models of Computer Programming. MIT Press (2004) (to appear) 22. Vlassov, V., Ayani, R.: Analytical Modeling of Multithreaded Architectures. J. of Systems Architecture 46 (2000) 1205–1230 23. Watts, D.L.: Small Worlds: the Dynamics of Networks Between Order and Randomness. Princeton University Press, New Jersy, USA (1999) 24. Wieland, F., Blair, E., Zukas, T.: Parallel Discrete-Event Simulation (PDES): A Case Study in Design, Development, and Performance Using SPEEDES. In: Proc. of PADS’95, Lake Placid, NY, USA. IEEE CS Press (1995) 103–110 25. Zeigler, B.P., Praehofer, H., Kim, T.G.: Theory of Modeling and Simulation: Integrating Discrete Event and Continuous Complex Dynamic Systems. Academic Press (2000)
Low Level Parallelization of Nonlinear Diffusion Filtering Algorithms for Cluster Computing Environments David Slogsnat1 , Markus Fischer1 , Andr´es Bruhn2 , Joachim Weickert2 , and Ulrich Br¨ uning1 1
Department of Computer Science and Engineering University of Mannheim, 68131 Mannheim, Germany
[email protected] 2 Faculty of Mathematics and Computer Science Saarland University, Building 27.1, 66041 Saarbr¨ ucken, Germany
Abstract. This paper describes different low level parallelization strategies of a nonlinear diffusion filtering algorithm for digital image denoising. The nonlinear diffusion method uses a so-called additive operator splitting (AOS) scheme. This algorithm is very efficient, but requires frequent data exchanges. Our focus was to provide different data decomposition techniques which allow for achieving high efficiency for different hardware platforms. Depending on the available communication performance, our parallelization schemes allow for high scalability when using fast System Area Networks (SAN), but also provide significant performance enhancements on slower interconnects by optimizing data structures and communication patterns. Performance results are presented for a variety of commodity hardware platforms. Our most important result is a speedup factor of 210 using 256 processors of a high end cluster equipped with Myrinet.
1
Introduction
Over the recent years, distributed processing has been a powerful method to reduce the execution time of computationally intensive applications. Using components off the shelf (COTS) in combination with a fast network has become a viable approach to achieve high performance known from supercomputers, however at a fraction of the cost. The platform independent message passing interface MPI [7] allows applications to run on a variety of systems. This reduces the overhead of porting applications significantly. Image processing is becoming a more and more important application area. Variational segmentation and nonlinear diffusion approaches have been very active research fields in the area of image processing and computer vision as well. Motivating factors for this research on PDE (partial differential equation) based models are for example the continuous simplification of images or shapes which help to understand what is depicted in the image. Image enhancement including H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 481–490, 2003. c Springer-Verlag Berlin Heidelberg 2003
482
D. Slogsnat et al.
denoising, edge enhancement, active contours and surfaces or closing interrupted lines are other points of interest. Within this context efficient numerical algorithms have been developed. They are the basis for this work, which focuses on reducing the execution time using distributed processing. AOS schemes for nonlinear diffusion filtering have been applied successfully on a parallel system with shared memory [2] and a moderate processor count. This motivated us to investigate their suitability on distributed memory systems with a large number of processors to enable close-to-real-time processing. One specific application area which we would like to address is the analysis of 2D and 3D medical images. For this scenario the approach of using clusters as computing resources gives a dynamic range from low- to high-end systems. In the following chapter, we will describe the algorithm and how it was parallelized. Chapter 3 briefly describes the clusters that have been used as testbeds. In Chapter 4 we will present the performance results. Chapter 5 concludes our description.
2
Parallelization of the Algorithm
This section describes the algorithm and explains two possible solutions for data decomposition, a dimension segmentation and a mesh segmentation method. The algorithm is described for the three dimensional case, the two dimensional case is analogous. Figure 2 shows a filtered 2D image, figure 1 shows the original image.
Fig. 1. Image with Noise
2.1
Fig. 2. Image after Filtering
Algorithm Components and Constraints
In the last decade, PDE based models have become very popular in the fields of image processing and computer vision. The nonlinear diffusion models as we know them today were first introduced by a work of Perona and Malik [3], who developed a model that allows denoising images while retaining and enhancing
Low Level Parallelization of Nonlinear Diffusion Filtering Algorithms
483
edges. This model has been improved by Catt´e et al. [11] from both a theoretical and practical viewpoint. Anisotropic extensions with a diffusion tensor are described in [5]. In practice, nonlinear diffusion filters require numerical approximations. In [1] a finite difference scheme based on an additive operator splitting (AOS) technique [4] is used for this purpose. This AOS technique is the basis for our parallelization efforts. Typical AOS schemes are one order of magnitude more efficient than simple diffusion algorithms. The AOS scheme is applied in an iterative way until the desired level of denoising is reached. The AOS iteration loop starts with a Gaussian convolution, which needs to be processed for each dimension. Based on this smoothed image, derivatives are computed that are of need to determine the diffusitivity values. Finally the diffusion processes for all dimensions are calculated and the results are averaged in a final recombination step. Gaussian Convolution. The Gaussian convolution is processed seperately for each dimension d ∈ {1,2,3}. At iteration step n, the algorithm computes a matrix n = fgauss (M n−1 ). M n using the matrix from the previous step n−1, thus Mgauss The convolution takes places for one dimension after another. The computation itself is a stencil operation with a stencil size of 1 × g for every dimension, with a typical value for g = 3. As a consequence, communication takes place before the computation for each dimension, but not during the computation for a single dimension. Diffusitivity. The diffusitivity is calculated from the output matrix of the n n Gaussian convolution step: Mdif f = fdif f (Mgauss ). In contrast to the Gaussian convolution, the diffusitivity is not calculated seperately for every dimension, but only once for all dimensions. The stencil has a fixed size of 3 × 3 × 3 pixels. As a result, communication between nodes has to be performed only once before the diffusity is calculated, but not during the calculation process. Diffusion. The diffusion is computed from two matrices, those are the matrix from the previous iteration M n−1 and the output matrix of the diffusitivity n Mdif f . To compute the diffusion, d tridiagonal linear systems have to be solved by a fast Gaussian algorithm, where every system describes diffusion in one of the dimensions. These systems can be solved independently of each other, although they are solved one after another for practical reasons. Also, they can again be decomposed into many small independent equation systems, where each system corresponds to a single line in diffusion direction. These lines are solved by an algorithm using first forward and then backward propagation. This means that if a line crosses process boundaries, these processes have to communicate with each other during the computational phase. This is different from the Gaussian convolution and diffusitivity steps, where no communication is required. Finally, all three resulting matrices have to be merged into the final matrix M n by taking the average.
484
2.2
D. Slogsnat et al.
Data Decomposition
In the following we present two data decomposition methods for 2D and 3D scenarios that have both been applied to the algorithm. Figure 3 depicts schemes for the decomposition methods. The first decomposition method is a multidimensional slice decomposition [14]. Given a number of NP processes and a discrete 3D image f [x1 , x2 , x3 ], x1 ∈ {0, .., X1 − 1}, x2 ∈ {0, .., X2 − 1} and x3 ∈ {0, .., X3 − 1} then the partial image f [x1 , x2np , x3 ] with x2np ∈ {X2 /N P ∗ np, .., X2 /N P ∗ (np + 1) − 1}, will be processed by processor np, np ∈ {0, .., N P − 1}. In a similar manner, the image can also by sliced up in one of the other two dimensions. Since the diffusion algorithm performs filters in all three dimensions, this means that all three decomposition types are used to reduce communication between processes. When the program switches between filters of different dimensions, the direction of decomposition is changed accordingly. However, this implies that an all-to-all communication has to take place with a total traffic of NNPP−1 ∗ Imagesize pixels, since every processor has to aquire the actual values for all pixels in the new slice. 1
0 2
3
0 1 2 3
0 1 2 3
....
n-1
....
n-1
n-1
(a)
(b)
(c)
Fig. 3. Column (a) and Row- (b) versus Mesh Decomposition (c)
The second method is a mesh partitioning which divides the data as shown in the figure. Mesh partitioning has the advantage that it requires less interconnect performance than the slice decomposition method. In contrast to slice partitioning, the image does not have to be redistributed among the processors, but only the borders of the meshes have to be exchanged with the neighboring meshes. This leads to a lower amount of data that has to be communicated. Also, communication has only to be performed with a small number of neighbors in contrast to the all-to-all communication required by the slice decomposition method. However, mesh partitioning has a downside too. Communication between processes takes places during computational steps, rather than in-between them. This increases the chance of processes waiting for results of neighboring processes. This problem can be reduced by means of pipeling. This has been implemented for the diffusion step of the algorithm: the processing of different lines is interleaved.
Low Level Parallelization of Nonlinear Diffusion Filtering Algorithms
3
485
Available Cluster Environments
Ethernet type solutions for a network of workstations or a small cluster are currently still being considered as a solution for distributed computing. One of the reasons is that it is available as a built-in solution with no additional costs, and applications using sockets will work on any type of Ethernet ranging from Fast Ethernet to Gigabit Ethernet. In contrast to Ethernet network interfaces, System Area Networks (SANs) always provide direct user access. Also, SAN network adapters implement reliable transport services directly in hardware. They also deliver very high bandwidths (typically more than 1 GB/s) with very low latency. Currently, the most popular SAN is Myrinet from Myricom[8]. Other SAN implementations are the Scalable Coherent Interface (SCI) from Dolphin[9], Qsnet from Quadrics[10] and ATOLL from the University of Mannheim[13]. Our goal was to provide efficient implementations on clusters, using a variety of interconnection networks. Although SANs are clearly the choice of system for parallel processing, we analyzed our algorithm on Ethernet clusters too. The reason to do so is to provide an algorithm that performs best on any type of cluster, not only on dedicated high performance systems. In the following, we will give a short overview of the platforms which were available for our performance evaluations.
Paderborn Center for Parallel Computing, Germany. The PC2 hosts a hpcLine system which includes 96 nodes, consisting of 2 PIII 850 Mhz CPU’s each. A Linux 2.4 is running in SMP mode. Besides regular Ethernet, a faster interconnect is available with the SCI network plugged into a 32Bit/33Mhz PCI interface. The MPI library is a commercial MPI implementation from Scali, Norway, called Scampi.
Real World Computing Partnership, Japan. The ScoreIII cluster consists of 524 nodes, two PIII 933 Mhz processors each, running a modified Linux 2.4 SMP Kernel. The Cluster is fully interconnected to a CLOS network using a Myrinet2000 network interface. Although Myrinet offers a universal 64Bit/66Mhz interface, the motherboard only supports the 64Bit/33Mhz mode. The cluster makes use of the Score cluster system software[6], which provides fast access to both Myrinet and Ethernet devices. The cluster is ranked 90th in the November 2002 Top500 Supercomputer List.
FMI Passau, Germany. The FMI in Passau is also equipped with a hpcLine, offering latest SCI cards with 64Bit/66Mhz. The dual Intel PIII 1000 Mhz Processors are running a Linux 2.4 Kernel. Just as the PC2 cluster, the FMI uses the Scampi environment. All clusters do also have Fast Ethernet network interfaces.
486
D. Slogsnat et al.
Fig. 4. Speedup on Myrinet
Fig. 5. Speedup on SCI with 32bit/33MHz PCI and 64bit/66MHz PCI
4
Performance
In the following, the results of a number of different versions of the algorithm will be presented. All are using non blocking communication calls. In contrast to collective scatter communication, non blocking communication allows for better
Low Level Parallelization of Nonlinear Diffusion Filtering Algorithms
487
scalability with the system size. We expected a performance difference when switching from a SAN to Fast Ethernet and we were interested in breaking down the ratio of communication and computation.
Fig. 6. Myrinet execution time breakdown using slice partitioning
As expected, the highest speedup can be achieved on Myrinet clusters. As shown in figure 4, a speedup factor of 210 on 256 processors using the slice decomposition method reveals an almost linear increase in performance. Mesh partitioning scales well too, but it is by far outperformed by the slice decomposition method, which is up to 50% faster than mesh partitioning. Much to our surprise, we observed a contrarian scaling behaviour on SCI, illustrated in figure 5. While mesh partitioning scales almost as good on SCI as on Myrinet, slice partitioning does not scale at all on SCI. The scaling behaviour using Ethernet is very similar to the behaviour using SCI. However, the total speedup achieved is noticeable smaller: the use of 64 processors leads to a speedup factor of 22. A breakdown of the total execution time into computation and communication helps to reveal bottlenecks. For Myrinet, figure 6 shows that the time spent for communication is negligible. In contrast, the communication overhead for SCI increases significantly, as shown in figure 7. According to our results and as indicated by benchmarks like the Effective Bandwidth Benchmark [12], the bandwidth of the SCI interface itself is not the bottleneck. We measured a peak bandwidth of 85 MB/s and a one-way latency of 5.1µs for the slower 32bit/33MHz PCI cards, which is more than sufficient for our algorithm. It is more likely that the SCI ring structure can not deal very well with high data traffic. On the unidirectional SCI ring, applications may also be hampered by
488
D. Slogsnat et al.
Fig. 7. SCI execution time breakdown using slice partitioning
Fig. 8. Ethernet execution time breakdown using slice partitioning
other applications running on the cluster. Weak scalability is especially the case for low end networks such as Fast Ethernet. Figure 8 shows that the execution time of the algorithm is clearly dominated by the communication.
Low Level Parallelization of Nonlinear Diffusion Filtering Algorithms
489
It can be concluded that using Myrinet, the communication overhead is negligible for our algorithms. On SCI and FastEthernet, communication is still the major bottleneck. Therefore mesh partitioning, solely optimized for a low interconnect usage, is the method of choice on the slower FastEthernet and SCI interconnects. Slice decomposition does not scale well on these interconnects.
5
Conclusion
We have shown the potential of cluster computing for parallelization of a nonlinear diffusion algorithm. Two major versions have been implemented and analyzed. With different variants, they offer solutions for both environments with highest network performance, as well as systems which are only loosely coupled. Our current focus is to generate an autonomous initial mapping based on cluster characteristics. This includes the processor performance as well as network features. Another investigation is to support heterogeneous computing resources in which processors obtain tasks based on their raw performance and current load. This will lead to an adaptive method which will enable the application to perform most efficient under several environments ranging from networks of workstations to tightly integrated systems. Acknowledgements. We want to thank Andr´es Bruhn and Timo Kohlberger for their valuable contributions, Joachim Weickert and Christoph Schn¨ orr for the excellent cooperation in our joint project, and Tobias Jakob for his analysis of various data distribution schemes. Thanks go also to the operators of above clusters for granting us access to their systems. This work was sponsored by the DFG unter project number Schn 457/4-1.
References [1] J. Weickert, B.M. ter Haar Romeny, M.A. Viergever. Efficient and reliable schemes for nonlinear diffusion filtering, IEEE Transactions on Image Processing, Vol. 7, 398–410, 1998. [2] J. Weickert, J. Heers, C. Schn¨ orr, K.J. Zuiderveld, O. Scherzer, H.S. Stiehl, Fast parallel algorithms for a broad class of nonlinear variational diffusion approaches, Real-Time Imaging, Vol. 7, 31–45, 2001. [3] P. Perona and J. Malik. Scale space and edge detectioin using anisotropic diffusion IEEE Trans. Pattern Anal. Mach. Intell. 12, 629–639, 1990. [4] T. Lu, P. Neittaanmaki and X.-C. Tai. A parallel splitting up method for partial differential equations and its application to Navier–Stokes equations, RAIRO Mathematical Models and Numerical Analysis 26(6), 673–708, 1992. [5] J. Weickert Anisotropic diffusion in image processing Teubner, 1998 [6] Y. Ishikawa, H. Tezuka, A. Hori, S. Sumimoto, T. Takahashi, F. O’Carroll, and H. Harada. RWC PC Cluster II and SCore Cluster System Software – High Performance Linux Cluster. In Proceedings of the 5th Annual Linux Expo, pages 55–62, 1999.
490
D. Slogsnat et al.
[7] Message Passing Interface MIT Press, 1994. [8] N. J. Boden, D. Cohen, R. E. Felderman, A. E. Kulawik, C. L. Seitz, J. N. Seizovic and W. Su, Myrinet: A Gigabit-per-Second Local Area Network, IEEE Micro, Vol. 15, 29–36, 1995 [9] IEEE Std for Scalable Coherent Interface (SCI). Inst. of Electrical and Electronical Eng., Inc., New York, NY 10017, IEEE std 1596-1992, 1993. [10] F. Petrini, A. Hoisie, W. Feng, and R. Graham. Performance Evaluation of the Quadrics Interconnection Network. In Workshop on Communication Architecture for Clusters (CAC ’01), San Francisco, CA, April 2001. [11] F. Catt´e, P. L. Lions, J. M. Morel, and T. Coll. Image selective smoothing and edge detection by nonlinear diffusion. SIAM J. Num. Anal., 29(1):182–193, 1992. [12] K. Solchenbach. EMP: Benchmarking the Balance of Parallel Computers. SPEC Workshop on Benchmarking Parallel and High-Performance Computing Systems, Wuppertal, Germany, Sept. 13, 1999 [13] Ulrich Br¨ uning, Holger Fr¨ oning, Patrick R. Schulz, Lars Rzymianowicz. ATOLL: Performance and Cost Optimization of a SAN Interconnect. IASTED Parallel and Distributed Computing and Systems (PDCS), Nov. 4–6, 2002, Cambridge, USA [14] A. Bruhn, T. Jakob, M. Fischer, T. Kohlberger, J. Weickert, U. Br¨ uning, and C. Schn¨ orr. Designing 3-d nonlinear diffusion filters for high performance cluster computing. Pattern Recognition, Proc 24th DAGM Symposium, volume 2449 of Lect. Not. Comp. Sci., pages 290–297, Z¨ urich, Switzerland, 2002. Springer.
Implementation of Adaptive Control Algorithms in Robot Manipulators Using Parallel Computing Juan C. Fern´ andez1 , Vicente Hern´andez2 , and Lourdes Pe˜ nalver3 1
Dept. de Ingenier´ıa y Ciencia de los Computadores, Universidad Jaume I, 12071-Castell´ on (Spain), Phone: +34-964-728265; Fax: +34-964-728435,
[email protected] 2 Dept. de Sistemas Inform´ aticos y Computaci´ on, Universidad Polit´ecnica de Valencia, 46071-Valencia (Spain), Tel: +34 96 3877356, Fax: +34 963877359,
[email protected] 3 Dept. de Inform´ atica de Sistemas y Computadores, Universidad Polit´ecnica de Valencia, 46071-Valencia (Spain), Phone: +34-96-3877572; Fax: +34-96-3877579,
[email protected] Abstract. The dynamics equation of robot manipulators is non linear and coupled. An inverse dynamic control algorithm that requires a full knowledge of the dynamics of the system is one way to solve the control movement. Adaptive control is used to identify the unknown parameters (inertial parameters, mass, etc). The adaptive control algorithms are based on the linear relationship of inertial parameters in the dynamic equation. A formulation to generalize this relationship is applied to the Johansson adaptive algorithm. The objective of this paper is to present the implementation of this relationship using parallel computing and apply it to an on-line identification problem in real-time.
1
Introduction
The dynamic equation of robot manipulator torque in open chain is determined by highly coupled and non linear differential equation systems. It is necessary to use approximate or cancelling techniques to apply some control algorithms, such as inverse dynamic control, over the full system. To apply these control techniques it is necessary to know the dynamics of the system. This knowledge allows the existing relations among the different links to be established. The links to establish the kinematic relations of the system are defined from Denavit-Hartenberg parameters. Full knowledge of the dynamics, inertial parameters, mass and inertial moments for each arm, is usually unavailable. These parameters should be estimated using least square or adaptive control techniques. Using adaptive control it is possible to solve both movement control and parameter identification problems. Some of these algorithms can be found in [1,
This work is supported by the CICYT Project TIC2000-1151-C07-06.
H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 491–498, 2003. c Springer-Verlag Berlin Heidelberg 2003
492
Juan C. Fern´ andez, Vicente Hern´ andez, and Lourdes Pe˜ nalver
5,7,4]. One of the problems in applying this kind of algorithm is that of obtaining the relationship ˙ q¨)θ, (1) τ = Yτ (q, q, where Yτ (q, q, ˙ q¨), known as regressor, is an n × r matrix, n being the number of links and r the number of parameters; and θ is the r × 1 parameter vector. Considering all different parameters, r will be 10n. The linear relationship for the adaptive Johansson algorithm using several algebraic properties and definitions given in [6] is employed. This formulation is a computable and general solution for any robot manipulator in open chain using Denavit-Hartenberg parameters and is an extension of the Lagrange-Euler formulation. The main problem of the Lagrange-Euler formulation is its high computational cost, but there are several studies, [8,2], where this is reduced. In this paper the parallel algorithm to obtain the linear relationship for the adaptive Johansson algorithm using the Lagrange-Euler formulation is presented. The structure of the paper is the following: In section two the dynamic model is presented. Section three describes the Johansson adaptive control algorithm. Section four presents the parallel algorithm to obtain the linear relationship for the adaptive Johansson algorithm. The results for a Puma robot are described in section five. And finally the conclusions of this paper are presented in the last section.
2
The Dynamic Model
The dynamic equation of rigid manipulators with n arms in matrix form is τ = D(q)¨ q + h(q, q) ˙ + c(q),
(2)
where τ is the n × 1 vector of nominal driving torques, q is the n × 1 vector of nominal generalized coordinates, q˙ and q¨ are the n × 1 vectors of the first and second derivatives of the vector q respectively, D is the inertia matrix, h(q, q) ˙ is the vector of centrifugal and Coriolis forces and c(q) is the vector of gravitational forces.
3
Johansson Adaptive Control Algorithm
The desired reference trajectory followed by the manipulator is assumed to be available as bounded functions of time in terms of joint accelerations q¨r , angular velocities q˙r , and angular positions qr . A stable non linear reference model is also possible if the errors of accelerations, velocities and positions are defined as e¨ q¨ − q¨r e˙ = q˙ − q˙r . (3) q−qr e The control objective is to follow a given bounded reference trajectory qr , with no position errors e or velocity errors e. ˙ Let Pqq , Ω, S ∈ Rn×n and Pθθ ∈ Rr×r
Implementation of Adaptive Control Algorithms in Robot Manipulators
493
−1 be positive definite matrices and define P12 = Pqq Ω. Let YJ ∈ Rn×r and YJ0 ∈ Rn×1 be defined from the relation
YJ (qr , q, q˙r , q, ˙ q¨r )θ + YJ0 (qr , q, q˙r , q, ˙ q¨r ) 1 ˙ = − D(q)(e˙ + P12 e) + D(q)(¨ qr − P12 e) ˙ + h(q, q) ˙ q˙ + c(q). 2
(4)
T T > 0, Pθθ = Pθθ > 0, Ω = Ω T > 0, S = S T > 0, the For any choice of Pqq = Pqq adaptive control law is given by
3.1
−1 T ˆ˙ r , q, q˙r , q, ˙ q¨r ) = −Pθθ YJ (e˙ + P12 e), θ(q
(5)
ˆ = YJ θˆ + YJ − (S + Pqq Ω −1 Pqq )(e˙ + P12 e)+Pqq e. τ (qr , q, q˙r , q, ˙ q¨r , θ) 0
(6)
Reformulation
Using several algebraic properties and definitions given in [6] it is possible to obtain a computable version of YJ (qr , q, q˙r , q, ˙ q¨r ). Considering u = e˙ + P12 e, the expression for the derivative of the inertial matrix is given by ˙ D(q)u = YDu ˙ (q)θD ˙, where
) rtr (BDu12 ) · · · rtr (BDu1n ) rtr (BDu11 ˙ ˙ ˙ . 0 rtr (BDu22 ) . . rtr (BDu2n ) ˙ ˙ , (q) = YDu ˙ .. .. . 0 0 . ) 0 0 0 rtr (BDunn ˙ T θD˙ = ν(J1 ) ν(J2 ) · · · ν(Jn ) ,
(7)
with BDuij = ˙
j j
(Uij ⊗ Ulkj + Ulij ⊗ Ukj ) q˙l uk .
(8)
(9)
(10)
k=1 l=1
where Jk is the inertia tensor related to link k, Ujk is the effect of the movement of link k on all the points of link j and Ukjl is the effect of the movement of links j and l on all the points of link k. The operator rtr of a matrix is a row vector whose components are the traces of the columns in this matrix, and ν is the operator vector-column of an m × n matrix where the first m components of ν are the components of the first column of the matrix, the second m components of ν are the components of the second column of the matrix, and so on, [6]. Considering v = q¨r − P12 e, ˙ the expression for the inertial matrix is given by D(q)v = YDv (q)θD , where
(11)
494
Juan C. Fern´ andez, Vicente Hern´ andez, and Lourdes Pe˜ nalver
rtr (BDv11 ) rtr (BDv12 ) · · · rtr (BDv1n ) . 0 rtr (BDv22 ) . . rtr (BDv2n ) , θD = θ ˙ , YDv (q) = D .. .. . 0 0 . 0 0 0 rtr (BDvnn )
with BDvik =
k
(Uik ⊗ Ujk )vj .
(12)
(13)
j=1
The expression for the centrifugal and Coriolis forces is given by ˙ h h(q, q) ˙ = Yh (q, q)θ where
rtr (Bh11 ) rtr (Bh12 ) · · · rtr (Bh1n ) . 0 tr (Bh22 ) . . rtr (Bh2n ) , θh = θ ˙ , ˙ = Yh (q, q) D .. .. . 0 0 . 0 0 0 rtr (Bhnn )
with Bhij =
j j
(Uij ⊗ Ulkj )q˙k q˙l .
(14)
(15)
k=1 l=1
The vector of gravitational forces can be expressed as a linear relationship with the inertial parameters c(q) = Yc (q)θc , where
Yc11 0 Yc (q) = − 0 0
T g U11 g T U12 Yc12 · · · Yc1n T Yc22 · · · Yc2n 0 g U22 = − . . 0 0 . . .. 0 0 0 Ycnn 0 0 T θc = m1 r¯1 m2 r¯2 · · · mn r¯n ,
(16) · · · g T U1n · · · g T U2n .. , .. . . T 0 g Unn
(17)
(18)
where r¯i is the position of the centre of mass of link i (mi ) with respect to the origin of coordinates of link i. From the previous results, the computable version of YJ is given by 1 YJ = − YDu ˙ + YDv + Yh + Yc . 2
(19)
The next section describes the parallel algorithm to obtain expressions (5), (6) and (19).
Implementation of Adaptive Control Algorithms in Robot Manipulators
4
495
Parallel Algorithm
The parallel algorithm to obtain (19) is presented below, where n is the number of links and p is the number of processors. To determine the calculations of each processor, two parameters have been defined, ik , the initial link, and fk the final link of processor Pk . Then Pk computes the operations of the links ik , ik+1 , · · · , fk . In this case: – Processor P1 has the values i1 = 1 and f1 = n − p + 1 to obtain the matrices and vectors involved in the Johansson parallel algorithm. – Processors Pk , k = 2 : p, have the values ik = fk = n − p + k to obtain the matrices and vectors involved in the Johansson parallel algorithm. To obtain (19) matrices YDu ˙ , YDv , Yh and Yc can be computed. These matrices are calculated using matrices Uij (structure U Eq. (20)) and matrices Uijk (structure DU Eq. (22)). Matrices Uij mean the effect of movement of link j on all the points of link i, and matrices Uijk mean the effect of movement of links j and k on all the points of link i. The structure U is given by Q1 0 A1 Q1 0 A2 · · · U11 U12 · · · U1n Q1 0 A n 0 U22 · · · U2n A1 Q2 1 A2 · · · 0 A1 Q2 1 An = U = , (20) .. . . .. .. . . . . 0 n−1 An−1 Qn An Unn where Qi is the constant matrix that allows us to calculate the partial derivative of i Aj , the transformation matrix of a robot manipulator. To obtain U , matrices i Aj (structure A Eq. (21)) are necessary. This structure is given by 0 A1 0 A2 · · · 0 An 1 A2 · · · 1 An A= (21) .. , .. . . n−1
An
where i Aj = i Aj−1 j−1 Aj , i = 0 : n − 2. The matrices of the diagonal, i−1 Ai , i = 1 : n, are obtained from the robot parameters [3]. In the parallel algorithm Pk computes i Aj , i = 0 : fk − 1, j = i + 1 : fk . This is the unique case where the values of parameters ik and fk are different from the parameters defined previously, in this case: – Processor P1 has the values i1 = 1 and f1 = n − p + 1 to obtain the matrices of A. – Processors Pk , k = 2 : p, have the values ik = 1 and fk = n − p + k to obtain the matrices of A. Processor Pk needs 0 Ai , i = ik : fk , to obtain matrices U1i , i = ik : fk . To obtain the remaining matrices of U , Uij , j = ik : fk , i = 2 : j, each processor needs 0 Ai−1 and i−1 Aj , j = ik : fk , i = 2 : j. All these matrices have been computed previously.
496
Juan C. Fern´ andez, Vicente Hern´ andez, and Lourdes Pe˜ nalver
To obtain matrices Uijk the following structure, DU , is defined T DU = DU1 DU2 · · · DUn .
(22)
As Ukjl = Ujkl , it is only necessary to compute the following block of DUi , i=1:n
Uiii Uiii+1 0 Uii+1i+1 DUi (i : n, i : n) = . .. .. . 0 0
· · · Uiin · · · Uii+1n .. . .. . .
(23)
· · · Uinn
Given that Pk , k = 1 : p, has the required U matrices, it computes U1ij = Q1 Uij , j = ik : fk , i = 1 : j. To obtain the remaining matrices of DU , Pk , k = 1 : p, computes Uijk , k = ik : fk , i = 2 : k and j = i : k. There are two situations: – To obtain the matrices of row i, Uiij = Vi Qi i−1 Aj , processor Pk needs Vi , i = 2 : fk , and i−1 Aj , j = ik : fk , i = 2 : j. These matrices have been computed previously. – To obtain the matrices of row l > i, Uilj = Uil−1 Ql l−1 Aj , l−1 Aj is first computed by Pk . But Uil−1 has been computed in another processor. In order to avoid communications Pk replicates the computation of this matrix. Then, each processor calculates the block of columns of the structures U ), and DU corresponding to rank [ik : fk ]. Pk , k = 1 : p, computes rtr(BDuij ˙ rtr(BDvij ), rtr(Bhij ) and Ycij , j = ik : fk and i = 1 : j. As the processors have all the information to compute BD˙ , BD and Bh , no communication among them is necessary. To obtain matrix Yc no communication is necessary because each processor has the required U matrices. With this information Pk computes expression (19). To obtain (5), each processor Pk computes the following expression i
˙ −1 T −Pθθii Y jji uj , θˆi =
(24)
j=1
for i = ik : fk . And finally, the control law τ = YJ θˆ − (S + Pqq Ω −1 Pqq )u + Pqq e must be computed. Pθθ and Ω can be considered as diagonal matrices. Each processor Pk computes τi , i = 1 : fk , using the matrices Y jij and θˆi that it has calculated. Each processor sends the computed vector τi to processor Pp . This processor receives these values and it obtains the final value of τ . This is the only communication in the algorithm. The term (S + Pqq Ω −1 Pqq )u + Pqq e is also computed in Pp .
Implementation of Adaptive Control Algorithms in Robot Manipulators
5
497
Experimental Results
The sequential algorithm is evaluated using sequential execution time, Ts . The parallel algorithms are evaluated using parallel execution time Tp (p processors), Speed-up, Sp = T1 /Tp and efficiency, Ep = Sp /p. The results have been obtained using the parameters of a Puma 600 robot with six links. In the parallel algorithms 2, 3 and 4 processors have been used. In each case the links have been distributed among the processors according to ik and fk parameters. To present the results the following notation is used: pxax · · · x, where px is the number of processors used and ax · · · x is the number of links computed by each processor. For example, in p2a31, the first three links are calculated in P1 and the fourth link is computed in P2 . A Beowulf cluster with 32 nodes connected via Myrinet switch has been used. Each node is an Intel Pentium-II processor at 300MHz with 128 MBytes RAM. Communication routines in MPI and C language are used. Table (1) shows the results, in miliseconds, of computing the parallel algorithm of the adaptive Johansson control where Ts = 9.14, using this cluster. Table 1. Experimental results with n = 6 links when a Beowulf cluster is used. Algorithm p2a33 p2a42 p2a51 p3a222 p3a321 p3a411 p4a3111
p 2 2 2 3 3 3 4
Tp Speed-up Efficiency 8.36 1.085 54.67% 7.1 1.286 64.31% 5.446 1.678 83.94% 5.5 1.65 55.33% 4.86 1.88 62.69% 4.86 1.88 62.69% 4.922 1.857 46.43%
The best efficiency is obtained using two processors, where the first processor computes links one to five and the second processor computes the last link. The shortest execution time is obtained using three processors, where the first processor calculates links one to four, the second processor computes the fifth link, and the last link is computed by processor three. Execution time increases when four processors are used because the load balance is not good. The objective is to reduce execution time even though efficiency is not good.
6
Conclusions
Given the generalized formulation for the linear relationship between variable dynamic and inertial terms it is possible to apply this formulation to the Johansson adaptive algorithm using the Lagrange-Euler formulation. Although the use of
498
Juan C. Fern´ andez, Vicente Hern´ andez, and Lourdes Pe˜ nalver
the Lagrange-Euler formulation has a high computational cost, it is possible to reduce it in two ways: – Eliminating the high quantity of null terms and exploiting the properties of the matrices. – Using parallel computing to obtain the different matrices of the dynamic equation and the linear relationship of the Johansson adaptive algorithm. Using these two techniques it is possible to obtain the linear relationship of the Johansson adaptive algorithm and apply it to an on-line identification because we have reduced the time requirements. The two techniques can be used in other adaptive algorithms, such as Slotine-Li, and in optimal control. The shortest execution time is obtained using three processors, and the best efficiency is obtained with two. The parallel algorithm can be used for a robot manipulator with more than two links. In this paper this formulation is applied to a six links Puma manipulator.
References 1. Craig, J.: Adaptive Control of Mechanical Manipulators, Addison-Wesley (1988). 2. Fern´ andez, J.C: Simulaci´ on Din´ amica y Control de Robots Industriales Utilizando Computaci´ on Paralela, Ph.D. Univ. Polit´ecnica de Valencia (1999). 3. Fu, K.S., Gonz´ alez, R.C., Lee, C.S.G: Robotics: Control, Sensing, Vision and Intelligence, New York, McGraw-Hill, 580 pages (1987). 4. Johansson, R.: Adaptive Control of Robot Manipulator Motion, IEEE Transactions on Robotics and Automation, 4(6), 483–490 (1990). 5. Ortega, J.M., Spong M.: Adaptive Motion Control of Rigid Robots: A Tutorial, Automatica 25(6), 877–888 (1989). 6. Pe˜ nalver, L.: Modelado Din´ amico e Identificaci´ on Param´etrica para el Control de Robots Manipuladores, Ph.D. Univ. Polit´ecnica de Valencia (1998). 7. Slotine, J.J., Li, W.: On Adaptive Control of Robot Manipulators, International Journal Robotics Research, 6(3), 49–59 (1987). 8. Zomaya, A.Y.: Modelling and Simulation of Robot Manipulators. A Parallel Processing Approach, World Scientific Series in Robotics and Automated Systems, 8, (1992).
Interactive Ray Tracing on Commodity PC Clusters State of the Art and Practical Applications Ingo Wald, Carsten Benthin, Andreas Dietrich, and Philipp Slusallek Saarland University, Germany
Abstract. Due to its practical significance and its high degree of parallelism, ray tracing has always been an attractive target for research in parallel processing. With recent advances in both hardware and software, it is now possible to create high quality images at interactive rates on commodity PC clusters. In this paper, we will outline the “state of the art” of interactive distributed ray tracing based on a description of the distributed aspects of the OpenRT interactive ray tracing engine. We will then demonstrate its scalability and practical applicability based on several example applications.
1
Introduction
The ray tracing algorithm is well-known for its ability to generate high quality images but has also been infamous for its long rendering times. Speeding up ray tracing for interactive use has been a long standing goal for computer graphics research. Significant efforts have been invested, mainly during the 1980ies and early 90ies, as documented for example in [11]. For users of graphical applications the availability of real-time ray tracing offers a number of interesting benefits: The ray tracing algorithm closely models the physical process of light propagation by shooting imaginary rays into the scene. Thus, it is able to accurately compute global and advanced lighting and shading effects. It exactly simulates shadows, reflection, and refraction on arbitrary surfaces even in complex environments (see Figures 1 and 2). Furthermore, ray tracing automatically combines shading effects from multiple objects in the correct order. This allows for building the individual objects and their shaders independently and have the ray tracer automatically take care of correctly rendering the resulting combination of shading effects (cf. Section 3.1). This feature is essential for robust industrial applications, but is not offered by current graphics hardware. Finally, ray tracing efficiently supports huge models with billions of polygons showing a logarithmic time complexity with respect to scene size, i.e. in the number of triangles in a scene (cf. Section 3.2). This efficiency is due to inherent pixel-accurate occlusion culling and demand driven and output-sensitive processing that computes only visible results. H. Kosch, L. B¨osz¨orm´enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 499–508, 2003. c Springer-Verlag Berlin Heidelberg 2003
500
1.1
I. Wald et al.
Fast and Parallel Ray Tracing
However, ray tracing is a very costly process, since it requires to trace millions of rays into a virtual scene and to intersect those with the geometric primitives. In order to improve performance to interactive rates requires to combine highly optimized ray tracing implementations with massive amounts of computational power. Due to its high degree of parallelism, together with its practical significance for industrial applications (e.g. for lighting simulation, visualization, and for the motion picture industry), especially parallel ray tracing has attracted significant research, not only from the graphics community (e.g. [16,5]), but also from the parallel computing community (e.g. [15]). Muuss et al. [13] and Parker et al. [14] were the first to show that interactive ray tracing is possible by massive parallelization on large shared memory supercomputers. More recently, Wald et al. [19] demonstrated that interactive frame rates can also be achieved on commodity PC clusters. In their research, Wald et. al have accelerated software ray tracing by more than a factor of 15 compared to other commonly used ray tracers. This has been accomplished by algorithmic improvements together with an efficient implementation designed to fit the capabilities of modern processors. For instance, changing the standard ray tracing algorithm to tracing packets of rays and to perform computations within each packet in breadth first order improves coherence, and enables efficient use of data parallel SIMD extensions (e.g. SSE) of modern processors. Paying careful attention to coherence in data access also directly translates to better use of processor caches, which increasingly determines the runtime of today’s programs. The combination of better usage of caches and SIMD extensions is crucial to fully unfold the potential of today’s CPUs, and will probably be even more important for future CPU designs. Even though these optimizations allow some limited amount of interactive ray tracing even on a single modern CPU, one PC alone still cannot deliver the performance required for practical applications, which use complex shading, shadows, reflections, etc. In order to achieve sufficient performance on today´s hardware requires combining the computational resources of multiple CPUs. As has already been shown by Parker et al. [14], ray tracing scales nicely with the number of processors, but only if fast access to scene data is provided, e.g. by using shared memory systems. Today, however, the most cost-effective approach to compute power is a distributed memory PC cluster. Unfortunately they provide only low bandwidth with high latencies for data access across the network. In a related publication, Wald et al. [21] have shown how interactive ray tracing can be realized on such a hardware platform. In the following we briefly discuss the main issues of high-performance implementations in a distributed cluster environment, by taking a closer look at the distribution framework of the OpenRT interactive ray tracing engine [18].
2
Distribution Aspects of the OpenRT Interactive Ray Tracing Engine
Before discussing some of the parallel and distribution details, we first have to take a closer look at the overall system design.
Interactive Ray Tracing on Commodity PC Clusters
2.1
501
General System Design
Client-Server Approach: Even though our system is designed to run distributed on a cluster of PCs, we assume this ray tracer to be used by a single, non-parallel application running on a single PC. As a consequence, we have chosen to follow the usual client/server approach, where a single master centrally manages a number of slave machines by assigning a number of tasks to each client. The clients then perform the actual ray tracing computations, and send their results back to the server in the form of readily computed, quantized color values for each pixel. Screen Space Task Subdivision: Effective parallel processing requires to break the task of ray tracing into a set of preferably independent subtasks. For predefined animations (e.g. in the movie industry), the usual way of parallelization is to assign different frames to different clients in huge render farms. Though this approach successfully improves throughput, it is not applicable to a real-time setting, where only a single frame is to be computed at any given time. For real-time ray tracing, there are basically two approaches: object space and screen space subdivision [16,5]. Object space approaches require the entire scene data-base to be distributed across a number of machines, usually based on an initial spatial partitioning scheme. Rays are then forwarded between clients depending on the next spatial partition pierced by the ray. However, the resulting network bandwidth would be too large for our commodity environment. At today’s ray tracing performance individual rays can be traced much faster than they can be transferred across a network. Finally, this approach often tends to create hot-spots (e.g. at light sources that concentrate many shadow rays), which would require dynamic redistribution of scene data. Instead, we will follow the screen-based approach by having the clients compute disjunct regions of the same image. The main disadvantage of screen-based parallelization is that it usually requires a local copy of the whole scene to reside on each client, whereas splitting the model over several machines allows to render models that are larger than the individual clients’ memories. In this paper, we do not consider this special problem, and rather assume that all clients can store the whole scene. In a related publication however, it has been shown how this problem can be solved efficiently by caching parts of the model on the clients (see [21]). Load Balancing: In screen space parallelization, one common approach is to have each client compute every n-th pixel (so-called pixel-interleaving), or every n-th row or scanline. This usually results in good load balancing, as all clients get roughly the same amount of work. However, it also leads to a severe loss of ray coherence, which is a key factor for fast ray tracing. Similarly, it translates to bad cache performance resulting from equally reduced memory coherence. An alternative approach is to subdivide the image into quadrangular regions (called tiles) and assign those to the clients. Thus, clients work on neighboring pixels that expose a high degree of coherence. The drawback is that the cost for computing different tiles can significantly vary if a highly complex object (such as a complete power plant as shown in Figure 2) projects onto only a few tiles, while other tiles are empty. For static task assignments – where all tiles are distributed among the clients before any actual
502
I. Wald et al.
computations – this variation in task cost would lead to extremely bad client utilization and therefore result in bad scalability. Thus, we have chosen to use a tile-based approach with a dynamic load balancing scheme in our system: Instead of assigning all tiles in advance, corresponding to a data driven approach, we pursue a demand driven strategy by letting the clients themselves ask for work. As soon as a client has finished a tile, it sends its results back to the server, and requests the next unassigned tile from the master. Hardware Setup: To achieve the best cost-effective setting, we have chosen to use a cluster of dual-processor PCs interconnected by commodity networking equipment. Currently, we are using up to 24 dual processor AMD AthlonMP 1800+ PCs with 512 MB RAM each. The nodes are interconnected by a fully switched 100 Mbit Ethernet using a single Gigabit uplink to the master display and application server for handling the large amounts of pixel data generated in each image. Note, that this hardware setup is not even state of the art, as much faster processors and networks are available today. 2.2
Optimization Details
While most of the above design issues are well-known and are applied in similar form in almost all parallel ray tracing systems, many low-level details have to be considered in order to achieve good client utilization even under interactivity constraints. Though we can not cover all of them here, we want to discuss the most important optimizations used in our system. Communication Method: For handling communication, most parallel processing systems today use standardized libraries such as MPI [8] or PVM [10]. Although these libraries provide very powerful tools for development of distributed software, they do not meet the efficiency requirements that we face in an interactive environment. Therefore, we had to implement all communication from scratch with standard UNIX TCP/IP calls. Though this requires significant efforts, it allows to extract the maximum performance out of the network. For example, consider the ’Nagle’ optimization implemented in the TCP/IP protocol, which delays small packets for a short time period to possibly combine them with successive packets to generate network-friendly packet sizes. This optimization can result in a better throughput when lots of small packets are sent, but can also lead to considerable latencies, if a packet gets delayed several times. Direct control of the systems communication allows to use such optimizations selectively: For example, we turn the Nagle optimization on for sockets in which updated scene data is streamed to the clients, as throughput is the main issue here. On the other hand, we turn it off for e.g. sockets used to send tiles to the clients, as this has to be done with an absolute minimum of latency. A similar behavior would be hard to achieve with standard communication libraries. Differential Updates: Obviously, the network bandwidth is not high enough for sending the entire scene to each client for every frame. Thus, we only send differential updates from each frame to the next: Only those settings that have actually changed from the previous frame (e.g. the camera position, or a transformation of an object) will be sent
Interactive Ray Tracing on Commodity PC Clusters
503
to the clients. Upon starting a new frame, all clients perform an update step in which they incorporate these changes into their scene database. Asynchronous Rendering: Between two successive frames, the application will usually change the scene settings, and might have to perform considerable computations before the next frame can be started. During this time, all clients would run idle. To avoid this problem, rendering is performed asynchronously to the application: While the application specifies frame N , the clients are still rendering frame N − 1. Once the application has finished specifying frame N , it waits for the clients to complete frame N −1, displays that frame, triggers the clients to start rendering frame N , and starts specifying frame N + 1. Note, that this is similar to usual double-buffering [17], but with one additional frame of latency. Asynchronous Communication: As just mentioned, the application already specifies the next frame while the clients are still working on an old one. Similarly, all communication between the server and the clients is handled asynchronously: Instead of waiting for the application to specify the complete scene to be rendered, scene updates from the application are immediately streamed from the server to all clients in order to minimize communication latencies. Asynchronously to rendering tiles for the old frame, one thread at the client already receives the new scene settings and buffers them for future use. Once the rendering threads have finished, (most of) the data for the next frame has already arrived, further minimizing latencies. These updates are then integrated into the local scene database, and computations can immediately be resumed without losing time receiving scene data. Multithreading: Due to a better cost/performance ratio, each client in our setup is a dualprocessor machine. Using multithreading on each client then allows to share most data between these threads, so the cost of sending scene data to a client can be amortized over two CPUs. Furthermore, both client and server each employ separate tasks for handling network communication, to ensure minimum network delays. Task Prefetching: If only a single tile is assigned to each client at any time, a client runs idle between sending its results back to the server and receiving the next tile to be computed. As a fast ray tracer can compute tiles in at most a few milliseconds, this delay can easily exceed rendering time, resulting in extremely bad client utilization. To avoid these latencies, we let each client request (“prefetch”) several tiles in advance. Thus, several tiles are ’in flight’ towards each client at any time. Ideally, a new tile is just arriving every time a previous one is sent on to the server. Currently, each client is usually prefetching about 4 tiles. This, however depends on the actual ratio of compute performance to network latency, and might differ for other hardware configurations. Spatio-Temporal Coherence: In order to make best use of the processor caches on the client machines load balancing also considers the spatio-temporal coherence between rays by assigning the same image tiles to the same clients in subsequent frames whenever possible. As soon as a client has processed all of ’its’ old tasks, it starts to ’steal’ tasks from a random other machine.
504
I. Wald et al.
3 Applications and Experiments In the following, we demonstrate the potential and scalability of our system based on several practical examples. If not mentioned otherwise, all experiments will run at video resolution of 640 × 480 pixels. As we want to concentrate on the practical results, we will but closely sketch the respective applications. For more details, also see [18,20,3]. 3.1
Classical Ray Tracing
Plug’n Play Shading: One of the main advantages of ray tracing is its potential for unsurpassed image realism, which results from its ability to support specialized shader programs for different objects (e.g. a glass shader), which can then easily be combined with shaders on other objects in a plug and play manner. This is also the reason why ray tracing is the method of choice of almost all major rendering packages. The potential of this approach can be seen in Figure 1: An office environment can be simulated with several different shaders which fit together seamlessly. For example, a special shader realizing volume rendering with a skull data set seamlessly fits into the rest of the scene: i.e. it is correctly reflected in reflective objects, and casts transparent shadows on all other objects. Using our distribution framework, the ray tracer scales almost linearly to up to 48 CPUs, and achieves frame rates of up to 8 frames per second (see Figure 4). Visualisation of Car Headlights for Industrial Applications: Being able to efficiently simulate such advanced shading effects also allows to offer solutions to practical industrial problems that have so far been impossible to tackle. For example, the limitations of todays graphics hardware force current Virtual Reality (VR) systems to make heavy use of approximation, which in turn makes reliable quantitative results impossible, or at least hard to achieve. Examples are the visualization of reflections of the car’s dash board in the side window where it might interfere with backward visibility through the outside mirror at night. Another example is the high-quality visualization of car headlights that are important design features due to being the “eyes of the car”. Figure 1 shows an 800.000-triangle VRML model of a car headlight rendered with our ray tracing engine [3]. Special reflector and glass shaders are used that carefully model the physical material properties and intelligently prune the ray trees. The latter is essential for achieving real-time frame rates because the ray trees otherwise tend to get very large due to many recursive reflections and refractions. For good visual results we still have to simulate up to 25 levels of recursion for certain Pixels (cf. Figure 1d).
Fig. 1. Classical ray tracing: (a) Typical office scene, with correct shadows and reflections, and with programmable procedural shaders. (b) The same scene with volume and lightfield objects. Note how the volume casts transparent shadows, the lightfield is visible through the bump-mapped reflections on the mirror, etc. (c) The headlight model with up to 25 levels of reflection and refraction. (d) False-color image showing the number of reflection levels per pixel (red: 25+).
Interactive Ray Tracing on Commodity PC Clusters
505
Simulating this level of lighting complexity is currently impossible to compute with alternative rendering methods. Thus, for the first time this tool allows automotive designers and headlight manufacturers to interactively evaluate a design proposal. This process previously took several days for preparing and generating predefined animations on video. Now a new headlight model can be visualized in less than 30 minutes and allows designers the long missing option of freely interacting with the model for exploring important optical effects. 3.2
Interactive Visualization of Highly Complex Models
Apart from its ability to simulate complex lighting situations, another advantage of ray tracing is that in practice it’s time complexity is logarithmic in scene size [12], allowing to easily render even scenes with several million up to billions of triangles. This is of great importance to VR and engineering applications, in which such complex models have to be visualized. Currently, such models have to be simplified before they can be rendered interactively. This usually requires expensive preprocessing [1,2], significant user intervention, and often negatively affects the visualization quality. With our distributed ray tracing system, we can render such models interactively (see Figures 2 and 4), and – even more importantly – without the need for geometric simplifications. Figure 2 shows screenshots from two different models: The image on the left shows three complete power plants consisting of 12.5 million triangles each. Using our ray tracer, we can easily render several such power plants at the same time, and can even interactively move parts of the power plant around. Especially the latter is very important for design applications, but is usually impossible with alternative technologies, as simplification and preprocessing ususally work only in static environments. Yet another advantage of ray tracing is its possibility to use instantiation, the process of re-using parts of a model several times in the same scene. For example, the second model in Figure 2 consists of only ten different kinds of sunflowers (of roughly 36,000 triangles each) and one type of tree, which are then instantiated several thousand times to form a complete landscape of roughly one billion triangles. Furthermore, the scene is rendered including transparency textures for the leaves, and computes even pixel-accurate shadows cast by the sun onto the leaves (see Figures 2c and 2d). Using our framework, both scenes can be rendered interactively, and achieve almost-linear scalability, as can be seen in Figures 2 and 4.
Fig. 2. Complex models: (a) Three powerplants of 12.5 million individual triangles each, rendering interactively at 23 fps. (b) A closeup on the highly detailed geometry. (b) An outdoor scene consisting of roughly 28,000 instances of 10 different kinds of sunflowers with 36,000 triangles each together with several multi-million-triangle trees. The whole scene consists of roughly one billion triangles and is rendered including shadows and transparency. (d) A closeup of the highly detailed shadows cast by the sun onto the leaves.
506
3.3
I. Wald et al.
Interactive Lighting Simulation
Even though classical ray tracing as described above considers only direct lighting effects, it already allows for highly realistic images that made ray tracing the preferred rendering choice for many animation packages. The next step in realism can be achieved by including indirect lighting effects computed by global illumination algorithms as a standard feature of 3D graphics (see Figure 3). Global illumination algorithms account for the often subtle but important effects of indirect lighting effects in a physically-correct way [6,7] by simulating the global light transport between all mutually visible surfaces in the environment. Due to the need for highly flexible visibility queries, virtually all algorithms today use ray tracing for this task. Because of the amount and complexity of the computations, rendering with global illumination is usually even more complex than classical ray tracing, and thus slow and far from interactive, taking several minutes to hours even for simple diffuse environments. The availability of real-time ray tracing should now enable to compute full global illumination solutions also at interactive rates (see Figure 4). We will not cover the details of the algorithm here, which are described in close detail in [20] and [4]. Using our system, we can interactively simulate light propagation in a virtual scene, including soft shadows, reflections and indirect illumination. As each frame is recomputed from scratch, interactive changes to the environment can be handled well, allowing to modify geometry, lights and materials at interactive rates.
4
Results and Conclusions
In this paper, we have shown how efficient parallelization of a fast ray tracing kernel on a cluster of PCs can be used to achieve interactive performance even for high-quality applications and massively complex scenes. We have sketched the parallelization and distribution aspects of the OpenRT distributed interactive ray tracing engine [18], which uses screen-space task subdivision and demand-driven load balancing, together with low-level optimization techniques to minimize bandwidth and to hide latencies. The combination of these techniques allow the system to be used even in a real-time setting, where only a few milliseconds are available for each frame. We have evaluated the performance of the proposed system (see Figure 4) in a variety of test scenes, for three different applications: Classical ray tracing, visualising massively complex models, and interactive lighting simulation: Due to the limited bandwidth of our display server (which
Fig. 3. An interactive global illumination application in different environments. From left to right: (a) A conference room of 280.000 triangles, (b) The “Shirley 6” scene, with global illumination including complex procedural shaders, (c) An animated VRML model, and (d) global illumination in the power plant with 37.5 million triangles. All scenes render at several frames per second.
Interactive Ray Tracing on Commodity PC Clusters
507
frames per second
20
15
10
Office Headlight Power Plant Sunflowers Conference Room (Global Illumination) Shirley 6 (Global Illumination) Power Plant (Global Illumination)
5
0 1
8
16
24
32
40
48
CPUs
Fig. 4. Scalability of our system on a cluster of up to 48 CPUs. Note that the system is limited to about 22–24 frames per second due to the limited network connection of the display server. Up to this maximum framerate, all scenes show virtually linear scalability.
is connected to the Gigabit uplink of the cluster), our framerate is limited to roughly 22–24 frames per second at 640×480 pixels, as we simply cannot transfer more pixels to the server for display. Up to this maximum framerate however, all scenes show virtually linear scalability for up to 48 CPUs (see Figure 4). The presented applications have also shown that a fast and distributed software implementation of ray tracing is capable of delivering completely new types of interactive applications: While todays graphical hardware solutions (including both the newest high-performance consumer graphics cards and expensive graphics supercomputers) can render millions of triangles per second, they can not interactively render whole scenes of many million to even billions of triangles as shown in Section 3.2. Furthermore, such hardware architectures can not achieve the level of image quality and simulation quality that we have shown to be achievable with our system (see Section 3.1). This is especially true for interactive lighting simulation (Section 3.3), which – due to both its computational cost and its algorithmic complexity – is unlikely to be realized on graphics hardware any time soon. With this capability of enabling completely new applications, our system provides real value for practical applications, and is already being used in several industrial projects. Still, typical VR applications demand even higher resolutions and even higher framerates. Typically, resolutions up to 1600 × 1200 are required to drive equipment such as PowerWalls etc, and ’real-time’ applications usually demand frame rates of 25 frames per second and more. This leaves enough room for even more parallelization, and also requires to eventually think about more powerful network technologies (such as Myrinet [9]) to provide the required network performance.
Acknowledgements. The RTRT/OpenRT project has been supported by Intel Corp. The Sunflowers and Powerplant models have been graciously provided by Oliver Deussen and Anselmo Lastra.
508
I. Wald et al.
References 1. D. Aliaga, J. Cohen, A. Wilson, E. Baker, H. Zhang, C. Erikson, K. Hoff, T. Hudson, W. St¨urzlinger, R. Bastos, M. Whitton, F. Brooks, and D. Manocha. MMR: An Interactive Massive Model Rendering System Using Geometric and Image-Based Acceleration. In ACM Symposium on Interactive 3D Graphics, pages 199–206, Atlanta, USA, April 1999. 2. William V. Baxter III, Avneesh Sud, Naga K Govindaraju, and Dinesh Manocha. Gigawalk: Interactive Walkthrough of Complex Environments. In Rendering Techniques 2002, pages 203 – 214, June 2002. (Proceedings of the 13th Eurographics Workshop on Rendering 2002). 3. Carsten Benthin, Ingo Wald, Tim Dahmen, and Philipp Slusallek. Interactive Headlight Simulation – A Case Study for Distributed Interactive Ray Tracing. In Proceedings of Eurographics Workshop on Parallel Graphics and Visualization (PGV), pages 81–88, 2002. 4. Carsten Benthin, Ingo Wald, and Philipp Slusallek. A Scalable Approach to Interactive Global Illumination. to be published at Eurographics 2003, 2003. 5. Alan Chalmers, Timothy Davis, and Erik Reinhard, editors. Practical Parallel Rendering. AK Peters, 2002. ISBN 1-56881-179-9. 6. Micheal F. Cohen and John R. Wallace. Radiosity and Realistic Image Synthesis. Morgan Kaufmann Publishers, 1993. ISBN: 0121782700. 7. Philip Dutre, Kavita Bala, and Philippe Bekaert. Advanced Global Illumination. In SIGGRAPH 2001 Course Notes, Course 20. 2001. 8. MPI Forum. MPI – The Message Passing Interface Standard. http://www-unix.mcs.anl.gov/mpi. 9. Myrinet Forum. Myrinet. http://www.myri.com/myrinet/overview/. 10. Al Geist, Adam Beguelin, Jack Dongarra, Weicheng Jiang, Robert Manchek, and Vaidyalingam S. Sunderam. PVM: Parallel Virtual Machine. A User’s Guide and Tutorial for Network Parallel Computing. MIT Press, Cambridge, 1994. 11. Andrew Glassner. An Introduction to Raytracing. Academic Press, 1989. 12. Vlastimil Havran. Heuristic Ray Shooting Algorithms. PhD thesis, Czech Technical University, 2001. 13. Michael J. Muuss. Towards Real-Time Ray-Tracing of Combinatorial Solid Geometric Models. In Proceedings of BRL-CAD Symposium ’95, June 1995. 14. Steven Parker, Peter Shirley,Yarden Livnat, Charles Hansen, and Peter Pike Sloan. Interactive Ray Tracing. In Proceedings of Interactive 3D Graphics (I3D), pages 119–126, 1999. 15. Tomas Plachetka. Perfect Load Balancing for Demand-Driven Parallel Ray Tracing. In B. Monien and R. Feldman, editors, Lecture Notes in Computer Science, pages 410–419. Springer Verlag, Paderborn, August 2002. (Proceedings of Euro-Par 2002). 16. Erik Reinhard. Scheduling and Data Management for Parallel Ray Tracing. PhD thesis, University of East Anglia, 1995. 17. B. Schachter. Computer Image Generation. Wiley, New York, 1983. 18. Ingo Wald, Carsten Benthin, and Philipp Slusallek. OpenRT - A Flexible and Scalable Rendering Engine for Interactive 3D Graphics. Technical report, Saarland University, 2002. Available at http://graphics.cs.uni-sb.de/Publications. 19. Ingo Wald, Carsten Benthin, Markus Wagner, and Philipp Slusallek. Interactive Rendering with Coherent Ray Tracing. Computer Graphics Forum, 20(3):153–164, 2001. (Proceedings of Eurographics 2001). 20. Ingo Wald, Thomas Kollig, Carsten Benthin, Alexander Keller, and Philipp Slusallek. Interactive Global Illumination using Fast Ray Tracing. Rendering Techniques 2002, pages 15–24, 2002. (Proceedings of the 13th Eurographics Workshop on Rendering). 21. Ingo Wald, Philipp Slusallek, and Carsten Benthin. Interactive Distributed Ray Tracing of Highly Complex Models. Rendering Techniques 2001, pages 274–285, 2001. (Proceedings of the 12th Eurographics Workshop on Rendering).
Toward Automatic Management of Embarrassingly Parallel Applications Inˆes Dutra1 , David Page2 , Vitor Santos Costa1 , Jude Shavlik2 , and Michael Waddell2 1
Dep of Systems Engineering and Computer Science Federal University of Rio de Janeiro, Rio de Janeiro, RJ, Brazil {ines,vitor}@cos.ufrj.br 2 Dep of Biostatistics and Medical Informatics, University of Wisconsin-Madison, USA {page,shavlik,mwaddell}@biostat.wisc.edu
Abstract. Large-scale applications that require executing very large numbers of tasks are only feasible through parallelism. In this work we present a system that automatically handles large numbers of experiments and data in the context of machine learning. Our system controls all experiments, including re-submission of failed jobs and relies on available resource managers to spawn jobs through pools of machines. Our results show that we can manage a very large number of experiments, using a reasonable amount of idle CPU cycles, with very little user intervention.
1
Introduction
Large-scale applications may require executing very large numbers of tasks, say, thousands or even hundreds of thousands of experiments. These applications are only feasible through parallelism and are nowadays often executed in clusters of workstations or in the Grid. Unfortunately, running these applications in an unreliable environment can be a complex problem. The several phases of computation in the application must be sequenced correctly: dependencies, usually arising through data written to and read from files, must be respected. Results will be grouped together, a summarised report over the whole computation should be made available. Errors, both from the application itself and from the environment, must be handled correctly. One must check whether experiments terminated successfully, and verify integrity of the output. Most available software for monitoring applications in parallel and distributed environments, and more recently, in grid environments, concentrate on modelling and analysing hardware and software performance [8], prediction of lost cycles [9] or visualisation of parallel execution [12], to mention some. Most of them focus on parallelised applications. Few efforts have been spent on managing huge number of independent experiments and the increasing growth of interdisciplinary databases such as the ones used in biological or biomedical applications. Only recently, we have seen work in the context of the Grid such as H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 509–516, 2003. c Springer-Verlag Berlin Heidelberg 2003
510
I. Dutra et al.
the GriPhyN project for Physics [1], and the development of the general purpose system Chimera [6]. In this work we present a system originally designed to support the very large numbers of experiments and data in machine learning applications. We are interested in machine learning toward data mining of relational data from domains such as biochemistry [10,4], and security [13]. Machine learning tasks are often computationally intensive. For instance, many learning systems, such as the ones that generate decision trees and logical clauses, must explore a socalled “search space” in order to find models that characterise well a set of correct (and possibly incorrect) examples. The size of the search space usually grows exponentially with the size of the problem. A single run of the learning algorithm can thus easily take hours over real data. Moreover, in order to study the effectiveness of the system one often needs to repeat experiments on different data. Splitting the original examples into different subsets or f olds and learning on each fold is also common. Thus, a single learning task may involve several independent coarse-grained tasks, providing an excellent source of parallelism. In fact, parallelism is often the only way one can actually perform such experiments in reasonable time. Our system has successfully run several large-scale experiments. Our goal is to extract maximum advantage of the huge parallelism available in machine learning applications. The system supports non-interactive error-handling, including re-submission of failed jobs, while allowing users to control the general performance of the application. Whenever possible, we rely on available technology: for example, we use the Condor resource manager [2] to spawn jobs through pools of machines. We have been able to run very large applications. One example included over 50 thousand experiments: in this case we consumed about 53,000 hours of CPU, but the system took only 3 months to terminate, achieving peak parallelism of 400 simultaneous machines at a time, and requiring very little user intervention. The paper is organised as follows. First, we present in more detail the machine learning environment and its requirements. We then discuss the motivation for an automatic tool. In section 2 we present the architecture of our system and the methodology applied to the two machine learning phases: experimentation and evaluation. We then discuss some performance figures and possibilities of enhancements. Last, we offer our conclusions and suggest future work.
2
A Tool for Managing Large Numbers of Experiments
In order to be able to run the many possible combinations of experiments one needs to perform to have statistically meaningful results in machine learning applications, in a feasible time, and handle the results with least possible user intervention to avoid manual errors, we developed a tool for job management of learning applications, currently supporting Linux and Solaris environments. This tool makes use of available resource manager systems to manage idle resources available. We address the following issues: Data Management: each experiment
Toward Automatic Management of Embarrassingly Parallel Applications
511
will have its own output and temporary files, but several experiments share input files. The system needs to create a directory structure such that the output and temporary files for individual experiments can be kept on separate, but easily accessed, directories for each experiment. Control : given a problem description our tool creates a set of build files to launch the actual jobs. Each script inputs the data required for a specific experiment, and sets the files that will be output. Task Supervision: Our tool must allow one to inspect successful and unsuccessful job termination. As discussed in more detail next, the probability that some jobs will fail is extremely high, so most cases of unsuccessful termination should be handled within the system. User Interface: a large number of results must be collected and displayed. Throughout, the user should be able to understand what is the current status of the computation, and plot the results. In the experiments presented in this paper, we used as a resource manager, the Condor system, developed at the Computer Sciences Department of the University of Wisconsin-Madison. Condor is a specialised workload management system for compute-intensive jobs. Like other full-featured batch systems, Condor provides a job queueing mechanism, scheduling policy, priority scheme, resource monitoring, and resource management. Among other things, Condor allows transparent migration of jobs from overloaded machines to idle machines and checkpointing, which permits that jobs can restart in another machine without the need to start from the beginning. These are typical tasks of a resource manager. Putting it all Together. Our architecture is composed of two main modules: (1) the Experimentation Module and (2) the Evaluation Module. The Experimentation Module is responsible for setting up the environment to start running experiments for one or several applications, for launching jobs for tuning and cross-validation, and checking if results are terminated and complete. Once all experiments have terminated, the Evaluation Module will consult the tuning results, if any, compute accuracy functions, and plot the results. If some job leaves the queue, but the output result is incorrect or corrupted, the check termination program will re-submit the job either to tuning or to cross-validation. We provide a user-friendly web interface in to the system with options for choosing different learners and learning techniques. At the moment, our system deals only with Inductive Logic Programming [11] learners, but this interface is easily extensible to other learners. Not Everything is Neat. Several problems may arise while processing thousands or hundreds of thousands of experiments. We classify these problems as either hardware-dependent or software-dependent. As we use off-the-shelf technology to launch our jobs, we expect such technology will deal correctly with hardware-dependent problems. In fact, the Condor system we use in our experiments deals with dynamic hardware changes such as inclusion of a new machine in the network or a machine or network failure. In the case of a new machine being included in the network, Condor updates the
512
I. Dutra et al.
central manager table to mirror this change. In the case of a machine fault, Condor employs a checkpointing mechanism to migrate the job to another “healthy” machine. With respect to software-dependent problems, we can enumerate several sources of possible failure: jobs get lost because a daemon stops working for some reason, a job breaks because of lack of physical memory, bugs in the machine learning system, bugs in the resource manager, or even bugs at the operating system level, corrupted data due to network contention, or lack of disk space. None of the systems that we use is totally stable: during our experiments, it has happened that the operating system was upgraded while we were performing our experiments, and the task management software was incompatible with the new upgrade. Unforeseen situations, common in large software development, can lead the execution to crash. Problems can arise from any one of these components: the machine learning system, in some cases the software that runs the machine learning system, and the resource manager. As we rely in off-the-shelf components to build our system, some of these problems will be outside our control. We have found memory management to be a critical issue for our application. The machine learning system that we use relies on dynamic memory allocation: thus, we do not know beforehand how much memory we use, and we can expect memory requirements to grow linearly with execution time. In general, we set the experiments to only run in machines with a minimal amount of memory, say M . If we set M too low, many experiments will fail. If we set M too high, we will severely restrict parallelism. Moreover, as machines with lots of memory are likely to be more recent and powerful machines, we can expect them to be busier. In practice we can be sure that runs will fail. Our approach to deal with these problems is to have a daemon that inspects the job queues, and the application output files. So, if the daemon detects that some output file is not yet generated, and the job responsible for that output is no longer in the queue, it will re-submit the job. If some output file is not yet generated and the job is taking too long to produce an output, the daemon will remove the job from the queue and resubmit the job. An output file can be generated, but be corrupted. In that case, we also need to check for output syntax to be sure the next phase will collect correct data. Note that these steps do not need any human intervention,1 and do not require any change to the available software being used.
3
Performance Results
We ran three sets of experiments in a period of 6 months. The experiments concerned three relational learning tasks using Inductive Logic Programming techniques. The tasks included two biocomputing problems, one concerning the 1
Of course, if the failure rate becomes excessive, the user is informed and is allowed to terminate the experiments.
Toward Automatic Management of Embarrassingly Parallel Applications
513
prediction of carcinogenicity tests on rodents, and the second concerning a characterisation of genes involved in protein metabolism by means of their correlation with other genes. The third task used data on a set of smuggling events. The goal was to detect whether two events were somehow related. We ran our jobs in the Condor pools available at the Biostatistics and Medical Informatics and the Computer Sciences Departments of the University of Wisconsin-Madison. We used PC machines running Linux and SUN machines running Solaris. Throughout, Condor collects statistics about the progress of the jobs, but as this consumes a large amount of disk storage, not all Condor pools keep all data about jobs. Therefore in Table 1 we show statistics only for one of our pools that stores the data information about jobs progress in disk. The first column shows the month when the experiments were running. The second column shows the total number of CPU hours spent by our jobs. The third column shows the average number of jobs running, followed by the average of jobs idle, and the peak of jobs running and idle. We can observe that the system has two activity peaks in June/July and in September. During September in average the tool was able to keep almost one hundred processors busy, just in this pool. The maximum number of jobs running in this pool was around 400. The execution priority of jobs in Condor depends on the configuration and on the pool. If a job migrates to a remote pool of machines, the priority of this job may be lowered to guarantee that local users in the remote pool are not interfered by the “foreign” jobs. The statistics shown in Table 1 are related to a remote pool. Table 1. Statistics of Jobs Period Tot Alloc Time JobsRunning JobsIdle JobsRunning JobsIdle (Hours) Average Average Peak Peak May 639.6 12.8 337.6 61.0 1057.0 Jun 39375.5 67.3 5322.8 400.0 15312.0 Jul 18406.2 70.1 2243.9 207.0 13366.0 Aug 110.1 14.9 0.1 17.0 4.0 Sep 18283.8 93.5 6713.9 397.0 11754.0 Oct 1185.8 47.0 4027.3 122.0 5933.0
In order to better understand the table, Figure 1 shows the state of the above mentioned pool during the most active month, September. The X-axis corresponds to days of the week, and the Y-axis corresponds to total machines available in the pool. The red area (bottom colour in the figure) shows usercontrolled machines, the blue area in the middle, idle machines, and the green area above shows machines running Condor tasks. Pool size varies between 700 and 900 machines, most of them Linux and Solaris machines. User activity is higher during week days, where users are working on their own machine. Condor most often keeps around 400 to 500 machines busy at any one time. This provides
514
I. Dutra et al.
an insight into the maximum amount of parallelism we can take advantage of. Comparing with the results in Table 1 we can observe that there are points in time where we could actually take advantage of almost the full pool, even if the pool was shared with the whole campus.
Fig. 1. The Condor Pool
Figure 2 shows that even during week days the system can provide an average of about 200 processors for long periods of time. This is about half of what we could expect from the cluster. Although we have little control over the cluster and we may be preempted by local tasks, even at the worst moments the pool can achieve above 50 machines simultaneously running our jobs. Last, error handling was a very important issue in practice. From the 45,000 experiments we launched (three lots of 15,000 experiments for each application), around 20% failed for several reasons, and had to be re-submitted. This was done without user intervention.
4
Conclusions and Future Work
We described an automatic tool to manage large amounts of experiments in the machine learning context. The main advantage of such a tool is to provide a highlevel user interface that can hide details of embarrassingly parallelisation, and that can allow for more automatic management of user experiments. Our system is capable of launching jobs automatically, check their integrity and termination, re-submit corrupted jobs, evaluate the results, plot relevant data, and inform the user about location of data and graphs. A second advantage of our tool is that we integrate off-the-shelf components in order to take advantage of already popular technologies. We used our tool to run several learning experiments with multirelational data. As an example, one of our experiments consumed about 53,000 hours of CPU, using a peak of 400 machines simultaneously.
Toward Automatic Management of Embarrassingly Parallel Applications
515
Fig. 2. Condor Activity
Most software available for parallel and distributed environments, and more recently, for grid environments, is designed to monitor applications by modelling and performance analysis, rather than managing a huge number of experiments. Condor has a limited form of handling experiments by allowing the user to express dependencies through the DAGMan tool [7]. But this tool requires that the user has some knowledge of ClassAds [14], a specification language for jobs and resources, in order to express dependencies explicitly. Chimera [6], one of the components of the Globus project [5], also has forms of automatically launching jobs depending on output produced by other jobs. Contrary to DAGMan, Chimera automatically generates a dependency execution graph, based on VDL (Virtual Data Language) [3] specifications. Our proposal is to require minimum interference from the user, especially because many of the users in computational biology, where machine learning systems are heavily applied, have none or shallow knowledge of how to use a programming language. As future work we intend to extend the tool to support other learning algorithms, such as the ones supported by the WEKA toolkit [15], and parallel boosting. We have been working on integrating our system with Chimera, in order to take advantage of one of the nicest features of Chimera that is to automatically infer dependencies between jobs in order to launch them without any user intervention.
Acknowledgments. This work was supported by DARPA EELD grant number F30602-01-2-0571, NSF Grant 9987841, NLM grant NLM 1 R01 LM07050-01, and CNPq. We would like to thank the Biomedical Group support staff and the Condor Team at the Computer Sciences Department for their invaluable help with Condor, Ashwin Srinivasan for his help with the Aleph system and the Carcinogenesis benchmark, and Yong Zhao from University of Chicago who has been helping us to integrate Chimera to our system.
516
I. Dutra et al.
References 1. The GriPhyN Project. White paper available at http://www.griphyn.org/ documents/white paper/index.php, 2000. 2. J. Basney and M. Livny. Managing network resources in Condor. In Proceedings of the Ninth IEEE Symposium on High Performance Distributed Computing (HPDC9), Pittsburgh, Pennsylvania, pages 298–299, Aug 2000. 3. A. Chervenak, I. Foster, C. Kesselman, C. Salisbury, and S. Tuecke. The data grid: Towards an architecture for the distributed management and analysis of large scientific datasets. Journal of Network and Computer Applications, 23:187–200, 2001. 4. I. Dutra, D. Page, V. Santos Costa, and J. Shavlik. An empirical evaluation of bagging in inductive logic programming. In Proceedings of the Twelfth International Conference on Inductive Logic Programming, Sydney, Australia, July 2002. Springer-Verlag. 5. I. Foster, C. Kesselman, J. Nick, and S. Tuecke. Grid services for distributed system integration. Computer, 35(6), 2002. 6. I. Foster, J. V¨ ockler, M. Wilde, and Y. Zhao. Chimera: A virtual data system for representing, querying, and automating data derivation. In Proceedings of the 14th conference on Scentific and Statistical Database Management (2002), 2002. 7. James Frey. Condor DAGMan: Handling Inter-Job Dependencies. http://www.cs.wisc.edu/condor/dagman/, 2002. 8. D. Gunter, B. Tierney, B. Crowley, M. Holding, and J. Lee. NetLogger: A Toolkit for Distributed System Performance Analysis. In Proceedings of the 8th International Workshop on Modeling, Analysis and Simulation of Computer and Telecommunication Systems (MASCOTS ’00), 2000. 9. W. Meira Jr., T. LeBlanc, and A. Poulos. Waiting time analysis and performance visualization in carnival. In SPDT96: SIGMETRICS Symposium on Parallel and Distributed Tools, Philadelphia, PA, ACM, pages 1–10, May 1996. 10. R. King, S. Muggleton, and M. Sternberg. Predicting protein secondary structure using inductive logic programming. Protein Engineering, 5:647–657, 1992. 11. N. Lavrac and S. Dzeroski. Inductive Logic Programming: Techniques and Applications. Artificial Intelligence. Ellis Horwood (Simon & Schuster), 1994. 12. B. Miller, M. Callaghan, J. Cargille, J. Hollingsworth, R. Irvin, K. Karavanic, K. Kunchithapadam, and T. Newhall. The paradyn parallel performance measurement tool. IEEE Computer, 28(11):37–46, November 1995. 13. R. Mooney, P. Melville, L. P. Rupert Tang, J. Shavlik, I. Dutra, D. Page, and V. Santos Costa. Relational data mining with inductive logic programming for link discovery. In Proceedings of the National Science Foundation Workshop on Next Generation Data Mining, Baltimore, Maryland, USA, 2002. 14. R. Raman, M. Livny, and M. Solomon. Matchmaking: Distributed resource management for high throughput computing. In Proceedings of the Seventh IEEE International Symposium on High Performance Distributed Computing, Chicago, IL, July 1998. 15. Ian H. Witten and Eibe Frank. Data Mining: Practical Machine Learning Tools and Techniques with Java Implementations. Morgan Kaufmann, 1999.
Comparing Two Long Biological Sequences Using a DSM System Renata Cristina F. Melo, Maria Emília Telles Walter, Alba Cristina Magalhaes Alves Melo, Rodolfo Batista, Marcelo Nardelli, Thelmo Martins, and Tiago Fonseca Department of Computer Science, Campus Universitario - Asa Norte, Caixa Postal 4466, University of Brasilia, Brasilia – DF, CEP 70910-900, Brazil {renata, mia, albamm, rodolfo, thelmo, tiago, marcelo}@cic.unb.br
Abstract. Distributed Shared Memory systems allow the use of the shared memory programming paradigm in distributed architectures where no physically shared memory exist. Scope consistent software DSMs provide a relaxed memory model that reduces the coherence overhead by ensuring consistency only at synchronisation operations, on a per-lock basis. Sequence comparison is a basic operation in DNA sequencing projects, and most of sequence comparison methods used are based on heuristics, that are faster but do not produce optimal alignments. Recently, many organisms had their DNA entirely sequenced, and this reality presents the need for comparing long DNA sequences, which is a challenging task due to its high demands for computational power and memory. In this article, we present and evaluate a parallelisation strategy for implementing a sequence alignment algorithm for long sequences in a DSM system. Our results on an eight-machine cluster presented good speedups, showing that our parallelisation strategy and programming support were appropriate.
1
Introduction
Distributed Shared Memory (DSM) is an abstraction that allows the use of the shared memory programming paradigm in parallel or distributed architectures. The first DSM systems tried to give parallel programmers the same guarantees they had when programming uniprocessors and this approach created a huge coherence overhead [10]. To alleviate this problem, researchers have proposed to relax some consistency conditions, thus creating new shared memory behaviours that are different from the traditional uniprocessor one. In the shared memory programming paradigm, synchronisation operations are used every time processes want to restrict the order in which memory operations should be performed. Using this fact, hybrid Memory Consistency Models (MCM) guarantee that processors only have a consistent view of the shared memory at synchronisation time [10]. This allows a great overlapping of basic read and write operations that can lead to considerable performance gains. By now, the most popular MCMs for DSM systems are Release Consistency [2] and Scope Consistency [6]. H. Kosch, L. Böszörményi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 517–524, 2003. © Springer-Verlag Berlin Heidelberg 2003
518
R.C.F. Melo et al.
JIAJIA is a scope consistent software DSM system proposed by [4] that implements consistency on a per-lock basis. When a lock is released, modifications made inside the critical section are made visible to the next process that acquires the same lock. On a synchronisation barrier, however, consistency is globally maintained and all processes are guaranteed to see all past modifications to the shared data. In DNA sequencing projects, researchers want to compare two sequences to find similar portions of them and obtain good local sequence alignments. In practice, two families of tools for searching similarities between two sequences are widely used BLAST [1] and FASTA[13], both based on heuristics and used for comparing long sequences. To obtain optimal local alignments, the most widely used algorithm is the one proposed by Smith-Waterman [12], with quadratic time and space complexity. Many works are known that implement the Smith-Waterman algorithm for long sequences of DNA. Specifically, parallel implementations were proposed using MPI [9] or specific hardware [3]. As far as we know, this is the first attempt to use a scope consistent DSM system to solve this kind of problem. In this article, we present and evaluate a parallelisation strategy for implementing the Smith-Waterman algorithm in a DSM system . Work is assigned to each processor in a column basis with a two-way lazy synchronisation protocol. An heuristic described in [11] was used to reduce the space complexity. The results obtained in an eight-machine cluster with large sequence sizes show good speedups when compared with the sequential algorithm. For instance, to align two 400KB sequences, a speedup of 4.58 was obtained, reducing the execution time from more than 2 days to 10 hours. The rest of this paper is organised as follows. Section 2 describes the sequence alignment problem and the optimal algorithm to solve it. In Section 3, DSM systems are presented. Section 4 describes our sequential and parallel algorithm. Some experimental results are discussed in Section 5. Finally, Section 6 concludes the paper.
2
Smith-Waterman’s Algorithm for Local Sequence Alignment
To compare two sequences, we need to find the best alignment between them, which is to place one sequence above the other making clear the correspondence between similar characters or substrings from the sequences [11]. We define alignment as the insertion of spaces in arbitrary locations along the sequences so that they finish with the same size. Given an alignment between two sequences s and t, an score is associated for them as follows. For each column, we associate +1 if the two characters are identical, -1 if the characters are different and –2 if one of them is a space. The score is the sum of the values computed for each column. The maximal score is the similarity between the two sequences, denoted by sim(s,t). In general, there are many alignments with maximal score. Figure 1 gives and example. Smith-Waterman [12] proposed an algorithm based on dynamic programming. As input, it receives two sequences s, |s|=m, and t, |t|=n. There are m+1 possible prefixes
Comparing Two Long Biological Sequences Using a DSM System G G
A A
T
C C
G G
G G
A A
T A
T T
A A
G G
+1
+1
–2
+1
+1
+1
+1
–1
+1
+1
+1
519
= 6
Fig. 1. Alignment of the sequences s= GACGGATTAG and t=GATCGGAATAG, with the score for each column. There are nine columns with identical characters, one column with distinct character and one column with a space, giving a total score 6 = 9*(+1)+1*(-1) + 1*(-2)
for s and n+1 prefixes for t, including the empty string. An array (m+1)x(n+1) is built, where the (i,j) entry contains the value of the similarity between two prefixes of s and t, sim(s[1..i],t[1..j]). Fig. 2 shows the similarity array between s=AAGC and t=AGC. The first row and column are initialised with zeros. The other entries are computed using Equation 1. sim ( s[1..i ], t[1.. j − 1]) − 2 sim ( s[1..i − 1], t[1.. j − 1]) + p (i, j ) sim ( s[1..i ], t[1.. j ]) = max sim ( s[1..i − 1], t[1.. j ]) − 2 0.
(1)
In equation 1, p(i,j) = +1 if s[i]=t[j] and –1 if s[i]t[j]. If we denote the array by a, the value of a[i,j] is the similarity between s[1..i] and t[1..j], sim(s[1..i],t[1..j]).
A A G C
0 0 0 0 0
A 0 1 1 0 0
G 0 0 0 2 0
C 0 0 0 0 3
Fig. 2. Array to compute the similarity between the sequences s=AAGC and t=AGC.
We have to compute the array a row by row, left to right on each row, or column by column, top to bottom, on each column. Finally arrows are drawn to indicate where the maximum value comes from, according to Equation 1. Figure 3 presents the basic dynamic programming algorithm for filling the array a. Algorithm Similarity Input: sequences s and t Output: similarity between s and t |s| m |t| n 0 to m do For i ixg a[i, 0] For j 0 to n do jxg a[0, j]
Ä Ä
Ä Ä Ä Ä For i Ä 1 to m do For j Ä 1 to n do a[i, j] Ä max( a[i
–1, j] –2, a[i –1, j –1] ±1, a[i, j –1] –2, 0)
Return a[m, n]
Fig. 3. Basic dynamic programming algorithm to build a similarity array a.
An optimal alignment between two sequences can be obtained as follows. We begin in a maximal value in array a, and follow the arrow going out from this entry until
520
R.C.F. Melo et al.
we reach another entry with no arrow going out, or until we reach an entry with value 0. Each arrow used gives us one column of the alignment. An horizontal arrow leaving entry (i,j) corresponds to a column with a space in s matched with t[j], a vertical arrow corresponds to s[i] matched with a space in t and a diagonal arrow means s[i] matched with t[j]. An optimal alignment is constructed from right to left. The detailed explanation of this algorithm can be found in [11]. Many optimal alignments may exist for two sequences because many arrows can leave an entry. The time and space complexity of this algorithm is 0(m n), and if both sequences 2 have approximately the same length, n, we get O(n ).
3
Distributed Shared Memory Systems
Distributed Shared Memory offers the shared memory programming paradigm in a distributed environment where no physically shared memory exists. DSM is often implemented a single paged, virtual address space over a network of computers that is managed by the virtual memory system [8]. Local references usually proceed without the interference of the DSM system and only generate exceptions by protection fault. When a non resident page is accessed, a page fault is generated and the DSM system is contacted to fetch the page from a remote node. The instruction that caused the page fault is restarted and the application can proceed. In order to improve performance, DSM systems usually replicate pages. Maintaining strong consistency among the copies was the approach used by the first DSM systems but it created a huge coherence overhead.[7] Relaxed memory models aim to reduce this overhead by allowing replicas of the same data to have, for some period of time, different values [10]. By doing this, relaxed models provide a programming model that is complex since, at some moments, the programmer is conscious of replication. Hybrid memory models are a class of relaxed memory models that postpone the propagation of shared data modifications until the next synchronisation point [10]. These models are quite successful in the sense that they permit a great overlapping of basic memory operations while still providing a reasonable programming model. Release Consistency (RC) [2] and Scope Consistency (ScC) [6] are the most popular memory models for software DSM systems. The goal of Scope Consistency (ScC) [6] is to take advantage of the association between synchronisation variables and ordinary shared variables they protect. In Scope Consistency, executions are divided into consistency scopes that are defined on a per lock basis. Only synchronisation operations and data accesses that are related to the same synchronisation variable are ordered. The association between shared data and the synchronisation variable that guards them is implicit and depends on program order. Additionally, a global synchronisation point can be defined by synchronisation barriers. JIAJIA [4] is an example of scope consistent software DSM. JIAJIA implements the Scope Consistency memory model with a write-invalidate multiple-writer home-based protocol. In JIAJIA, the shared memory is distributed among the nodes in a NUMA-architecture basis. Each shared page has a home node. A
Comparing Two Long Biological Sequences Using a DSM System
521
page is always present in its home node and it is also copied to remote nodes on an access fault. There is a fixed number of remote pages that can be placed at the memory of a remote node. When this part of memory is full, a replacement algorithm is executed. Each lock is assigned to a lock manager. The functions that implement lock acquire, lock release and synchronisation barrier in JIAJIA are jia_lock, jia_unlock and jia_barrier, respectively [5]. Additionally, JIAJIA provides condition variables that are accessed by jia_setcv and jia_waitcv, to signal and wait on conditions, respectively. The programming style provided is SPMD (Single Program Multiple Data) and each node is distinguished from the others by a global variable jiapid [5].
4 Parallel Algorithm to Compare DNA Sequences To analyse the performance of our parallel algorithm, we implemented a sequential variant of the algorithm described in Section 2 that uses two linear arrays [11]. The bidimensional array was not used since, for large sequences, the memory overhead would be prohibitive. In this algorithm, we simulate the filling of the bi-dimensional array just using two rows in memory, since, to compute entry a[i,j] we just need the values of a[i-1,j], a[i-1,j-1] and a[i,j-1]. So, the space complexity of this version is 2 linear, O(n). The time complexity remains O(n ). The algorithm works with two sequences s, |s|=m and t, |t|=n. First, one linear array is initialised with zeros. Then, each entry of the second array is obtained from the first one with the algorithm described in Section 2, but using a single character of s on each step. We denote a[i,j]=sim(s[1..i,1..j]) as current score. Each entry also contains: initial and final alignment coordinates, maximal and minimal score, gaps, matches and mismatches counters and a flag showing if the alignment is a candidate to be an optimal alignment. When computing the a[i,j] entry, all the information of a[i-1,j], a[i-1,j-1] or a[i,j-1] is passed to the current entry. The gaps, matches and mismatches counters are employed when the current score of the entry being computed comes from more than one previous entry. In this case, they are used to define which alignment will be passed to this entry. We use an expression (2*matches counter + 2*mismatches counter + gaps counter) to decide which entry to use [9]. The greater value is considered as the origin of the current entry. If the values are still the same, our preference will be to the horizontal, to the vertical and at last to the diagonal arrow, in this order. At the end of the algorithm, the coordinates of the best alignments are kept on the queue alignments. This queue is sorted and the repeated alignments are removed. The best alignments are then reported to the user. The access pattern presented by this variant of the Smith-Waterman algorithm leads to a non-uniform amount of parallelism. The parallelisation strategy that is traditionally used in this case is the “wave-front method” since the calculations that are done in parallel evolve as waves on diagonals. We propose a parallel version of this variant where each processor p acts on two rows, a writing and a reading row. Work is assigned in a column basis, i.e., each proc-
522
R.C.F. Melo et al.
essor calculates only a set of columns on the same row, as shown in figure 4. Synchronisation is achieved by locks and condition variables provided by JIAJIA [5,6]. Barriers are only used at the beginning and at the end of computation. In figure 4, p0 starts computing and, when value a1,3 is calculated, it writes this value at the shared memory and signals p1, that is waiting on jia_waitcv. At this moment, p1 reads the value from shared memory, signals p0, and starts calculating from a1,4 . P0 proceeds then calculating elements a2,1 to a2,3 When this new block is finished, p0 issues a jia_waitcv to guarantee that the preceeding value was already read by p1. The same protocol is executed by every processor pi and processor pi+1. P0 A
P1
A
T
C
G
P2 G
C
T
P3 C
A
T
G
C
a1,1 a1,2
a1,3
a1,4 a1,5 a1,6
a1,7 a1,8 a1,9
a1,10 a1,11 a1,12
A
a2,1 a2,2
a2,3
a2,4 a2,5 a2,6
a2,7 a2,8 a2,9
a2,10 a2,11 a2,12
A
a3,1 a3,2
a3,3
a3,4 a3,5 a3,6
a3,7 a3,8 a3,9
a3,10 a3,11 a3,12
a4,1 a4,2
a4,3
a4,4 a4,5 a4,6
a4,7 a4,8 a4,9
a4,10 a4,11 a4,12
T
Shared Data
Fig. 4. Work assignment in the parallel algorithm. Each processor p is assigned N/P rows, where P is the total number of processors and N is the length of the sequence.
5
Experimental Results
Our parallel algorithm was implemented in C, using the software DSM JIAJIA v.2.1 on top of Debian Linux 2.1. We ran our experiments on a dedicated cluster of 8 Pentium II 350 MHz, 160 MB RAM connected by a 100Mbps switch. Our tests used real DNA sequences obtained from www.ncbi.nlm.nih.gov/PMGifs/Genomes. Five sequence sizes were considered (15KB, 50KB, 80KB, 150KB and 400KB). Execution times and speedups for these sequences, with 1,2,4 and 8 processors are shown in Table 1. Speedups were calculated considering the total execution time and thus include times for initialisation and collecting results. Table 1. Total execution times (seconds) and speedups for 5 sequence comparisons Size 15K x 15K 50K x 50K 80K x 80K 150K x 150K 400K x 400K
Serial Exec 296 3461 7967 24107 175295
2 proc Exec /Speedup 283.18/1.04 2884.15/1.20 6094.19/1.31 19522.95/1.23 141840.98/1.23
4 proc Exec /Speedup 202.18/1.46 1669.53/2.07 3370.40/2.46 10377.89/2.32 72770.99/2.41
8 proc Exec /Speedup 181.29/1.63 1107.02/3.13 2162.82/3.68 5991.79/4.02 38206.84/4.58
As can be seen in table 1, for small sequence sizes, e.g. 15K, very bad speedups are obtained since the parallel part is not long enough to surpass the amount of synchronisation inherent to the algorithm. As long as sequence sizes increase, better speedups
Comparing Two Long Biological Sequences Using a DSM System
523
100% 80%
computation
60%
communication
40%
lock+cv
20%
barrier
0% 15K 50K 80K 150K 400K
Fig. 5. Execution time breakdown for 5 sequence sizes, containing the relative time spent in computation, communication, lock and condition variable and barrier.
are obtained. This effect can be better noticed in figure 5, which presents a breakdown of the execution time of each sequence comparison. We also compared the results obtained by our implementation (denoted GenomeDSM) with BlastN, FASTA and PipMaker [14]. For this task, we used two 50KB mithocondrial genomes, Allomyces acrogynus and Chaetosphaeridium globosum. In table 2, we present a comparison among these four programs, showing the alignments with the best scores found by GenomeDSM. Still in table 2, the second and third best alignments were not found by FASTA. In FASTA, the query sequence had to be broken, since our version of FASTA did not compute sequences greater that 20KB. Thus, the lack of these two sequence alignments can be due to this limitation. Table 2. Comparison among results obtained by GenomeDSM, BlastN, FASTA and PipMaker Alignment 1 Alignment 2 Alignment 3
Begin End Begin End Begin End
GenomeDSM (39109, 55559) (39839, 56252) (39475, 48905) (39755, 49188) (28637, 47919) (28753, 48035)
BlastN (39099, 55549) (39196, 55646) (39522, 48952) (39755, 49005) (28667, 47949) (28754, 48036)
FASTA (38396, 55317) (39840, 56673) -
PipMaker (38396, 54897) (39828, 56239) (39617, 49050) (39756, 49189) (28505, 47787) (28756, 48038)
We also developed a tool to visualise the alignments found by GenomeDSM. An example can be seen in figure 6.
Fig. 6. Visualisation of the alignments generated by GenomeDSM with the 50KB sequences. Plotted points show the similarity regions between the two genomes.
Martins et al. [9] presented a version of the Smith-Waterman algorithm using MPI that ran on a Beowulf system with 64 nodes each containing 2 processors. Speedups
524
R.C.F. Melo et al.
attained were very close to ours, e.g., for 800Kx500K sequence alignment, a speedup of 16.1 were obtained for 32 processors.
6
Conclusions and Future Work
In this paper, we proposed and evaluated a DSM implementation of the SmithWaterman algorithm that solve the DNA local sequence alignment problem. Work is assigned to each processor in a column basis and the wavefront method was used. The results obtained in an 8-machine cluster present good speedups which are improved as long as the sequence lengths increase. To compare sequences of 400KB, we obtained a 4.58 speedup on the total execution time, reducing execution time of the sequential algorithm from 2 days to 10 hours. This shows that that our parallelisation strategy and the DSM programming support were appropriate to our problem. As future work, we intend to port the algorithm implemented in MPI proposed in [9] to our cluster and compare its results with ours. Also, we intend to propose and evaluate a variant of our approach, which will use variable block size.
References l. 2.
3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14.
S. F. Altschul et al. – Gapped BLAST and PSI-BLAST: a new generation of protein database search programs. Nucleic Acids Research, v. 25, n. 17, p. 3389–3402, 1997. K. Gharachorloo et al, "Memory Consistency and Event Ordering in Scalable SharedMemory Multiprocessors", Proc. Int. Symp. On Computer Architecture, May, 1990, p. 15– 24. L. Grate, M. Diekhans, D. Dahle, R. Hughey, Sequence Analysis With the Kestrel SIMD Parallel Processor –1998. W. Hu., W. Shi., Z. Tang.: JIAJIA: An SVM System Based on A New Cache Coherence Protocol. In Proc. of HPCN’99, LNCS 1593, pp. 463–472, Springer-Verlag, April, 1999. W.Hu, W.Shi, “JIAJIA User´s Manual”, Technical report, CAS – China, 1999. Iftode L., Singh J., Li K.: Scope Consistency: Bridging the Gap Between Release Consisth tency and Entry Consistency, Proc. Of the 8 ACM SPAA’96, June, 1996, pages 277–287. Lamport L., How to Make a Multiprocessor Computer that Correctly Executes Multiprocess Programs, IEEE Transactions on Computers, 1979, 690–691. K. Li, “Shared Virtual Memory on Loosely Coupled Architectures”, PhDThesis, Yale, 1986. W. S. Martins, et al., A Multithread Parallel Implementation of a Dynamic Programming Algorithm for Sequence Comparison, Proc. SBAC-PAD, 2001, Pirenopolis, Brazil, p.1–8. Mosberger D.: Memory Consistency Models, Operating Systems Review, p. 18–26, 1993. J. C. Setubal, J. Meidanis, Introduction to Computational Molecular Biology. Pacific Grove, CA, United States: Brooks/Cole Publishing Company, 1997. T. F. Smith, M. S. Waterman, Identification of common molecular sub-sequences – Journal of Molecular Biology, 147 (1) 195-197–1981. W. R. Pearson; D. L. Lipman, Improved tools for biological sequence comparison. Proceedings Of The National Academy Of Science USA, v. 85, p. 2444–2448, April 1988. Schwartz Et Al. – PipMaker – A Web Server for Aligning Two Genomic DNA Sequences – Genome Research 10:577-586, April 2000 – http://bio.cse.psu.edu.
Two Dimensional Airfoil Optimisation Using CFD in a Grid Computing Environment Wenbin Song, Andy Keane, Hakki Eres, Graeme Pound, and Simon Cox School of Engineering Sciences University of Southampton Highfield, Southampton, SO17 1BJ, UK {w.song, ajk, eres, gep, sjc}@soton.ac.uk http://www.geodise.org
Abstract. In this paper, a two-dimensional airfoil shape optimisation problem is investigated using CFD within a grid computing environment (GCE) implemented in Matlab. The feature-based parametric CAD tool ProEngineer is used for geometry modelling. The industrial level mesh generation tool Gambit and flow solver Fluent are employed as remote services using the Globus Toolkit as the low level API. The objective of the optimisation problem is to minimize the drag-to-lift coefficient ratio for the given operating condition. A Matlab interface to the design exploration system (OPTIONS) is used to obtain solutions for the problem. The adoption of grid technologies not only simplifies the integration of proprietary software, but also makes it possible to harness distributed computational power in a consistent and flexible manner.
1
Introduction
Computational fluid dynamics (CFD) has been constantly developed over the past few decades and now both commercial and in-house codes can provide more and more robust and accurate results. Combined with the use of wind tunnel test data, CFD can be used in the design process to drive geometry change instead of being used mainly as a design validation tool. This aspect can be further exploited by bringing optimisation tools into the design process. Automation of the design process can significantly shorten the design cycle and lead to better designs compared to previous manual design modification approaches. Such manual approaches are still adopted by most engineers due to various reasons: lack of robustness and flexibility in automating the design process, the high computational cost associated with large numbers of iterations of high fidelity simulation codes, difficulties of collobaration in a heterogeneous computational environments, etc. In fact, the revolution brought by the World Wide Web with respect to information sharing has not yet delivered fundamental changes to engineering design practice for a number of reasons, including security problems in collaborative environments. The emerging Grid computing technologies [1] aim to fill this gap in providing a general architecture for building a geographically distributed, collaborative computational environment. H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 525–532, 2003. c Springer-Verlag Berlin Heidelberg 2003
526
W. Song et al.
Design search and optimisation is a process that can be used to improve the design of complex engineering products, systems, and processes. There are a large number of numerical optimisation methods available for this purpose, along with various strategies that provide even more complex scenarios for design search such as hybrid methods incorporating machine learning methods. In order to evaluate the strength and/or weakness of different designs, it is often necessary to use complicated computational tools such as CFD. One of the important aspects of these tools is the high computational cost related to the solution of large numbers of simultaneous algebraic equations, such as in computational fluid dynamics or structural analysis. The combination of the repetitive nature of search processes and the constant increase in demand for computational power of high-fidelity computational models has prompted much effort in the study of various aspects of the problem solving environment. For example, approximation techniques are often used to provide surrogate models for the high fidelity codes and to decouple the strong interactions between codes from different disciplines. However, an efficient approximation framework often requires capabilities to start the analysis codes on-demand for points anywhere in the parameter space and mining new data into existing datasets. It is clear that the strong coupling between optimisation methods and domain codes partly limits the ability to prototype different complicated search strategies, and thus impedes the wider use of optimisation technologies in the engineering design offices on a daily basis. The aim of this paper is to provide an exemplar of CFD-based shape optimisation using emerging grid technologies that address some of these issues. The paper is organized as follows. The next section gives a brief introduction on grid computing technology with a focus on the architecture and various techniques used. The third section defines a two-dimensional airfoil shape optimisation problem. The optimisation method used and some results are given in section four, with concluding remarks and future work described in section five.
2
Grid Computing
The design of increasingly complex engineering systems relies on knowledge drawn from various disciplines, and a multidisciplinary approach to tackle the interrelations between different domains. The availability of a generic infrastructure addressing the integration of software packages with a rich user interface is vital. Emerging grid computing techniques seem to be able to provide a general approach for integration and collaboration while retaining the division and autonomy of disciplinary domain experts. To address the organizational challenges that prevent a wider application of a multidisciplinary optimisation approach in a generic manner, a flexible environment supporting a powerful scripting language, rich visualization tools, and common mathematical computation capabilities is desirable. In this work, Matlab is chosen as the central stage for the problem solving environment, as it provides a broad spectrum of functions and algorithms for a wide range of applications including visualization and Graphical User In-
Two Dimensional Airfoil Optimisation Using CFD
527
terface (GUI) building features. However, our approach allows that the toolkits developed can be easily integrated with other environments such as Python. [2] The overall architecture adopted in this paper is shown in Figure 1, which is a simplified version of the general Grid-Enabled Optimisation and Design Search for Engineering (GEODISE) architecture [3] (http://www.geodise.org/). In the current implementation, Globus Toolkit 2.2 [4] is used to provide various low-level functionalities such as authentication, resource management, job submission, etc. These functionalities can then be exposed to end users via the interface to commodity technologies, such as Java, Python, Perl, etc.
Fig. 1. Overall structure of the grid-enabled design optimisation
In the Geodise project, The Java Cog Kit [5] is used to expose Globus functionalities in the Matlab environment. Matlab is widely used in academia and industry to prototype algorithms, and to analyse and visualize data. Matlab also enables programmers to access Java classes, and therefore provides the functionality for code re-use and further developments. Detailed discussion on how various technologies such as Condor, Globus, etc., are used in Geodise can be found in [6]. Via the use of the Matlab functions provided in the Geodise Toolkit, the user is also able to submit his/her own code to computing resources, or to run software packages installed on the server. Access to database functionalities such as file archiving and retrieval and user notifications are also provided in the form of Matlab functions. Table 1 lists the functions implemented in the current version of the Geodise toolkit.
528
W. Song et al. Table 1. Implemented commands in the Geodise toolkits gd gd gd gd gd gd gd gd gd gd gd gd gd gd
3
archive proxyinfo proxyquery createproxy destroyproxy getfile putfile jobsubmit jobstatus jobpoll jobkill listjobs query retrieve
Stores a file in repository with associated metadata Returns information about the user’s proxy certificate Queries whether a valid proxy certificate exists Creates a proxy certificate using user’s credentials Destroys the local copy of user’s proxy certificate Retrieves a file from a remote host using GridFTP Transfers a file to a remote host using GridFTP Submits a GRAM job to a Globus server Returns the status of the GRAM job specified a job handle Queries the status of a Globus GRAM job until complete Terminates the GRAM job specified by a job handle Returns job handles for all GRAM jobs Query metadata about a file based on certain criteria Retrieves a file from the repository to the local machine
Two Dimensional Airfoil Design Using Orthogonal Basis Functions
A two-dimensional airfoil design optimisation problem appropriate for concept wing design is studied here, using the Geodise toolkit. Instead of using airfoil coordinates directly, as is the case in many airfoil design applications, six basis airfoil functions are used to form a set of orthogonal functions to define the airfoil shape [7]. The goal of the optimisation is to minimize the drag/lift ratio by changing the weights of the six basis functions and the thickness-to-chord ratio. The basis airfoil functions were derived from a family of nine NASA SC(2) supercritical airfoils. The first three of these six basis functions are shown in Figure 2. A similar approach of using basis functions for defining airfoil shape was adopted by Ahn J. et al. [8], however, the basis functions used were not orthogonal and not derived from airfoil shapes. The use of orthogonal basis functions leads to a unique mapping from parameter space to the airfoil geometries, which is a desirable feature when optimisation methods are adopted to search for good designs. The airfoil geometry is defined first by upto six weight coefficients, followed by the adjustment to the specified thickness-to-chord ratio. A detailed discussion on geometry definition, meshing and solution is given in the next sections. 3.1
Problem and Geometry Definition
Shape parameterization methods in the context of multidisciplinary design optimisation have been investigated by a number of researchers. A comprehensive overview can be found in [9]. A combined shape, topology and configuration optimisation process for structures was recently reported in [10], in which a parametric model was constructed using the CAD tool ProEngineer. In most cases, the transfer of geometry between codes is implemented using standard neutral formats such as STEP, IGES, PATRAN, etc. Here, a parametric airfoil
Two Dimensional Airfoil Optimisation Using CFD
529
Fig. 2. First three airfoil basis functions used for airfoil geometry definition
geometry has been defined using ProEngineer. In this case the basis functions have been normalized with respect to the first member of the set and the thickness to chord ratio. Moreover, it has been shown from [7] that by adopting this approach good representation of the original airfoil can be recovered by simply varying the thickness to chord ratio and second weight, leaving the first function weight at unity and the remaining four at zero. This leads to a two-dimensional specification which is very simple to implement and interpret. These parameters, their initial values and ranges are listed in Table 2. 3.2
Mesh Generation and CFD Solution
To carry out optimization, mesh generation must be carried out in a batch mode, based on the geometry imported from ProEngineer. However, in general, a lot of work is needed to clean up the geometry in order to create robust, efficient meshes, to remove undesired features such as very short edges and surfaces (via surface merge or removal of short edges), and this process is often interactive in nature involving much graphic picking. Unique tagging can be used to replace the graphic picking and to deal with topology changes. However, this is not always possible, especially for complex models generated using a top-down design approach. To begin with here, an interactive session was used to generate a script file that is later used to run the meshing program in batch mode. The limitation of this approach is that the geometry topology must be maintained the same, and also that when a different version of meshing tool is to be used, another
530
W. Song et al. Table 2. Design variable for the airfoil optimisation problem
Name Lower bound Inital value Upper bound Meaning w (2) -1.0 -0.98 1.0 Weight for the second basis function tc ratio 0.06 0.08 0.18 Thickness-to-chord ratio
interactive session may be needed to generate the script file. The generated mesh file is then imported into Fluent [11]. Node spacing is specified on the edges to control the mesh density. A triangular mesh is used for the flow field. Boundary layer meshes are attached to the upper and lower edges of the airfoil. A Reynolds-averaged two-dimensional compressible Navier-Stokes solver is used to obtain the flow field solutions. Meshing jobs are submitted onto the computing server using the gd jobsubmit function after a valid proxy has been established using the user’s credentials. A unique job handle is returned and later used to check the status of the meshing job. Fluent CFD jobs are then submitted using the same mechanism. Results are stored into data files on the scratch directory on the server. After the jobs finish, gd getfile is used to retrieve the results files and values of lift and drag coefficients are obtained by analyzing the resulting files.
Fig. 3. Landscape of lift/drag coefficient ratio for airfoils with two design variables
4
Optimisation Using Genetic Algorithm
An implementation of a population-based genetic algorithm (GA) [12] is adopted in the search process due to its robustness in finding the location of globally optimum designs. However, the design search is not conducted on the original
Two Dimensional Airfoil Optimisation Using CFD
531
landscape of the objective function, instead, a surrogate model is used in lieu of the true objective function. The surrogate model was constructed from results evaluated at twenty sample points in design space using a random Latin hypercube sampling technique. The landscape of the lift/drag ratio is shown in Figure 3 based on the results at these twenty points. Here, a kriging surrogate model [13] is constructed and searched using the Genetic algorithm. The best design found on the krigging surrogate model is then validated using a full CFD solution. The resultant airfoil shape and values of design variables and objective function are given in Figure 4. It can be seen that using a surrogate model can significantly speed up the search process and enables the exploration of multiple search methods, which would be impossible on the high fidelity codes. Furthermore, discrepancies between the landscapes of the true objective function and the approximated one could be addressed by evaluating the approximation error or introducing some adaptive update scheme based on the criteria of expected improvements. It is expected that such consideration will improve the effectiveness of approximation methods, and this will be studied in the future work.
Fig. 4. Initial and Optimized airfoil (left) and corresponding pressure distribution (right). For the initial and optimized design, the weight for the second basis function is -0.9815/-0.5985, the thickness-to-chord ratio is 0.116/0.0716, and lift/drag ratio is 7.8363/14.9815
5
Concluding Remarks and Future Work
A two-dimensional airfoil design optimisation using CFD has been tackled using emerging grid technologies. Although the problem does not require as much computing power as a full-wing CFD analysis, it has demonstrated all the elements in a typical CFD-based shape optimisation problem. The Geodise computation toolkit is used in the study, along with a Genetic Algorithm and surrogate modeling methods. Grid technologies are still fast evolving, and the Open Grid Service
532
W. Song et al.
Architecture (OGSA) [14] seems to be the future direction for grid services developments. Other issues may involve the use of a job scheduler for job farming and monitoring. Additionally, a rich GUI is desired for novice designers as well as the scripting language for expert users and further developments. More complicated geometries and approximation and optimisation frameworks will also be studied in due course. Acknowledgements. This work is supported by the UK e-Science Pilot project (UK EPSRC GR/R67705/01). The authors gratefully acknowledge many helpful discussions with the Geodise team. The authors also wish to thank Dr. Neil Bressloff and Mr. Andras Sobester for helpful discussions.
References [1] Foster I., Kesselman C., Tuecke S.: The Anatomy of the Grid: Enabing Scalable Virtual Organizations. International Journal of Supercomputer Applications 15 (3) (2001) 200–222 [2] Python(2003) http://www.python.org [3] Cox, S.J., Fairman, M.J., Xue, G., Wason, J.L., Keane, A.J.: The GRID: Computational and data resource sharing in engineering optimisation and design search.(2001) IEEE Computer Society Press [4] The Globus Project (2003) http://www.globus.org [5] Laszewski, G., Foster I., Gawor, J., Lane P.: A Java Commodity Grid Toolkit. Concurrency: Practice and Experience 13 (8-9) (2001) 643–662 [6] Pound, G., Eres, M.H., Wason, J.L., Jiao, Z., Cox, S.J., Keane, A.J.: A Gridenabled Problem solving environment (PSE) for design Optimisation within MATLAB.(2003) International Parallel and Distributed Processing Symposium, April 22–26, 2003, Nice, France [7] Robinson, G.M., Keane, A.J.: Concise Orthogonal Representation of Supercritical Airfoils. Journal of Aircraft 38 (3) (2001) 580–583 [8] Ahn, J., Kim, H.-J., Lee, D.-H., Rho, O.-H.: Response Surface Method for Airfoil Design in Transonic Flow. Journal of Aircraft 38 (2) (2001) 231–238 [9] Samareh, J.A.: Survey of shape parameterization techniques for high-fidelity multidisciplinary shape optimisation. AIAA Journal 39 (5) (2001) 877–884 [10] Langer, H., Puhlhofer, T., Baier, H.: An approach for shape and topology optimization integrating CAD parameterization and evolutionary algorithms. (2002) [11] Fluent (2003) http://www.fluent.com [12] Keane, A.J.: OPTIONS Design exploration system, (2003) http://www.soton.ac.uk/˜ajk/options.ps [13] Jones D.R., Schonlau, M., Welch W.J.: Efficient global optimisation of expensive black-box functions. Journal of global optimisation 13 (1998) 455–492 [14] Foster, I., Kesselman C., Nick, J., Tuecke, S.: The Physiology of the Grid: An Open Grid Services Architecture for Distributed Systems Integration. Open Grid Service Infrastructure WG, Global Grid Forum, June 22, 2002
Applied Grid Computing: Optimisation of Photonic Devices Duan H. Beckett1 , Ben Hiett1 , Ken S. Thomas1 , and Simon J. Cox2 1
2
Department of Electronics and Computer Science, University of Southampton, SO17 School of Engineering Sciences, University of Southampton, SO17 1BJ
Abstract. In this paper, we present an application of grid computing to solve an important industrial problem: that of optimising the band gap of photonic crystals, which are an important technology in future generation telecomms and sensing. The computational power grid enabled months of experimentation to be performed in a weekend. Of particular interest was the necessity to run jobs on both Linux and Windows resources.
1
Introduction
Applications such a sharp angle waveguides, wave division multiplexing, single mode lasers and sensors have motivated research into photonic crystals; which are nanostructured devices that filter light at particular frequencies. Photonic crystals are modeled by finding periodic solutions to Maxwell’s equations [1], [2], [3]. A typical finite element analysis will comprise: mesh generation for the domain, selection of suitable basis functions and the solution of a generalised eigenvalue problem A(k)c = λBc. (1) The eigenvalue problem is solved for a large sample of quasi momentum vectors k in the Brillouin zone, because the spectrum of the original problem in R2 is the union of the spectrum of (1) over k ∈ B [3]. The band gaps are the resolvent set of the original problem. The absolute size of the band gap is not a meaningful measurement because there is no fundamental length scale in electromagnetics [4] and so we find the configuration of the rods that gives the largest gap-midgap ratio. This optimisation and the important role played by a computational grid is the aim of this paper. The University of Southampton has invested in grid-enabled computational facilities. We used a Beowulf cluster [5] running Linux and an experimental Condor system [6] that uses Windows. One important factor was that our chosen meshing program Geompack [7]is only available for Windows, thus requiring a heterogeneous mix of operating systems in our grid. H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 533–536, 2003. c Springer-Verlag Berlin Heidelberg 2003
534
D.H. Beckett et al. Generate rod coordinates and radii Check to see if any rods overlap or have been placed outside the unit cell Yes
No
Generate a mesh of the rod geometry Solve the generalised eigenvalue problem for a set of approximately 100k vectors, calculating the lowest 20 eigenvalues
Calculate and return the gap-midgap ratio
Fig. 1. Figure showing how the objective function for maximising the PC gap-midgap ratio is constructed.
2
Optimising Photonic Crystal Structures
A photonic crystal is a periodic configuration of air-filled rods in a substrate (for example Gallium Arsenide). The physical properties of a Photonic Crystal (PC) depend critically on the configuration of air rods in a substrate. A useful PC would be one such that the gap-midgap ratio [4] is very large. The objective in the optimisation is to maximize the gap-midgap ratio; the independent variables are the rod co-ordinates and the rod radii. The details of the computation are shown in Figure 1. Our approach is more direct than [8], [9], who only consider the TM mode and used a gradient method. To investigate various PC geometries we performed a design of experiment which consisted of constructing several thousand random configurations of a number of rods in a unit cell. Two different unit cells were used: a square unit cell and a rhombic unit cell. We only report on the latter in this paper. The rhombic unit cell was varied from 1 to 10 rods, although only the radius was altered when a single rod was used as the position for a single rod is meaningless when the unit cell is tiled. For the rhombic unit cell 100 meshes per rod radius and number of rods was generated for varying filling fractions of approximately 30% to 78%. The number of rods varied from 2 to 10. In total 1400 random meshes were created. The meshes were generated on a computer with a Windows operating system then transferred to the Beowulf cluster to solve the generalized eigenvalue problem(a compiled MATLAB program). The best 10 structures from each unit cell, which gave the largest gap-midgap ratio were optimised using the robust (but slow) Nelder Mead method [10] on the Condor system. The Nelder Mead algorithm was chosen for the optimisation because it only requires function evaluations and not derivatives which is ideal for our problem because the objective function has no calculable derivative. There
Applied Grid Computing: Optimisation of Photonic Devices
535
are a number of constraints imposed on the variables mainly to ensure that it is possible to fabricate the proposed structure. Each iteration is computationally intensive as it requires not only meshing the photonic crystal geometry but also solving the generalized eigenvalue problem.
Fig. 2. Figure on the top left shows a tiled rhombic unit cell with a filling fraction of 0.753. On the top right is the density of states diagram for the unit cell. The band gap to mid-gap ratio is 0.118. On the bottom left is the optimised tiled unit cell with a filling fraction of 0.838. On the bottom right is the density of states for the optimised unit cell with a band gap to mid-gap ratio of 0.182.
The results in this section were computed with an in house FEM code. The most improved mesh increased its gap-midgap ratio by 264.3% after optimisation. A density states graph is a concise method of visualising the results and is shown in Figure 2.
3
Computer Costs
In total 8940 meshes were created. Each generalized eigenvalue problem took approximately 6 minutes to run which equates to 37 days to compute the eigenvalues of each mesh if only one computer was used. When using the Beowulf cluster the workload was split up such that each processor would compute the eigenvalues for 100 meshes. However, the nodes used were dual processors so 200 meshes could be run on each node. As a result 45 separate jobs were submitted to our cluster using the EASY scheduler each taking approximately 10 hours to run. If the cluster was not being used at all all the jobs could have been run simultaneously in roughly 10 hours. However, the rate at which all the jobs are computed is dependent on how busy the cluster is. In our case, despite a heavily used cluster, it only took a few days to calculate the generalized eigenvalue problem from all the meshes.
536
D.H. Beckett et al.
From the design of experiment 20 of the best PC geometries were optimised. The optimisation process was run for 12 hours for each structure. This equates to 10 days of computation on a single computer. Each of the 20 optimisation jobs were submitted to the Condor pool. With 20 free nodes, all the optimisation could be performed in only 12 hours. Once again, due to how busy the cluster was, a couple of days were required to complete all the jobs.
4
Conclusion
The randomisation proved effective in providing candidates for further analysis. Moreover, it confirmed the (known) good behaviour of the triangular lattice [4]. The Nelder Mead algorithm was successful in improving the good candidates. The main contribution of the grid facilities was the reduction of about two months intense computation to a weekend. The scheduler on the Beowulf system was straightforward to use, but Condor had a shaper learning curve and less predictable completion times. We have demonstrated the use of heterogeneous grid-enabled resources for an important industrial design process and in the future we hope that a widespread grid will enable the use of resources via brokering or pay-per-use and permit this sort of design to be a routine part of the process of developing new photonic devices.[4]
References 1. Axmann, W., Kuchment, P.: An efficient finite element method for computing spectra of photonic and acoustic band-gap materials – i. scalar case. Journal of Computational Physics 150 (1999) 468–481 2. Dobson, D.C.: An efficient method for band structure calculations in 2d photonic crystals. Journal of Computational Physics 149 (1999) 363–376 3. K.S.Thomas, Cox, S., Beckett, D., Hiett, B., Generowicz, J., Daniell, G.: Eigenvalue spectrum estimation and photonic crystals. In: 7th International Europar Conference, Springer (2001) 578–586 4. Joannopoulos, J., Meade, R., Winn, J.: Photonic Crystals Molding the Flow of Light. Princeton University Press (1995) 5. University of Southampton: Research support services: Iridis (2002) http://www.iss.soton.ac.uk/research/iridis/. 6. University of Southampton: Research support services: Condor (2002) http://www.iss.soton.ac.uk/research/e-science/condor/. 7. Joe, B.: Geompack mesh generating software (2001) http://members.attcanada.ca/ bjoe/. 8. Cox, S.J., Dobson, D.C.: Maximizing band gaps in two-dimensional photonic crystals. Siam Journal on Applied Mathematics 59 (1999) 2108–2120 9. Cox, S.J., Dobson, D.C.: Band structure optimization of two-dimensional photonic crystals in h-polarization. Journal of Computational Physics 158 (2000) 214–224 10. Brent, R.: Algorithms for Minimization Without Derivitives. Englewood Cliffs, NJ:Prentice Hall (1973)
Parallel Linear System Solution and Its Application to Railway Power Network Simulation Muhammet F. Ercan1, Yu-fai Fung2, Tin-kin Ho2, and Wai-leung Cheung2 1
School of Electrical and Electronic Eng., Singapore Polytechnic, Singapore
[email protected] 2 Dept. of Electrical Eng., The Hong Kong Polytechnic, University, Hong Kong SAR {eeyffung,eetkho,eewlcheung}@polyu.edu.hk
Abstract. The Streaming SIMD extension (SSE) is a special feature embedded in the Intel Pentium III and IV classes of microprocessors. It enables the execution of SIMD type operations to exploit data parallelism. This article presents improving computation performance of a railway network simulator by means of SSE. Voltage and current at various points of the supply system to an electrified railway line are crucial for design, daily operation and planning. With computer simulation, their time-variations can be attained by solving a matrix equation, whose size mainly depends upon the number of trains present in the system. A large coefficient matrix, as a result of congested railway line, inevitably leads to heavier computational demand and hence jeopardizes the simulation speed. With the special architectural features of the latest processors on PC platforms, significant speed-up in computations can be achieved.
1 Introduction An electric railway is a dynamic system with a number of sub-systems, such as signaling, power supply and traction drive, which interact with each other continuously. Because of high cost of on-site evaluation, simulation is perceived to be the most feasible means for engineering analysis. Advances in hardware and software allow enhancement on simulators, in terms of computation speed and data management [3]. Numerous simulation packages have been developed and found successful applications. However, most of the simulators were tailor-made for specific studies. In view of this deficiency, a whole-system simulation suite, which includes the modeling of most possible functions and behavior of a railway system, is essential for accurate and reliable research and development on railway engineering. Such simulation suite should allow multi-train movement simulation under various signaling requirements and power system features on a railway network. We have designed a simulator with the above features to cater for different railway systems with user-defined track geometry, signaling system and traction equipment [4]. The computation time is crucial in a real time simulator; in this application, identifying solution of linear system equations repetitively was the most time consuming operation. Hence, we aim to speed up these operations by the SSE features provided in standard PCs. H. Kosch, L. Böszörményi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 537–540, 2003. © Springer-Verlag Berlin Heidelberg 2003
538
M.F. Ercan et al.
2 Problem Modeling The commonly adopted computing platform in the railway industry is the Intel microprocessor based PCs. Simulation imposes heavy computational demands when the railway network gets complicated and number of trains increases. Both off-line studies and real-time control prompt for more computation power. We explore the parallel computing features of Pentium CPUs to solve the linear equations. An electrified railway line is in fact a huge electrical circuit with the substations as sources and the trains as moving loads (sources if the re-generative braking is allowed) [7]. As illustrated in Fig. 1, trains traveling on two opposite directions draw energy from the feeding substations along the line and their locations define the nodes in the electrical circuit. The supply wires and rail resistance are lumped together to form a resistor between nodes. The voltage seen by a train may vary with time and then determine its traction performance which in turn affects train movement. Thus, it is crucial to attain the voltages at certain nodes of this electrical circuit at consecutive time intervals, which requires the nodal analysis and thus the solution of a matrix equation. The size of the coefficient matrix depends upon the number of nodes in the circuit at each time interval, which is determined by the number of trains on track.
Fig. 1. Typical two- road network.
With an efficient bandwidth oriented pre-ordering scheme [2], the coefficient matrix of the power network is sparse, narrowly-banded and symmetric for a simple two-track network. The solution is obtained by Cholesky decomposition, followed by forward and backward substitutions. Unsurprisingly, this stage is the most CPU timeconsuming part of the simulation. When the topology of the network becomes more complicated with branches and loops, the size and bandwidth of the coefficient matrix increases and sparsity deteriorates. Computational demand is hugely raised as a result. In order to extend the applicability of the simulator to real-time studies, it is necessary to solve the matrix equation rapidly within the limited computing resources.
3 Computational Demand and Parallel Computing Intel microprocessor based PCs are the most commonly used platforms for railway simulators. For a simple DC railway line with two separate tracks and 20 trains on each one, the coefficient matrix is only around 50x50. The simulation time is still much faster than real-time. Power network calculation has not caused too many problems in most off-line studies even with a single processor. However, dealing with
Parallel Linear System Solution and Its Application
539
a large, complicated and busy network (larger and less sparse coefficient matrix) and/or AC supply system (complex numbers involved) presents a formidable challenge to the simulator in the increasingly demanding real-time applications. Parallel computing is an obvious direction to deal with the computation demand. Multiple-processor hardware is a convenient way out, but it usually requires substantial modifications within the simulator and increases the system cost. Equal workload assignment to the processors and suitable processor architecture are other critical considerations. Our objective is to exploit the SIMD computing capabilities of the latest processors so that simulator can be sped up on a single processor system with minimum alterations on the source codes. The Intel Pentium III or above processors include SSE registers where parallel data operations are performed simultaneously [6]. The SSE architecture provides the flexibility of operating with various data types, such as registers holding eight 16-bit integers or four 32-bit floating-point values. SIMD operations, add, multiply, etc., can be performed between the two registers and significant speed-up can be achieved. In particular, its support for floating point manipulations plays a key role in this application. The matrix equation resulted from an electrical railway system is in the form of: Ax = b
(1)
A is a symmetrical sparse matrix representing the admittance linking the nodes, vector x represents the unknown voltage at the nodes along the track and vector b defines the net current from each node. In our earlier work, we have demonstrated SSE based parallel algorithms for solving the above system equations with dense matrix in general [1]. Similar to many engineering problems, the system matrix of this railway network problem is also sparse. When dealing with the sparse matrix problem with SSE, the non-zero elements can be handled effectively. SSE registers can process four values at each operation; if most of the elements processed in an operation are equal to zero then the algorithm will not be efficient.
4 Experimental Results The input to the simulator is a set of linear equations related to the electrical circuit in the railway system. Integration of the network solution algorithm to the simulator is usually realized in either modular or embedded format. The timing results, based on a 733MHz Pentium III machine, obtained from the simulator were included in Table 1. Different sizes of the matrix represent different number of trains running on the network and the matrix sizes of 102x102, 132x132, 162x162 and 192x192 correspond to 5, 10, 15, and 20 trains. From Table 1, the speedup ratio obtained is close to 2 in most cases and when the number of trains is 20. With SSE, the computing time is reduced by 8 minutes. We have, therefore, demonstrated the benefit of SSE in solving one type of engineering problem, and our current research direction is to study how SSE can be embedded in other forms of parallel mechanism.
540
M.F. Ercan et al. Table 1. Timing results (in seconds) obtained from railway network simulator.
Matrix size 102x102 132x132 162x162 192x192
Non-SSE 210 436 764 1312
SSE 135 167 402 645
Speed-up 1.56 2.61 1.90 2.03
5 Conclusions In this paper, we have utilized the latest SIMD extensions included in Pentium processors to speedup LU decomposition involved in a railway simulator. According to our results, a speedup ratio of more than two is obtained. The results are satisfactory; and more importantly such improvement is achieved simply by minor modifications to the existing software. Furthermore, additional hardware support is not required for such performance enhancement. On the other hand, drawback of SSE is the overhead, which is caused by the operations to pack data into, and then unpack data from, the 128-bit format. The number of packing and unpacking operations is 2
proportional to N , where N is the size of the matrix. The gain in performance is reduced when matrix size is larger than 400x400. Our study demonstrates that the SSE features are valuable tools for developing PC based application software. Acknowledgement. This work is supported by the Hong Kong Polytechnic University under the grant number A-PD59.
References 1. Fung Y. F., Ercan M. F., Ho T.K., and Cheung W. L., A Parallel Solution to Linear Systems, Microprocessors and Microsystems 26 (2001) 39–44. 2. Goodman, C.J., Mellitt, B. and Rambukwella, N.B., CAE for the Electrical Design of Urban Rail Transit Systems, COMPRAIL’87 (1987) 173–193. 3. Goodman, C.J., Siu, L.K. and Ho, T.K., A Review of Simulation Models for Railway Systems, Int. Conf. on Developments in Mass Transit Systems (1998) 80–85. 4. Ho, T.K., Mao, B.H., Yuan, Z.Z., Liu, H.D. and Fung, Y.F., Computer Simulation and Modeling in Railway Applications, Comp. Physics Communications 143 (2002) 1–10. 5. http://www.pserc.cornell.edu/matpower/matpower.html, 2003. 6. Intel C/C++ Compiler Class Libraries for SIMD Operations User’s Guide, Intel, 2000. 7. Mellitt, B., Goodman, C.J. and Arthurton, R.I.M., Simulator for Studying Operational and Power-Supply Conditions in Rapid-Transit Railways, Proc. IEE 125 (1978) 298–303.
Topic 8 Parallel Computer Architecture and Instruction-Level Parallelism Stamatis Vassiliadis, Nikitas Dimopoulos, Jean-Francois Collard, and Arndt Bode Topic Chairs
Parallel computer architecture and instruction-level parallelism are hot topics at Euro-Par conferences, since these techniques are present in most contemporary computing systems. At Euro-Par 2003, 18 papers were submitted to the topic, from which 1 distinguished, 4 regular and 4 short papers were accepted. The scope of this topic includes but is not limited to parallel computer architectures, processor architecture (architecture and micro architecture as well as compilation), the impact of emerging microprocessor architectures on parallel computer architectures, innovative memory designs to hide and reduce the excess latency, multi-threading, and impact of emerging applications on parallel computer architecture design. The distinguished paper by George Almasi et al. gives an overview of the Blue Gene/L System Software Organization. With its 360 Teraflops of peak computing power and 65536 compute nodes it is a special purpose architecture well beyond the first entries of the TOP 500 list. The authors explain, how the system software will deal with this extreme amount of parallelism. The paper is therefore general for the future developments of parallel computer architectures. Trace substitution is the topic of Hans Vandierendonck, Hans Logie and Koen De Bosschere from Gent University, Belgium. This technique is useful for wide-issue superscalar processors and their ability to react on branches in the instruction stream. The authors show that their new technique improves the fetch bandwidth of such processors. Counteracting bank misprediction in sliced first-level caches again deals with possible cache misses and is presented by E.F. Torres, P. Ibanez, V. Vinals, J.M. Llaberia. The authors discuss techniques to reduce the bank misprediction penalty in future processors having sliced memory pipelines. Juan Moure, Dolores Rexachs, Emilio Luque from the Computer Architecture and Operating Systems Group of the Universidad Aut´ onoma de Barcelona present a mechanism to optimize a decoupled front-end architecture: the indexed fetch target buffer (iFTB). Once again, wide issue superscalar processors are addressed and an indexed variation of the fetch target buffer, a decoupled front-end architecture is presented to increase the fetch rate of such processors. Seong-Won Lee from the University of Southern California at Los Angeles and Jean-Luc Gaudiot from the University of California at Irvine present the technique of clustered microarchitecuture simultaneous multithreading. SMT is H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 541–542, 2003. c Springer-Verlag Berlin Heidelberg 2003
542
S. Vassiliadis et al.
favourable for future generation microprocessors, since it exploits instructionlevel parallelism and thread-level parallelism. The clustered SMT architecture proposed by the authors significantly reduces power consumption without performance degradation. Spiros Kalogeropulos from Sun proposes a global instruction scheduling technique addressing processors with moderate support for parallelism. The enhanced trace scheduler includes techniques for trace formation, a renaming scheme and a cost benefit analysis. Masamichi Takagi and Kei Hiraki from the Department of Computer Science, University of Tokyo, propose compression in data caches with compressible field isolation for recursive data structures. The field array compression technique tends to alleviate the penalty for cache misses in program execution. Hideyuki Miura et al. from the University of Tokyo discuss a compiler-assisted thread level control speculation for speculative multithreading execution. Two new techniques are presented. The last paper of this topic again deals with caches in microprocessors. In this paper, Carles Aliagas et al. address the reduction of power to run the cache and show with their paper on “value compression to reduce power in data caches” the way to power reduction and reduction in die area without a performance penalty.
An Overview of the Blue Gene/L System Software Organization George Alm´asi1 , Ralph Bellofatto1 , Jos´e Brunheroto1 , C˘alin Ca¸scaval1 , Jos´e G. Casta˜nos1 , Luis Ceze2 , Paul Crumley1 , C. Christopher Erway1 , Joseph Gagliano1 , Derek Lieber1 , Xavier Martorell1 , Jos´e E. Moreira1 , Alda Sanomiya1 , and Karin Strauss2 1
IBM Thomas J. Watson Research Center Yorktown Heights, NY 10598-0218 {gheorghe,ralphbel,brunhe,cascaval,castanos,pgc,erway, jgaglia,lieber,xavim,jmoreira,sanomiya}@us.ibm.com 2 Department of Computer Science University of Illinois at Urbana-Champaign Urabana, IL 61801 {luisceze,kstrauss}@uiuc.edu
Abstract. The Blue Gene/L supercomputer will use system-on-a-chip integration and a highly scalable cellular architecture. With 65,536 compute nodes, Blue Gene/L represents a new level of complexity for parallel system software, with specific challenges in the areas of scalability, maintenance and usability. In this paper we present our vision of a software architecture that faces up to these challenges, and the simulation framework that we have used for our experiments.
1
Introduction
In November 2001 IBM announced a partnership with Lawrence Livermore National Laboratory to build the Blue Gene/L (BG/L) supercomputer, a 65,536-node machine designed around embedded PowerPC processors. Through the use of system-on-a-chip integration [10], coupled with a highly scalable cellular architecture, Blue Gene/L will deliver 180 or 360 Teraflops of peak computing power, depending on the utilization mode. Blue Gene/L represents a new level of scalability for parallel systems. Whereas existing large scale systems range in size from hundreds (ASCI White [2], Earth Simulator [4]) to a few thousands (Cplant [3], ASCI Red [1]) of compute nodes, Blue Gene/L makes a jump of almost two orders of magnitude. The system software for Blue Gene/L is a combination of standard and custom solutions. The software architecture for the machine is divided into three functional entities (similar to [13]) arranged hierarchically: a computational core, a control infrastructure and a service infrastructure. The I/O nodes (part of the control infrastructure) execute a version of the Linux kernel and are the primary off-load engine for most system services. No user code directly executes on the I/O nodes. Compute nodes in the computational core execute a single user, single process minimalist custom kernel, and are dedicated to efficiently run user applications. No system daemons or sophisticated system services reside on compute nodes. These are treated as externally controllable entities (i.e. H. Kosch, L. B¨osz¨orm´enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 543–555, 2003. c Springer-Verlag Berlin Heidelberg 2003
544
G. Almasi et al.
devices) attached to I/O nodes. Complementing the Blue Gene/L machine proper, the Blue Gene/L complex includes the service infrastructure composed of commercially available systems, that connect to the rest of the system through an Ethernet network. The end user view of a system is of a flat, toroidal, 64K-node system, but the system view of Blue Gene/L is hierarchical: the machine looks like a 1024-node Linux cluster, with each node being a 64-way multiprocessor. We call one of these logical groupings a processing set or pset. The scope of this paper is to present the software architecture of the Blue Gene/L machine and its implementation. Since the target time frame for completion and delivery of Blue Gene/L is 2005, all our software development and experiments have been conducted on architecturally accurate simulators of the machine. We describe this simulation environment and comment on our experience. The rest of this paper is organized as follows. Section 2 presents a brief description of the Blue Gene/L supercomputer. Section 3 discusses the system software. Section 4 introduces our simulation environment and we conclude in Section 5.
2 An Overview of the Blue Gene/L Supercomputer Blue Gene/L is a new architecture for high performance parallel computers based on low cost embedded PowerPC technology. A detailed description of Blue Gene/L is provided in [8]. In this section we present a short overview of the hardware as background for our discussion on its system software and its simulation environment. 2.1
Overall Organization
The basic building block of Blue Gene/L is a custom system-on-a-chip that integrates processors, memory and communications logic in the same piece of silicon. The BG/L chip contains two standard 32-bit embedded PowerPC 440 cores, each with private L1 32KB instruction and 32KB data caches. Each core also has a 2KB L2 cache and they share a 4MB L3 EDRAM cache. While the L1 caches are not coherent, the L2 caches are coherent and act as a prefetch buffer for the L3 cache. Each core drives a custom 128-bit double FPU that can perform four double precision floating-point operations per cycle. This custom FPU consists of two conventional FPUs joined together, each having a 64-bit register file with 32 registers. One of the conventional FPUs (the primary side) is compatible with the standard PowerPC floating-point instruction set. We have extended the PPC instruction set to perform SIMD-style floating point operations on the two FPUs. In most scenarios, only one of the 440 cores is dedicated to run user applications while the second processor drives the networks. At a target speed of 700 MHz the peak performance of a node is 2.8 GFlop/s. When both cores and FPUs in a chip are used, the peak performance per node is 5.6 GFlop/s. The standard PowerPC 440 cores are not designed to support multiprocessor architectures: the L1 caches are not coherent and the architecture lacks atomic memory operations. To overcome these limitations BG/L provides a variety of synchronization devices in the chip: lockbox, shared SRAM, L3 scratchpad and the blind device. The lockbox unit contains a limited number of memory locations for fast atomic test-and-sets
An Overview of the Blue Gene/L System Software Organization
545
and barriers. 16 KB of SRAM in the chip can be used to exchange data between the cores and regions of the EDRAM L3 cache can be reserved as an addressable scratchpad. The blind device permits explicit cache management. The low power characteristics of Blue Gene/L permit a very dense packaging as shown in Figure 1. Two nodes share a node card that also contains SDRAM-DDR memory. Each node supports a maximum of 2 GB external memory but in the current configuration each node directly addresses 256 MB at 5.5 GB/s bandwidth with a 75 cycle latency. Sixteen compute cards can be plugged in a node board. A cabinet with two midplanes contains 32 node boards for a total of 2048 CPUs and a peak performance of 2.9/5.7 TFlops. The complete system has 64 cabinets and 16 TB of memory. In addition to the 64K compute nodes, BG/L contains a number of I/O nodes (1024 in the current design). Compute nodes and I/O nodes are physically identical although I/O nodes are likely to contain more memory. The only difference is in their card packaging which determines which networks are enabled.
Fig. 1. High-level organization of the Blue Gene/L supercomputer. All 65,536 compute nodes are organized in a 64 × 32 × 32 three-dimensional torus.
2.2
Blue Gene/L Communications Hardware
All inter-node communication is done exclusively through messages. The BG/L ASIC supports five different networks: torus, tree, Ethernet, JTAG, and global interrupts. The main communication network for point-to-point messages is a three-dimensional torus.
546
G. Almasi et al.
Each node contains six bi-directional links for direct connection with nearest neighbors. The 64K nodes are organized into a partitionable 64x32x32 three-dimensional torus. The network hardware in the ASICs guarantees reliable, unordered, deadlock-free delivery of variable length (up to 256 bytes) packets, using a minimal adaptive routing algorithm. It also provides simple broadcast functionality by depositing packets along a route. At 1.4 Gb/s per direction, the unidirectional bisection bandwidth of a 64K node system is 360 GB/s. The I/O nodes are not part of the torus network. The tree network supports fast configurable point-to-point messages of fixed length (256 bytes) data. It also implements broadcasts and reductions with a hardware latency of 1.5 microseconds for a 64K node system. An ALU in the network can combine incoming packets using bitwise and integer operations, and forward a resulting packet along the tree. Floating point reductions can be performed in two phases (one for the exponent and another one for the mantissa) or in one phase by converting the floating-point number to an extended 2048-bit representation. I/O and compute nodes share the tree network. Tree packets are the main mechanism for communication between I/O and compute nodes. The torus and the tree networks are memory mapped devices. Processors send and receive packets by explicitly writing to (and reading from) special memory addresses that act as FIFOs. These reads and writes use the 128-bit SIMD registers. A separate set of links provides global OR/AND operations (also with a 1.5 microseconds latency) for fast barrier synchronization. The Blue Gene/L computational core can be subdivided into partitions, which are electrically isolated and self-contained subsets of the machine. A partition is dedicated to the execution of a single job. The tree and torus wires between midplanes (half a cabinet) are routed through a custom chip called the Link Chip. This chip can be dynamically configured to skip faulty midplanes while maintaining a working torus and to partition the torus network into multiple, independent torii. The smallest torus in BG/L is a midplane (512 nodes). The tree and torus FIFOs are controlled by device control registers (DCRs). It is possible to create smaller meshes by disabling the FIFOs of the chips at the boundary of a partition. In the current packaging schema, the smallest independent mesh contains 128 compute nodes and 2 I/O nodes. Finally, each BG/L chip contains a 1Gbit/s Ethernet macro for external connectivity and supports a serial JTAG network for booting, control and monitoring of the system. Only I/O nodes are attached to the Gbit/s Ethernet network, giving 1024x1Gbit/s links to external file servers. Completing the Blue Gene/L machine, there is a number of service nodes: a traditional Linux cluster or SP system that resides outside the Blue Gene/L core. The service nodes communicate with the computational core through the IDo chips. The current packaging contains one IDo chip per node board and another one per midplane. The IDo chips are 25MHz FPGAs that receive commands from the service nodes using raw UDP packets over a trusted private 100 Mbit/s Ethernet control network. The IDo chips support a variety of serial protocols for communication with the core. The I 2 C network controls temperature sensors, fans and power supplies. The JTAG protocol is used for reading and writing to any address of the 16 KB SRAMs in the BG/L chips, reading and writing to registers and sending interrupts to suspend and reset the cores. These services
An Overview of the Blue Gene/L System Software Organization File servers
Pset 0
tree
Console Front end
Linux
547
BLRTS
BLRTS torus
MMCS lib ciod
user apps
user apps
I/O Node 0
Compute Node 0
Compute Node 63
Ethernet
MMCS DB
JTAG
torus
torus
IDo lib
MMCS lib
Service Node
Linux
BLRTS
BLRTS torus
Scheduler ciod
Control Ethernet
I/O Node 1023 tree IDo
user apps
user apps
Compute Node 0
Compute Node 63 Pset 1023
JTAG
Fig. 2. Outline of the Blue Gene/L system software. The computational core is partitioned into 1024 logical processing sets (psets), each with one I/O node running Linux and 64 compute nodes running the custom BLRTS kernel. External entities connect the computational core through two Ethernet networks for I/O and low level management.
are available through a direct link between the IDo chips and nodes, and bypass the system software running on the target nodes.
3
Blue Gene/L System Software
To address the challenges of scalability and complexity posed by BG/L we have developed the system software architecture presented in Figure 2. This architecture is described in detail in this section. 3.1
System Software for the I/O Nodes
The Linux kernel that executes in the I/O nodes (currently version 2.4.19) is based on on a standard distribution for PowerPC 440GP processors. Although Blue Gene/L uses standard PPC 440 cores, the overall chip and card design required changes in the booting sequence, interrupt management, memory layout, FPU support, and device drivers of the standard Linux kernel. There is no BIOS in the Blue Gene/L nodes, thus the configuration of a node after power-on and the initial program load (IPL) is initiated by the service nodes through the control network. We modified the interrupt and exception handling code to support Blue Gene/L’s custom Interrupt Controller (BIC). The implementation of the kernel MMU remaps the tree and torus FIFOs to user space. We support the new EMAC4 Gigabit Ethernet controller We also updated the kernel to save and restore the double FPU registers in each context switch.
548
G. Almasi et al.
The nodes in the Blue Gene/L machine are diskless, thus the initial root file system is provided by a ramdisk linked against the Linux kernel. The ramdisk contains shells, simple utilities, shared libraries, and network clients such as ftp and nfs. Because of the non-coherent L1 caches, the current version of Linux runs on one of the 440 cores, while the second CPU is captured at boot time in an infinite loop. We are investigating two main strategies to effectively use the second CPU in the I/O nodes: SMP mode and virtual mode. We have successfully compiled a SMP version of the kernel, after implementing all the required interprocessor communications mechanisms, because the BG/L’s BIC is not OpenPIC [6] compliant. In this mode, the TLB entries for the L1 cache are disabled in kernel mode and processes have affinity to one CPU. Forking a process in a different CPU requires additional parameters to the system call. The performance and effectiveness of this solution is still an open issue. A second, more promising mode of operation runs Linux in one of the CPUs, while the second CPU is the core of a virtual network card. In this scenario, the tree and torus FIFOs are not visible to the Linux kernel. Transfers between the two CPUs appear as virtual DMA transfers. We are also investigating support for large pages. The standard PPC 440 embedded processors handle all TLB misses in software. Although the average number of instructions required to handle these misses has significantly decreased, it has been shown that larger pages improve performance [23]. 3.2
System Software for the Compute Nodes
The “Blue Gene/L Run Time Supervisor” (BLRTS) is a custom kernel that runs on the compute nodes of a Blue Gene/L machine. BLRTS provides a simple, flat, fixed-size 256MB address space, with no paging, accomplishing a role similar to PUMA [19]. The kernel and application program share the same address space, with the kernel residing in protected memory at address 0 and the application program image loaded above, followed by its heap and stack. The kernel protects itself by appropriately programming the PowerPC MMU. Physical resources (torus, tree, mutexes, barriers, scratchpad) are partitioned between application and kernel. In the current implementation, the entire torus network is mapped into user space to obtain better communication efficiency, while one of the two tree channels is made available to the kernel and user applications. BLRTS presents a familiar POSIX interface: we have ported the GNU Glibc runtime library and provided support for basic file I/O operations through system calls. Multiprocessing services (such as fork and exec) are meaningless in single process kernel and have not been implemented. Program launch, termination, and file I/O is accomplished via messages passed between the compute node and its I/O node over the tree network, using a point-to-point packet addressing mode. This functionality is provided by a daemon called CIOD (Console I/O Daemon) running in the I/O nodes. CIOD provides job control and I/O management on behalf of all the compute nodes in the processing set. Under normal operation, all messaging between CIOD and BLRTS is synchronous: all file I/O operations are blocking on the application side. We used the CIOD in two scenarios:
An Overview of the Blue Gene/L System Software Organization
549
1. driven by a console shell (called CIOMAN), used mostly for simulation and testing purposes. The user is provided with a restricted set of commands: run, kill, ps, set and unset environment variables. The shell distributes the commands to all the CIODs running in the simulation, which in turn take the appropriate actions for their compute nodes. 2. driven by a job scheduler (such as LoadLeveler) through a special interface that implements the same protocol as the one defined for CIOMAN and CIOD. We are investigating a range of compute modes for our custom kernel. In heater mode, one CPU executes both user and network code, while the other CPU remains idle. This mode will be the mode of operation of the initial prototypes, but it is unlikely to be used afterwards. In co-processor mode, the application runs in a single, nonpreemptable thread of execution on the main processor (cpu 0). The coprocessor (cpu 1) is used as a torus device off-load engine that runs as part of a user-level application library, communicating with the main processor through a non-cached region of shared memory. In symmetric mode, both CPUs run applications and users are responsible for explicitly handling cache coherence. In virtual node mode we provide support for two independent processes in a node. The system then looks like a machine with 128K nodes. 3.3
System Management Software
The control infrastructure is a critical component of our design. It provides a separation between execution mechanisms in the BG/L core and policy decisions in external nodes. Local node operating systems (Linux for I/O nodes and BLRTS for compute nodes) implement services and are responsible for local decisions that do not affect overall operation of the machine. A “global operating system” makes all global and collective decisions and interfaces with external policy modules (e.g., LoadLeveler) and performs a variety of system management services, including: (i) machine booting, (ii) system monitoring, and (iii) job launching. In our implementation, the global OS runs on external service nodes. Each BG/L midplane is controlled by one Midplane Management and Control System (MMCS) process which provides two paths into the Blue Gene/L complex: a custom control library to access the restricted JTAG network and directly manipulate Blue Gene/L nodes; and sockets over the Gbit/s Ethernet network to manage the nodes on a booted partition. The custom control library can perform: – low level hardware operations such as: turn on power supplies, monitor temperature sensors and fans, and react accordingly (i.e. shut down a machine if temperature exceeds some threshold), – configure and initialize IDo, Link and BG/L chips, – read and write configuration registers, SRAM and reset the cores of a BG/L chip. As mentioned in Section 2, these operations can be performed with no code executing in the nodes, which permits machine initialization and boot, nonintrusive access to performance counters and post-mortem debugging. This path into the core is used for control only; for security and reliability reasons, it is not made visible to applications running in the BG/L nodes. On the other hand,
550
G. Almasi et al.
the architected path through the functional Gbit/s Ethernet is used for application I/O, checkpoints, and job launch. We chose to maintain the entire state of the global operating system using standard database technology. Databases naturally provide scalability, reliability, security, portability, logging, and robustness. The database contains static state (i.e., the physical connections between each HW component) and dynamic state (i.e., how the machine is partitioned, how each partition is configured, and which parallel application is running in each partition). Therefore, the database is not just a repository for read-only configuration information, but also an interface for all the visible state of a machine. External entities (such as a scheduler) can manipulate this state by invoking stored procedures and database triggers, which in turn invoke functions in the MMCS processes. Machine Initialization and Booting. The boot process for a node consists of the following steps: first, a small boot loader is directly written into the (compute or I/O) node memory by the service nodes using the JTAG control network. This boot loader loads a much larger boot image into the memory of the node through a custom JTAG mailbox protocol. We use one boot image for all the compute nodes and another boot image for all the I/O nodes. The boot image for the compute nodes contains the code for the compute node kernel, and is approximately 64 kB in size. The boot image for the I/O nodes contains the code for the Linux operating system (approximately 2 MB in size) and the image of a ramdisk that contains the root file system for the I/O node. After an I/O node boots, it can mount additional file systems from external file servers. Since the same boot image is used for each node, additional node specific configuration information (such as torus coordinates, tree addresses, MAC or IP addresses) must be loaded. We call this information the personality of a node. In the I/O nodes, the personality is exposed to user processes through an entry in the proc file system. BLRTS implements a system call to request the node’s personality. System Monitoring in Blue Gene/L is accomplished through a combination of I/O node and service node functionality. Each I/O node is a full Linux machine and uses Linux services to generate system logs. A complementary monitoring service for Blue Gene/L is implemented by the service node through the control network. Device information, such as fan speeds and power supply voltages, can be obtained directly by the service node through the control network. The compute and I/O nodes use a communication protocol to report events that can be logged or acted upon by the service node. This approach establishes a completely separate monitoring service that is independent of any other infrastructure (tree and torus networks, I/O nodes, Ethernet network) in the system. Therefore, it can be used even in the case of many system-wide failures to retrieve important information. Job Execution is also accomplished through a combination of I/O nodes and service node functionality. When submitting a job for execution in Blue Gene/L , the user specifies the desired shape and size of the partition to execute that job. The scheduler
An Overview of the Blue Gene/L System Software Organization
551
selects a set of compute nodes to form the partition. The compute (and corresponding I/O) nodes selected by the scheduler are configured into a partition by the service node using the control network. We have developed techniques for efficient allocation of nodes in a toroidal machine that are applicable to Blue Gene/L [16]. Once a partition is created, a job can be launched through the I/O nodes in that partition using CIOD as explained before. 3.4
Compiler and Run-Time Support
Blue Gene/L presents a familiar programming model and a standard set of tools. We have ported the GNU toolchain (binutils, gcc, glibc and gdb) to Blue Gene/L and set it up as a cross-compilation environment. There are two cross-targets: Linux for I/O nodes and BLRTS for compute nodes. IBM’s XL compiler suite is also being ported to provide advanced optimization support for languages like Fortran90 and C++. 3.5
Communication Infrastructure
The Blue Gene/L communication software architecture is organized into three layers: the packet layer is a thin software library that allows access to network hardware; the message layer provides a low-latency, high bandwidth point-to-point message delivery system; MPI is the user level communication library. The packet layer simplifies access to the Blue Gene/L network hardware. The packet layer abstracts FIFOs and devices control registers into torus and tree devices and presents an API consisting of essentially three functions: initialization, packet send and packet receive. The packet layer provides a mechanism to use the network hardware but doesn’t impose any policies on its use. Hardware restrictions, such as the 256 byte limit on packet size and the 16 byte alignment requirements on packet buffers, are not abstracted by the packet layer and thus are reflected by the API. All packet layer send and receive operations are non-blocking, leaving it up to the higher layers to implement synchronous, blocking and/or interrupt driven communication models. In its current implementation the packet layer is stateless. The message layer is a simple active message system [12,17,21,22], built on top of the torus packet layer, which allows the transmission of arbitrary messages among torus nodes. It is designed to enable the implementation of MPI point-to-point send/receive operations. It has the following characteristics: No packet retransmission protocol. The Blue Gene/L network hardware is completely reliable, and thus a packet retransmission system (such as a sliding window protocol) is unnecessary. This allows for stateless virtual connections between pairs of nodes, greatly enhancing scalability. Packetizing and alignment. The packet layer requires data to be sent in 256 byte chunks aligned at 16 byte boundaries. Thus the message layer deals with the packetizing and re-alignment of message data. Re-alignment of packet data typically entails memoryto-memory copies.
552
G. Almasi et al.
Packet ordering. Packets on the torus network can arrive out of order, which makes message re-assembly at the receiver non-trivial. For packets belonging to the same message, the message layer is able to handle their arrival in any order. To restore order for packets belonging to different messages, the sender assigns ascending numbers to individual messages sent out to the same peer. Cache coherence and processor use policy. The expected performance of the message layer is influenced by the way in which the two processors are used (as discussed in Section 3.2). Co-processor mode is the only one that effectively overlaps computation and communication. This mode is expected to yield better bandwidth, but slightly higher latency, than the others. MPI: Blue Gene/L is designed primarily to run MPI [20] workloads. We are in the process of porting MPICH2 [5], currently under development at Argonne National Laboratories, to the Blue Gene/L hardware. MPICH2 has a modular architecture. The Blue Gene/L port leaves the code structure of MPICH2 intact, but adds a number of plug-in modules: Point-to-point messages. The most important addition of the Blue Gene/L port is an implementation of ADI3, the MPICH2 Abstract Device Interface [14]. A thin layer of code transforms e.g. MPI Request objects and MPI Send function calls into calls into sequences of message layer function calls and callbacks. Process management. The MPICH2 process management primitives are documented in [7]. Process management is split into two parts: a process management interface (PMI), called from within the MPI library, and a set of process managers (PM) which are responsible for starting up/shutting down MPI jobs and implementing the PMI functions. MPICH2 includes a number of process managers (PM) suited for clusters of general purpose workstations. The Blue Gene/L process manager makes full use of its hierarchical system management software, including the CIOD processes running on the I/O nodes, to start up and shut down MPI jobs. The Blue Gene/L system management software is explicitly designed to deal with the scalability problem inherent in starting, synchronizing and killing 65,536 MPI processes. Optimized collectives. The default implementation of MPI collective operations in MPICH2 generates sequences of point-to-point messages. This implementation is oblivious of the underlying physical topology of the torus and tree networks. In Blue Gene/L optimized collective operations can be implemented for communicators whose physical layouts conform to certain properties. – The torus hardware can be used to efficiently implement broadcasts on contiguous 1, 2 and 3 dimensional meshes, using a feature of the torus that allows depositing a packet on every node it traverses. The collectives best suited for this (e.g. Bcast, Allgather, Alltoall, Barrier) involve broadcast in some form. – The tree hardware can be used for almost every collective that is executed on the MPI COMM WORLD communicator, including reduction operations. Integer operand reductions are directly supported by hardware. IEEE compliant floating point reductions can also be implemented by the tree using separate reduction phases for the mantissa and the exponent.
An Overview of the Blue Gene/L System Software Organization
553
– Non MPI COMM WORLD collectives can also be implemented using the tree, but care must be taken to ensure deadlock free operation. The tree is a locally class routed network, with packets belonging to one of a small number of classes and tree nodes making local decisions about routing. The tree network guarantees deadlock-free simultaneous delivery of no more than two class routes. One of these routes is used for control and file I/O purposes; the other is available for use by collectives.
File servers Pset Console Front end MMCS lib
Linux
BLRTS
BLRTS
ciod
user apps
user apps
bglsim
bglsim
bglsim
Ethernet
MMCS DB Tapdaemon
IDo lib
MMCS lib
Ethernet gateway
Service Node
CommFabric library IDo simulator
Scheduler Control Ethernet
Linux
BLRTS
BLRTS
ciod
user apps
user apps
bglsim
bglsim
bglsim Pset
Fig. 3. Overview of the Blue Gene/L simulation environment. Complete Blue Gene/L chips are simulated by a custom architectural simulator (bglsim ). A communication library (CommFabric) simulates the Blue Gene/L networks.
4
Blue Gene/L Simulation Environment
The first hardware prototypes of the Blue Gene/L ASIC are targeted to become operational in mid-2003. To support the development of system software before hardware is available, we have implemented an architecturally accurate, full system simulator for the Blue Gene/L machine [11]. The node simulator, called bglsim , is built using techniques described in [18,15]. Each component of the BG/L ASIC is modeled separately with the desired degree of accuracy, trading accuracy for performance. In our simulation environment, we model the functionality of processor instructions. That is, each instruction correctly changes the visible architectural state, while it takes one cycle to execute. We also model memory system behavior (cache, scratch-pad, and main memory) and all the Blue Gene/L specific devices: tree, torus, JTAG, device control registers, etc. A bglsim process boots the Linux kernel for the I/O nodes and BLRTS for the compute nodes. Applications run on top of these kernels, under user control.
554
G. Almasi et al.
When running on 1.2 GHz Pentium III machine, bglsim simulates an entire BG/L chip at approximately 2 million simulated instructions per second – a slow-down of about 1000 compared to the real hardware. By comparison, a VHDL simulator with hardware acceleration has a slow-down of 106 , while a software VHDL simulator has a slow-down of 109 . As an example, booting Linux takes 30 seconds on bglsim , 7 hours on the hardware accelerated VHDL simulator and more that 20 days on the software VHDL simulator. Large Blue Gene/L system are simulated using one bglsim process per node, as shown in Figure 3. The bglsim processes run on different workstations and communicate through a custom message passing library (CommFabric), which simulates the connectivity within the system and outside. Additionally, different components of the system are simulated by separate processes that also link in CommFabric. Examples are: the IDo chip simulator, a functional simulator of an IDo chip that translates packets between the virtual JTAG network and Ethernet; the Tapdaemon and EthernetGateway processes to provide the Linux kernels in the simulated I/O nodes with connectivity to the outside network, allowing users to mount external file-systems and connect using telnet, ftp, etc. We use this environment to develop our communication infrastructure, the control infrastructure and we have successfully executed the MPI NAS Parallel Benchmarks [9].
5
Conclusions
Blue Gene/L is the first of a new series of high performance machines being developed at IBM Research. The hardware plans for the machine are complete and the first small prototypes will be available in late 2003. In this paper, we have presented a software system that can scale up to the demands of the Blue Gene/L hardware. We have also described the simulation environment that we are using to develop and validate this software system. Using the simulation environment, we are able to demonstrate a complete and functional system software environment before hardware becomes available. Nevertheless, evaluating scalability and performance of the complete system still requires hardware availability. Many of the implementation details will likely change as we gain experience with the real hardware.
References 1. 2. 3. 4. 5. 6. 7. 8.
ASCI Red Homepage. http://www.sandia.gov/ASCI/Red/. ASCI White Homepage. http://www.llnl.gov/asci/platforms/white. Cplant homepage. http://www.cs.sandia.gov/cplant/. Earth Simulator Homepage. http://www.es.jamstec.go.jp/. The MPICH and MPICH2 homepage. http://www-unix.mcs.anl.gov/mpi/mpich. Open Firmware Homepage. http://www.openfirmware.org. Process Management in MPICH2. Personal communication from William Gropp. N. R. Adiga et al. An overview of the BlueGene/L supercomputer. In SC2002 – High Performance Networking and Computing, Baltimore, MD, November 2002.
An Overview of the Blue Gene/L System Software Organization
555
9. G. Almasi, C. Archer, J. G. Castanos, M. G. X. Martorell, J. E. Moreira, W. Gropp, S. Rus, and B. Toonen. MPI on BlueGene/L: Designing an Efficient General Purpose Messaging Solution for a Large Cellular System. Submitted for publication to the 2003 Euro PVM/MPI workshop. 10. G. Almasi et al. Cellular supercomputing with system-on-a-chip. In IEEE International Solid-state Circuits Conference ISSCC, 2001. 11. L. Ceze, K. Strauss, G. Alm´asi, P. J. Bohrer, J. R. Brunheroto, C. Ca¸scaval, J. G. C. nos, D. Lieber, X. Martorell, J. E. Moreira, A. Sanomiya, and E. Schenfeld. Full circle: Simulating linux clusters on linux clusters. In Proceedings of the Fourth LCI International Conference on Linux Clusters: The HPC Revolution 2003, San Jose, CA, June 2003. 12. G. Chiola and G. Ciaccio. Gamma: a low cost network of workstations based on active messages. In Proc. Euromicro PDP’97, London, UK, January 1997, IEEE Computer Society., 1997. 13. D. Greenberg, R. Brightwell, L. Fisk, A. Maccabe, and R. Riesen. A system software architecture for high-end computing. In Proceedings of Supercomputing 97, San Jose, CA, 1997. 14. W. Gropp and E. Lusk. MPICHAbstract Device InterfaceVersion 3.3 Reference Manual: Draft of October 17, 2002. http://www-unix.mcs.anl.gov/mpi/mpich/adi3/adi3man.pdf. 15. S.A. Herrod. Using Complete Machine Simulation to Understand Computer System Behavior. PhD thesis, Stanford University, February 1988. 16. E. Krevat, J. Castanos, and J. Moreira. Job scheduling for the Blue Gene/L system. In Job Scheduling Strategies for Parallel Processing, volume 2537 of Lecture Notes in Computer Science, pages 38–54. Springer, 2002. 17. S. Pakin, M. Lauria, and A. Chien. High performance messaging on workstations: Illinois Fast Messages (FM) for Myrinet. In Supercomputing ’95, San Diego, CA, December 1999, 1995. 18. M. Rosenblum, S. A. Herrod, E. Witchel, and A. Gupta. Complete computer simulation: The SimOS approach. IEEE Parallel and Distributed Technology,, 1995. 19. L. Shuler, R. Riesen, C. Jong, D. van Dresser, A. B. Maccabe, L. A. Fisk, and T. M. Stallcup. The Puma operating system for massively parallel computers. In In Proceedings of the Intel Supercomputer Users’ Group. 1995 Annual North America Users’ Conference, June 1995. 20. M. Snir, S. Otto, S. Huss-Lederman, D. Walker, and J. Dongarra. MPI – The Complete Reference, second edition. The MIT Press, 2000. 21. T. von Eicken, A. Basu, V. Buch, and W. Vogels. U-net: A user-level network interface for parallel and distributed computing. In Proceedings of the 15th ACM Symposium on Operating Systems Principles, Copper Mountain, Colorado, December 1995. 22. T. von Eicken, D. E. Culler, S. C. Goldstein, and K. E. Schauser. Active Messages: a mechanism for integrated communication and computation. In Proceedings of the 19th International Symposium on Computer Architecture, May 1992. 23. S. J. Winwood, Y. Shuf, and H. Franke. Multiple page size support in the Linux kernel. In Proceedings of Ottawa Linux Symposium, Ottawa, Canada, June 2002.
Trace Substitution Hans Vandierendonck, Hans Logie, and Koen De Bosschere Dept. of Electronics and Information Systems Ghent University Sint-Pietersnieuwstraat 41 B-9000 Gent, Belgium {hvdieren,kdb}@elis.rug.ac.be
Abstract. Trace caches deliver a high number of instructions per cycle to wide-issue superscalar processors. To overcome complex control flow, multiple branch predictors have to predict up to 3 conditional branches per cycle. These multiple branch predictors sometimes predict completely wrong paths of execution, degrading the average fetch bandwidth. This paper shows that such mispredictions can be detected by monitoring trace cache misses. Based on this observation, a new technique called trace substitution is introduced. On a trace cache miss, trace substitution overrides the predicted trace with a cached trace. If the substitution is correct, the fetch bandwidth increases. We show that trace substitution consistently improves the fetch bandwidth with 0.2 instructions per access. For inaccurate predictors, trace substitution can increase the fetch bandwidth with up to 2 instructions per access.
1
Introduction
Many applications have short basic blocks and branches that are difficult to predict. This property makes it difficult to sustain a high fetch bandwidth and thereby limits the maximum IPC. Trace caches offer a solution to this problem. Instructions that are executed consecutively are packed together into a trace and are cached as one unit for future reference. A multiple branch predictor predicts the followed path, which is used to retrieve the correct trace. A trace cache typically requires 3 branch predictions per cycle. The accuracy of the multiple branch predictor is of extreme importance, as trace caches are targeted at very wide-issue machines. As branches are typically correlated to each other, certain paths or combinations of branches will be very unlikely and will almost never occur during the execution of a program. However, destructive interference in a multiple branch predictor may result in predicting these paths anyway. This paper studies the occurrence of such “obviously wrong” paths and shows that they are identified by trace cache misses. Based on this observation, we propose trace substitution, a simple and elegant technique to increase average fetch bandwidth. When an “obviously wrong” trace prediction is detected, a cached trace is substituted for the predicted trace. This technique, although very simple and requiring little additional hardware, consistently improves the fetch bandwidth. H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 556–565, 2003. c Springer-Verlag Berlin Heidelberg 2003
Trace Substitution
557
The remainder of this paper is organised as follows. Section 2 discusses the trace cache and multiple branch predictor models and introduces trace substitution. Section 3 studies opportunities and limitations of trace substitutions and evaluates its impact for varying trace cache sizes and branch predictor sizes. Related work is discussed in section 4 and section 5 concludes the paper.
2
Trace Caches and Trace Substitution
2.1
The Trace Cache
The trace cache mechanism packs consecutively executed instructions into traces of instructions and stores them in the trace cache as a single unit [1,2]. The trace cache is typically organised as a set-associative cache. Each trace consists of at most 16 instructions and 3 conditional branches.
fetch address
BTB
Instruction Cache
RAS
MBP
Trace cache
Next Address PC mask,interchange, shift
4:1 MUX
b0 b1
history
Fetch Buffer next fetch address
Execution core
b13-b2 b12-b1 b10-b0
hit
MUX
Fill Unit
retired instructions
2^14 counters with 2^12 rows and 4 counters/row
4:1 MUX
b0
4:1 MUX
P2
P1
P0
(b) The MGAg and Mgshare predictors
(a) The fetch unit Fig. 1. Block diagram of the fetch unit containing the trace cache, the multiple branch predictor and the instruction cache.
We use the trace cache organisation of [1,3,4] (Figure 1(a)). In this organisation, the instruction cache (a.k.a. core fetch unit) and the trace cache are probed in parallel with the fetch address. The multiple branch predictor predicts the direction of three conditional branches. The branch directions and the fetch address are used to perform tag matching in the trace cache, i.e., we use path-associativity [5], meaning that multiple traces can be present starting at the same program counter. When the trace cache hits, the trace is forwarded to the processor. On a miss, 16 instructions in two consecutive cache blocks are fetched from the instruction cache. The branch target buffer (BTB) and the predicted path are used to determine how many instructions should be forwarded to the processor. The instruction cache can deliver only 1 taken branch per access. When the trace cache misses, the fill unit is instructed to start building a trace. The fill unit only builds traces on the correct path of execution using retired instructions.
558
2.2
H. Vandierendonck, H. Logie, and K. De Bosschere
Multiple Branch Prediction
The MGAg multiple branch predictor was proposed in [1,6] (Figure 1(b)). In the MGAg, the global branch history is used as an index into a pattern history table (the PC is ignored in this case). The MGAg is a direct extension of the GAg branch predictor [7] to trace caches. Conceptually, the MGAg is a GAg that is accessed three times using the global branch history, which is updated between accesses with the predicted branch directions. This idea is implemented with three parallel pattern history table (PHT) accesses [6]. The Mgshare is derived from the MGAg predictor. In a gshare predictor that is iterated three times, the PHT is accessed each time using the exclusive-or (XOR) of the branch address and the global branch history [8]. When accessing a trace cache, only one branch address is available (i.e., the starting address of a trace). Hence, we XOR this address with each of the three PHT indices. 2.3
Trace Substitution
The trace cache only stores traces along paths of execution that have actually been followed. The multiple branch predictor has to predict the future path of execution and sometimes (due to destructive interference) it predicts paths that have not been executed yet. These paths are likely to be wrongly predicted and we can use the trace cache to detect them. We show in section 3.2 that a trace cache miss is a good indicator for incorrectly predicted branches. As a trace cache miss is a tell-tale for branch mispredictions, its occurrence can be used to improve fetch bandwidth. We argued that traces stored in the trace cache are more likely to be on the correct execution path than other traces. Thus, when a trace cache miss occurs, we return a different trace instead. In particular, we return the most recently used trace with the same starting address. We call this technique trace substitution. Trace substitution requires little additional hardware. First, the hit/miss indicator of the trace cache is reused to trigger trace substitution. Secondly, a second, parallel, tag match is required to find the most recently used trace with the requested starting address. Because the trace cache is indexed using this starting address, only one cache set needs to be searched. The most recently used trace is derived from the LRU bits of the replacement policy in the trace cache. The third hardware addition is that the multiplexor that selects the returned trace has to take into account the hit/miss indicator and the second tag match. This will probably increase the critical path of the trace cache with one multiplexor delay. The fill unit builds a trace when a trace cache miss occurs. We modify this behaviour and do not build a trace when the substituted trace is on the correct path. It is still possible for new traces to enter the trace cache. When a new trace is seen, then (i) a trace cache miss occurs, (ii) trace substitution is possibly applied and (iii) the substituted trace (if any) is on the wrong path, so the new trace is built after all and stored in the trace cache. If no trace could be found to substitute, then the trace is built and stored anyway.
Trace Substitution
3
559
Evaluation
We evaluate trace substitution using 6 SPECint95 benchmarks running training inputs. The harmonic mean is labelled as H-mean. The benchmarks are compiled with Compaq’s C compiler using -O4 optimisation. At most 500 million instructions are simulated per benchmark. The baseline fetch unit has a 256-entry trace cache organised as a 4-way set-associative cache. Each trace can contain up to 16 instructions and 3 conditional branches. Traces are terminated on indirect branches and system calls. The baseline MGAg predictor uses 12 bits of global branch history and the instruction cache is 32-KB large with 32-byte blocks and is 2-way set-associative.
3.1
The Branch Predictor Changes the Trace Cache Hit Rate
The overall performance measure FIPA (fetched instructions per trace cache access) depends on both the branch prediction accuracy and trace cache miss rate in a complex way. When the branch predictor is improved and the branch prediction accuracy increases, one also expects to see the FIPA increase. This is not always the case. We investigated a 12-bit history MGAg and Mgshare, an 8kbit hybrid predictor that is optimistically probed three times in succession and a perfect branch predictor. Still other predictors were investigated, but are not shown. The predictors are evaluated in a large 16K-entry 4-way set-associative trace cache in order to mitigate the impact of trace cache misses (Figure 2). For many benchmarks, the Mgshare has a higher branch prediction accuracy but a lower FIPA than the MGAg predictor. The trace cache miss rate seems to correlate better to the FIPA, but the trace cache hit rate for the Mgshare predictor is unproportionally low.
FIPA
MBP hits
TC hits
16
100%
14
95% 90%
10 8
85%
6
80%
Hit rate (%)
FIPA
12
4 75%
2
repeat
perfect
MGAg
Mgshare
repeat
vortex
perfect
MGAg
Mgshare
repeat
perl
perfect
MGAg
Mgshare
repeat
m88ksim
perfect
MGAg
Mgshare
repeat li
perfect
MGAg
Mgshare
repeat
ijpeg
perfect
MGAg
Mgshare
repeat
gcc
perfect
MGAg
70%
Mgshare
0
H-mean
Fig. 2. Fetch throughput (FIPA), branch prediction accuracy and trace cache hit rate in a 16K-entry trace cache.
560
H. Vandierendonck, H. Logie, and K. De Bosschere
These results clearly show that the branch predictor has a strong influence on the trace cache hit rate. When the branch predictor is perfect, the trace cache almost never misses. As the trace cache is very large (16K entries), it can contain practically any trace that is required, so trace cache misses should not occur. However, when the branch predictor becomes imperfect, the number of trace cache misses increases significantly. 3.2
On a Trace Cache Miss, the Branch Predictor Is Likely Wrong
The strong dependence of the trace cache miss rate on the branch prediction suggests that the branch predictor produces, at times, predictions that are so unlikely that the trace cache cannot provide them. This effect is measured for a 12-bit history MGAg predictor (Table 1). The left half of this table shows the fraction of trace cache misses that are due to a path misprediction. When the trace cache is sufficiently large, over 80% of trace cache misses is caused by an incorrect branch prediction. In the smaller trace caches, many misses occur due to contention (capacity misses) and are not related to the branch predictor. Table 1. Fraction of trace cache misses where the path is mispredicted (left) and the fraction of path mispredictions that are caught by monitoring trace cache misses. The trace cache size is varied and the branch history length is 12 bits. TC size gcc ijpeg li m88ksim perl vortex H-mean
Fraction of trace cache misses 64 256 1024 4096 16384 28.3% 34.0% 51.7% 78.5% 86.7% 7.1% 19.5% 65.7% 84.0% 85.0% 10.8% 35.6% 87.3% 92.5% 92.6% 9.8% 21.4% 55.9% 81.3% 82.9% 8.7% 17.6% 51.6% 80.1% 80.4% 15.8% 28.0% 50.7% 76.7% 82.3% 12.0% 25.1% 59.3% 82.0% 84.9%
Fraction of path mispredictions 64 256 1024 4096 16384 89.5% 85.8% 78.4% 69.0% 63.6% 44.1% 27.6% 16.5% 11.4% 10.5% 39.0% 29.6% 24.3% 23.3% 23.2% 82.5% 63.0% 53.5% 46.1% 46.2% 84.9% 71.0% 51.5% 45.7% 45.0% 98.7% 97.8% 95.0% 91.6% 88.3% 68.9% 55.9% 44.9% 39.0% 37.6%
The right part of Table 1 shows the fraction of mispredicted traces that actually lead to a trace cache miss. In a small trace cache, many misses occur, so there is a large probability of detecting path mispredictions. Up to 70% of the mispredicted paths can be detected. As the trace cache becomes larger, just around 40% of the path mispredictions can be detected. The branch prediction accuracy also has an important impact on the applicability and utility of trace substitution. We held the trace cache size constant at 256 entries and varied the branch history length of the MGAg predictor (Table 2). On average, between 14% and 71% of the trace cache misses indicate path mispredictions. The fraction of mispredicted paths that can be detected by trace cache misses decreases with increasing branch predictor size. This result is somewhat surprising, as one would expect that a higher prediction accuracy
Trace Substitution
561
Table 2. Fraction of trace cache misses where the path is mispredicted (left) and the fraction of path mispredictions that are caught by monitoring trace cache misses. The branch history length is varied and the trace cache is held constant at 256 traces. hist. length gcc ijpeg li m88ksim perl vortex H-mean
Fraction of trace cache misses 0 4 8 12 16 67.4% 63.6% 57.6% 34.0% 14.8% 56.0% 46.7% 34.8% 19.5% 12.8% 91.7% 85.2% 63.4% 35.6% 23.2% 79.4% 80.0% 50.3% 21.4% 10.5% 77.3% 77.7% 54.3% 17.6% 10.8% 57.9% 56.7% 46.0% 28.0% 14.3% 70.5% 66.8% 50.2% 25.1% 13.9%
Fraction of path mispredictions 0 4 8 12 16 88.4% 88.9% 89.4% 85.8% 77.1% 57.2% 51.4% 41.9% 27.6% 21.6% 67.2% 66.5% 53.4% 29.6% 22.1% 62.5% 95.3% 84.4% 63.0% 48.7% 90.3% 91.3% 87.6% 71.0% 58.3% 98.8% 98.3% 98.4% 97.8% 96.7% 75.8% 79.9% 72.5% 55.9% 46.5%
combined with a (more or less) constant trace cache miss rate would result in an increase in the fraction of detected mispredictions. However, not all path mispredictions will miss the trace cache. Those that do are “obvious” mispredictions; those that do not are harder to detect. As the branch predictor becomes larger, the amount of “obvious” mispredictions decreases faster than the others. Trace substitution can only work when the trace on the correct path is present in the trace cache. If it is not, then there is no point in returning another trace that is also on the wrong path. We have found that the correct path trace is present in the trace cache for 35%–87% of the mispredicted paths that miss in the trace cache, for trace caches with 64 to 16K traces and the baseline predictor. When varying the predictor size, the correct trace is available in 6%–38% of the cases. Hence, when a mispredicted path is detected, it will not always be possible to substitute the correct trace. However, even when the correct trace can not be substituted, no harm is done in trying, as both the predicted path and any other wrong-path trace are on the wrong path of execution. 3.3
Trace Substitution
Now that we have discussed the potentials and limitations of trace substitution, let us investigate its impact on performance. We measured the FIPA for a 12bit history MGAg predictor and trace cache of sizes 64 to 16K traces, all 4way set-associative (Figure 3(a)). The FIPA is shown for the baseline MGAg predictor, the MGAg predictor with trace substitution and a perfect branch predictor. In the small trace caches, the prominent performance bottleneck is trace cache misses related to the limited capacity of the trace cache. Hence, a perfect branch predictor does not improve performance much. As the trace cache becomes larger, the multiple branch predictor becomes a more important bottleneck. Consequently, the gap between the curves for the MGAg and perfect predictors increases with increasing trace cache size. Trace substitution increases the FIPA obtainable with the MGAg predictor; around 20% to 40% of the gap between the baseline MGAg predictor and the
562
H. Vandierendonck, H. Logie, and K. De Bosschere
12.0
14
11.5
13
11.0 10.5
FIPA
FIPA
12 11
10.0 9.5
10
9.0
MGAg MGAg+subst
9
perfect
MGAg
8.5
MGAg+subst
8.0
8 64
256
1024
4096
16384
Trace cache size (traces)
(a) Varying trace cache size
0
2
4
6
8
10
12
14
16
Branch history length
(b) Varying MGAg size
Fig. 3. The impact of trace substitution on performance (FIPA).
perfect predictor is closed. However, as trace substitution only kicks in on a trace cache miss, the obtainable performance increase strongly depends on the trace cache miss rate. Figure 3 shows that less and less of the performance potential is reached with larger trace caches. The cause of this can be found in Table 1: as the trace cache becomes larger, fewer path mispredictions are detected. The performance increase therefore stagnates as the trace cache becomes larger. Trace substitution can hide the imperfectness of a poor branch predictor. When varying the history length (and simultaneously the size) of the MGAg branch predictor, the benefits of trace substitution become much clearer. In this experiment, we used a 256-entry trace cache (Figure 3(b)). For the small and inaccurate branch predictors, trace substitution improves performance with 1 to 2 useful instructions fetched per trace cache access. Even for the MGAg with zero bits of global branch history, trace substitution raises the FIPA to the same level as a 10-bit history MGAg without trace substitution. For the largest of branch predictors, the FIPA is increased over a small amount, e.g., 0.2 instructions per access for the baseline predictor. More importantly, trace substitution enables one to achieve the same performance with a 16 times smaller branch predictor (e.g., 10 vs. 14 bits of history). Trace substitution depends totally on the co-occurrence of a mispredicted path and a trace cache miss. Hence, the utility decreases for a large trace cache (fewer trace cache misses and fewer possibilities to detect mispredicted paths) and larger branch predictors (fewer path misprediction and thus lack of reason to substitute the trace). Figure 4 shows the usage and accuracy of trace substitution. The usage is defined as the fraction of trace cache accesses for which a trace was substituted. The accuracy is defined as the ratio of the correct substitutions over the total number of substitutions. The usage drops of with larger
Trace Substitution
563
45% Accuracy
40%
Usage
35% 30% 25% 20% 15% 10% 5% 0% 0
2
4
6
8
10
12
14
16
Branch history length
Fig. 4. Accuracy of trace substitution in the baseline trace cache.
branch predictors, as more traces are predicted correctly by the branch predictor. On the other hand, when trace substitution is applicable, it is more accurate in larger branch predictors. Overall, the accuracy of trace substitution is low: the correct trace is returned in less than half of the cases. In the reported experiments, we always returned the most recently used trace having the correct start address. Experiments were made with other choices, e.g., returning a randomly selected trace and a statebased predictor, but these were found to have approximately the same accuracy. Notwithstanding its low accuracy, trace substitution improves the average fetch bandwidth. The reason for having a performance improvement with such low accuracy is that, assuming that the wrong path is predicted on a trace cache miss, it does not matter much whether we return a wrong-path trace from the trace cache, or fetch along another wrong path using the instruction cache. In the end, both paths are wrong. Another reason for the low accuracy is that the correct-path trace is not always present in the trace cache. Trace substitution is applied as soon as any trace with the correct start address is found but this trace is not always on the correct path of execution. In section 3.2 we estimated that the accuracy is limited to 6%–38%, depending on the branch predictor size, exactly for this reason. We have also performed the above experiments with the proposed Mgshare predictor. This predictor is by itself less accurate than the MGAg predictor. However, when trace substitution was added, it performed as good as the MGAg with trace substitution. Lack of space prohibits us from treating this in detail.
4
Related Work
Trace caches were introduced to increase the fetch bandwidth for superscalar processors over the single basic block limit [1,2,9]. Two approaches for multiple
564
H. Vandierendonck, H. Logie, and K. De Bosschere
branch prediction can be discerned. The first approach is to adapt a singlebranch predictor to multiple branch prediction. This approach was taken in [1, 2,5,6,10] and is also followed in this paper. A second approach is to predict full traces at once and has mainly been investigated in conjunction with blockbased trace caches [11,12,13]. In [13], it is proposed to predict the next trace at completion time. This has the advantage that the predictor can be indexed using non-speculative history information. Trace substitution is not useful when only cached traces are predicted, as in [11,12]. Selective trace storage does not store blue traces (i.e.: traces containing only not-taken branches) in the trace cache, as these can be retrieved equally fast from the instruction cache [3]. Trace substitution can be adapted for selective trace storage by adding additional tags to each set of the trace cache. These tags can only be used for blue traces and indicate that a specific blue path has been followed. The blue trace is still stored in the instruction cache. Trace substitution is not applied when a blue tag hits. Trace substitution is an idea similar to branch inversion [14], a technique applicable to single-branch predictors. With branch inversion, a confidence predictor is used to detect wrong branch predictions. When a branch prediction is assigned low confidence, the predicted branch direction is inverted. Trace substitution does not require a confidence predictor, but is triggered by trace cache misses. On the other hand, trace substitution is more complicated than branch inversion because in the former a different trace has to be predicted while in the latter, it suffices to simply invert the branch direction.
5
Conclusion
Multiple branch predictors sometimes predict a path of execution that is so unlikely that the trace cache cannot provide it. This paper showed that trace cache misses can be used to detect such mispredictions. We proposed trace substitution, a new technique that returns a cached trace when the predicted trace misses in the trace cache. We showed that between 40%–76% of the branch mispredictions can be detected by trace cache misses, depending on the trace cache and branch predictor sizes. Furthermore, between 12%–85% of the trace cache misses are indeed caused by mispredicted branches. The low numbers correspond to small trace caches with many capacity misses. Trace substitution increases the fetch bandwidth consistently. For small branch predictors, the FIPA is improved with 1 to 2 useful instructions per cycle, obtaining the fetch bandwidth of a 10 to 12-bit history MGAg. For large branch predictors, the FIPA is increased with 0.2 instructions per access or the same FIPA can be obtained with a 16-times smaller branch predictor. Acknowledgements. The authors thank the reviewers for their helpful comments. Hans Vandierendonck is supported by the Flemish Institute for the Promotion of Scientific-Technological Research in the Industry (IWT).
Trace Substitution
565
References 1. E. Rotenberg, S. Bennett, and J. E. Smith, “Trace cache: A low latency approach to high bandwidth instruction fetching,” in Proceedings of the 29th Conference on Microprogramming and Microarchitecture, pp. 24–35, Dec. 1996. 2. S. Patel, M. Evers, and Y. Patt, “Improving trace cache effectiveness with branch promotion and trace packing,” in Proceedings of the 25th Annual International Symposium on Computer Architecture, pp. 262–271, June 1998. 3. A. Ram´ırez, J. Larriba-Pey, and M. Valero, “Trace cache redundancy: Red & blue traces,” in Proceedings of the 6th International Symposium on High Performance Computer Architecture, Jan. 2000. 4. H. Vandierendonck, A. Ram´ırez, K. De Bosschere, and M. Valero, “A comparative study of redundancy in trace caches,” in EuroPar 2002, pp. 512–516, Aug. 2002. 5. S. Patel, D. Friendly, and Y. Patt, “Evaluation of design options for the trace cache fetch mechanism,” IEEE Transactions on Computers, vol. 48, pp. 193–204, Feb. 1999. 6. E. Rotenberg, S. Bennet, and J. Smith, “Trace cache: A low latency approach to high bandwidth instruction fetching,” Tech. Rep. 1310, University of Wisconsin Madison, Apr. 1996. 7. T.-Y. Yeh and Y. N. Patt, “Alternative implementations of two-level adaptive branch prediction,” in Proceedings of the 19th Annual International Conference on Computer Architecture, pp. 124–134, May 1992. 8. S. McFarling, “Combining branch predictors,” Tech. Rep. WRL-TN36, Western Research Laboratory, 250 University Avenue, Palo Alto, California 94301, USA, June 1993. 9. A. Peleg and U. Weiser, “Dynamic flow instruction cache memory organized around trace segments independent of virtual address line,” U.S. Patent Number 5.381.533, Jan. 1995. 10. T.-Y. Yeh, D. Marr, and Y. Patt, “Increasing the instruction fetch rate via multiple branch prediction and a branch access cache,” in ICS’93. Proceedings of the 1993 ACM International Conference on Supercomputing, pp. 67–76, July 1993. 11. B. Black, B. Rychlik, and J. Shen, “The block-based trace cache,” in Proceedings of the 26th Annual International Symposium on Computer Architecture, pp. 196–207, May 1999. 12. Q. Jacobson, E. Rotenberg, and J. Smith, “Path-based next trace prediction,” in Proceedings of the 30th Conference on Microprogramming and Microarchitecture, pp. 14–23, Dec. 1997. 13. R. Rakvic, B. Black, and J. Shen, “Completion time multiple branch prediction for enhancing trace cache performance,” in Proceedings of the 27th Annual International Symposium on Computer Architecture, pp. 47–58, May 2000. 14. S. Manne, A. Klauser, and D. Grunwald, “Branch prediction using selective branch inversion,” in Proceedings of the International Conference on Parallel Architectures and Compilation Techniques, pp. 48–56, Oct. 1999.
Optimizing a Decoupled Front-End Architecture: The Indexed Fetch Target Buffer (iFTB) Juan C. Moure, Dolores I. Rexachs, and Emilio Luque1 Computer Architecture and Operating Systems Group, Universidad Autónoma de Barcelona. 08193 Barcelona (Spain) {JuanCarlos.Moure, Dolores.Rexachs, Emilio.Luque}@uab.es
Abstract. A decoupled front-end architecture relies on a separate fetch predictor that is critical in determining the fetch bandwidth. The Fetch Target Buffer (FTB) is a multilevel fetch predictor design. In this paper we propose and analyze a variation of the FTB, the indexed Fetch Target Buffer (iFTB), that uses an extra, very fast, index table to speedup its operations. The iFTB achieves very accurate predictions by adapting a complex hybrid branch predictor to suit its short cycle time. With a low hardware cost, the iFTB delivers instructions between 15% and 45% faster than the FTB.
1
Introduction
The performance of a processor is limited by the performance of each of its parts. The front-end is one of the most critical elements. A decoupled front-end architecture is able to provide high fetch bandwidth while maintaining scalability, [13]. As shown in Fig. 1, a decoupled front-end isolates the prediction task from the rest of the front-end’s tasks (fetching the actual instructions and preparing them for execution). The fetch predictor generates references to fetch blocks and inserts them into a decoupling fetch target queue (FTQ). A fetch block is a set of instructions stored into consecutive memory addresses, and is defined by its starting address and size. One fetch block may involve several cache accesses and decode cycles. This allows the fetch predictor to go ahead of the front-end, and create a large fetch window in the FTQ that can be used to exploit parallelism. For example, a large L1 instruction cache may be efficiently pipelined and multi-banked to provide a very high bandwidth. This organization makes the fetch predictor become the critical element of the front-end.
Fetch Predictor
fetch blocks
FTQ
Decoupled Front-End iCache Decode
Execute target mispredict
Fig. 1. The elements of a decoupled front-end and its interface with the execution core 1
This work was supported by the MCyT-Spain under contract TIC 2001-2592 and partially supported by the Generalitat de Catalunya – Grup Recerca Consolidat 2001 SGR-00218
H. Kosch, L. Böszörményi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 566–575, 2003. © Springer-Verlag Berlin Heidelberg 2003
Optimizing a Decoupled Front-End Architecture
567
This paper presents and analyzes the indexed Fetch Target Buffer (iFTB), a very fast and scalable fetch predictor based on the Fetch Target Buffer (FTB), [13]. It uses microarchitectural techniques, such as way prediction and overridden prediction, to aggressively reduce its cycle time at the expense of a small increase in fetch hazards. Our work addresses fetch prediction and assumes it is the only performance bottleneck, therefore the FTQ never gets full. In this situation, simulation results indicate that the iFTB design, compared to the original FTB design, increases fetch bandwidth by between 15% and 45%. These results were obtained using CACTI to estimate cycle times and hazard penalties, [14], and using a cycle-by-cycle simulator to obtain the rate of hazards for the SPECint00 benchmark suit. Section 2 analyzes the performance issues of the fetch predictor, reviews previous and related work and discusses the paper contribution. The iFTB design is described in full detail in section 3. Section 4 shows the experimental methodology and the main results. Section 5 outlines the conclusions and sets out future lines of research.
2
Increasing the Fetch Predictor’s Performance
Equation 1 models the performance of the fetch predictor. Fetch bandwidth (BW) is augmented by increasing the number of instructions per fetch block (ifb), by reducing the fetch predictor’s cycle time (tc), or by reducing the average number of cycles per fetch block (cfb) (which measures the impact of hazards and the parallelism inside the fetch predictor). BW = ifb / (tc · cfb) .
(1)
Previous Work. The branch target buffer (BTB) predicts single branches and was initially designed for single-issue processors, [8],[12]. Yeh and Patt’s proposal, [17], increases fetch bandwidth by enlarging fetch blocks (ifb) so as to always include a branch instruction at the end. Reinmann et al., [13], proposed the decoupled front-end architecture and the Fetch Target Buffer (FTB). The FTB is a two-level fetch predictor that achieves high performance by reducing tc and further increasing ifb. A small and fast L1-FTB reduces tc, while the penalty of L1-FTB misses is kept down (and cfb is maintained low) by using a large L2-FTB. They also proposed embedding never-taken branches into a fetch block to further increase ifb. Target mispredictions are detected in the execution stage and involve large penalties. Two-level branch predictors, [17], and hybrid branch predictors, [9], increase prediction accuracy by exploiting the growing chip area available for storing branch history. However, as noticed by Jiménez et al., [5], prediction delay may become a performance concern. In fact, increasing ifb and reducing tc puts more pressure on the latency of branch predictors. An overridden organization may be a successful way to deal with this latency. Two examples of commercial processors that use such scheme are the Alpha 21264, [7], and the Intel Itanium, [16]. Predicting several branches per cycle increases the parallelism inside the fetch predictor and reduces the number of cycles per fetch block, cfb. However, current proposals, like [18],[10], rely on larger data structures and complex selection logic, which increase tc and may also increase the target misprediction ratio.
568
J.C. Moure, D.I. Rexachs, and E. Luque
The starting point. Each FTB’s entry contains the data of a fetch block and the data to predict the next fetch block. A fetch prediction requires two steps, which determine the FTB design’s cycle time (see Fig. 2.a). First, the current fetch block’s address (addri) indexes the FTB to obtain the fetch block’s size, target address, branch type, and branch local history. Second, the full fall-through address is assembled and the branch prediction outcome is used to select the next fetch block’s address (addri+1). As feature sizes shrink due to technology enhancements, the number of available transistors on a processor chip increases, but many trends indicate that circuit delays are becoming increasingly limited by the amount of wire (and memory) in the critical path, [1]. An FTB hierarchy solves this tradeoff, but we are trying to further reduce tc.
Branch Pred L1-FTB addri
addri+1
a) FTB-only design
Branch Pred FTQ L1-FTB lookahead update iTbl
FTQ
idxi
idxi+1
index override
b) iFTB design
Fig. 2. Block diagram of the FTB-only and iFTB designs. Only the hit case is depicted
Our proposal. We have modified the FTB design by including a small table to provide an extra level of indirection (Fig. 2.b). The index table (iTbl) has the same number of entries as the FTB, but each entry contains a single index that points to the iTbl/FTB entry containing the next fetch block. Fetch prediction is performed at the iTbl speed (as a sequence of indexes idxi, idxi+1 …) while the FTB (which is pipelined to accommodate its longer delay) is addressed with these indexes to generate the actual fetch block addresses and sizes. The design has the following issues: 1. iTbl indexes provide a form of way prediction, hiding the delay of a set-associative access scheme whilst maintaining its low miss ratio 2. the separate iTbl allows the L1-FTB access to be taken out of the critical path 3. the single index stored into an iTbl entry provides a form of lookahead branch prediction. This scheme aggressively reduces the iFTB’s cycle time, since the conditional branch predictor and the selection logic are taken out of the critical path 4. the branch predictor updates the iTbl’s lookahead prediction on the fly, and overrides the iTbl prediction to achieve high accuracy without affecting tc Contribution and Related Work. Way prediction applied to the iCache was studied in [3],[7],[19]. Our design is unique in using way prediction in a decoupled front-end, and storing way predictors in a separate table. This paper extends our proposal made in [11] with a more aggressive approach and the modeling of a 2-level hierarchy. We have adapted the hybrid local/global predictor of the Alpha 21264 to suit the very short cycle time requirements of our design. Like in [13], we use the lookahead proposal of [17] to hide the delay of the two-level local predictor. We also use an overridden scheme ([5],[7],[16]) to hide the latency of the hybrid selection. Finally, since the iFTB has a very short cycle time, access to the global and meta prediction tables may be critical. We then propose a solution to pipeline these tables using a variant of the proposal described in [18].
Optimizing a Decoupled Front-End Architecture
569
Our design optimizes the single branch prediction case. It could be extended to predict several branches per cycle, like in [18,10]. The use of a trace cache, [15], is also an orthogonal approach to ours. A trace cache simplifies the fetching task, reducing the penalty of FTB misses and target mispredictions. A decoupled front-end can take full advantage of a trace cache, since its larger access time may be pipelined. Finally, a collaboration of software and hardware allows hardware complexity to be reduced and cycle time to be decreased. Our hardware-based solution can be helped by the compiler, but it does not rely on it to provide good performance.
3
The indexed Fetch Target Buffer (iFTB)
The index table provides an extra level of indirection to the FTB. It holds a small representation of the graph of the predicted transitions between fetch blocks. The graph is traversed at the iTbl speed, while a pipelined FTB generates the actual fetch block addresses and sizes and inserts them into the FTQ. Parallel with this, a complex (slow) branch predictor dynamically modifies the graph (iTbl’s contents) to increase prediction accuracy. Sometimes but not often, the iTbl’s outcome is wrong (fetch hazard), and the iFTB must be stopped and rolled-back, incurring a delay penalty (increasing cfb). The iFTB predicts the fetch trace as a sequence of indexes. As shown in Fig 3.a, several fetch hazards may force the index sequence to be rolled-back: • the required fetch block is placed in a different L1-FTB way (way miss) • the fetch block is not found in the L1-FTB or L2-FTB (L1/L2 miss) • a new fetch block is identified at the decode stage (miss-fetch) • a target misprediction is detected at the execution stage • the lookahead prediction is overridden by the global predictor (global-use) 3.1 Way Prediction. Way and L1 misses are detected when comparing the address predicted by the L1-FTB with the address tag read from the L1-FTB at the next access. To reduce the occurrence and penalty of way misses, the L1-FTB stores the way of the fall-through and the target fetch blocks. These fields are updated after each way miss or L1 miss. After a fetch hazard, the appropriate way predictor is used. On a miss-fetch, the iTbl operation is restarted using a random way prediction. On a replacement, way predictors are copied into the L2-FTB to reduce way miss hazards. pipelined global and meta-prediction
L2-FTB L1-FTB update iTbl idxi+1
override
gh control logic
Fetch hazards: way/L1/L2 miss, missfetch, …
a) iFTB index recovery
hybrid lookahead two-level, local prediction selection
pi-1
sHT gPHT
L1-FTB
gpi lpi lhi-1
si lhi pi
lPHT
lpi+1
b) lookahead, overridden branch predictor
Fig. 3. iFTB design with a L2-FTB and a detailed block diagram of the branch predictor
570
J.C. Moure, D.I. Rexachs, and E. Luque
3.2 L2-FTB. To reduce the penalty of L1 misses, the access to the L2-FTB is initiated as soon as a fetch block address is available, before concluding the L1 tag comparison. If, finally, there is no L1 miss, the L2 access is cancelled. Otherwise, the data from the L2-FTB is copied into the L1-FTB and the replaced data from the L1FTB is copied into the L2-FTB. In both cases, a LRU replacement policy is used. A L2 miss stalls the iFTB. Then, when the FTQ is empty, the iCache provides consecutive instructions to the decode stage until a miss-fetch is detected and a new fetch block is inserted into the L1-FTB and L2-FTB. As in [13], not-taken branches are ignored when identifying a new fetch block. 3.3 Conditional Branch Predictor. Since the penalty of branch mispredictions is extremely high, we have integrated into our design the accurate branch predictor of the Alpha 21264, [7]. Its hybrid algorithm uses a meta-predictor that chooses between a local and a global prediction. The local history stored in the FTB (10 previous results of the fetch block’s ending branch) is used to index a local Pattern History Table (lPHT: 3-bit counter) and predict the branch’s next occurrence. The global Pattern History Table and selection History Table (gPHT, sHT: 2-bit counters) are indexed using the previous global history (the outcome of the last 12 branches). The high prediction accuracy of this algorithm, though, comes at the expense of an increase in prediction latency. We propose three modifications to the original algorithm to make it suit the iFTB’s short cycle time: • a lookahead scheme avoids the delay of the two-level local predictor • the long latency to access the gPHT and sHT tables is hidden by pipelining • an overridden scheme hides the delay of selection between local and global prediction in most of the cases Lookahead Local Predictor. Figure 3.b. illustrates the lookahead scheme suggested in [17] to hide the delay of the lPHT. As soon as the final prediction, pi, is generated (before it is validated), it is merged with the local history of a fetch block, lhi-1, to obtain the new local history, lhi, and indexes the lPHT to get the local prediction for the branch’s next occurrence, lpi+1. If the lookahead prediction changes (lpi+1 ≠ lpi) the iTbl is updated on-the-fly to reflect that change. This modification of [17] allows predicting at the iTbl’s speed with the accuracy of a two-level local predictor. The prediction must be recalculated after a branch misprediction. Pipelined gPHT/sHT. To achieve the required prediction rate, the gPHT/sHT must be pipelined. Our solution is based on the idea proposed in [18] for multiple branch prediction. The gPHT/sHT access starts before the full global history is available, using an incomplete index, so that all the adjacent entries corresponding to the missing bits are read together. Then, at the output of the gPHT/sHT, the remaining history bits (already available) are used to select the final outcome. Figure 3.b illustrates this technique assuming that there is only one missing bit (pi-1). Overridden Hybrid Predictor. The high prediction accuracy of a hybrid method relies on its ability to dynamically select the best predictor in each particular case. The iFTB design uses the two-level (lookahead) local predictor by default. Later, the hybrid prediction is calculated and overrides the previous prediction if the global
Optimizing a Decoupled Front-End Architecture
571
prediction differs and the meta-predictor chooses the global predictor. This hazard, that we call a global-use hazard, provokes an iFTB squash and a delay penalty. 3.4 Indirect Jump Prediction. Fetch blocks ending in a jump are also managed by the iTbl. Unconditional jumps are trivially predicted. Indirect jumps use the FTB’s lookahead bit as a hysteresis counter to avoid updating the iTbl until two consecutive mispredictions occur. The FTB stores the way predictors of the previous two targets. A 32-entry Return Address Stack (RAS), [6], is used to improve the prediction of return instructions. If the FTB’s type field identifies a return, an address popped from the RAS is compared to the fetch block’s address predicted by the iTbl. If they differ, the RAS prediction overrides the iTbl resulting in a delay penalty (ras-use hazard).
4
Performance Evaluation
We evaluate the performance advantage of our proposal by comparing the iFTB with the FTB-only design in Fig. 2 (which is very similar to the proposal in [13]). We have used SimpleScalar, [2], to build a simulator modeling the iFTB components, and have used the SPECint00 benchmark suit compiled for the Alpha ISA (cc DEC 5.9, -O4). 4.1 Preliminary Analysis of Hazards. Table 1 lists some data for the SPECint00 benchmark suit, used in our experiments. For each benchmark, several billion instructions are skipped (column 3) before simulation measures are taken for an execution window of 400 million instructions. Column 4 indicates the number of L1-FTB entries required to reduce the L1 miss rate to less than 1%. Columns 5 and 6 list the average fetch block size when nevertaken branches are not embedded and when they are. The increase in the fetch block size due to embedding is 9.4% on average. Columns 7 and 8 list the misprediction rate per retired instruction due to conditional branches and indirect jumps. A 32-entry RAS reduces return mispredictions to less than 0.003%. This data provides an insight into the instruction working set and the predictability of branches for each benchmark. Table 1. Simulation data and results (unbounded L1-FTB). Branch and jump misprediction rates are per retired instruction, while other rates are per retired fetch block 1.Bench bzp crafty eon gap gcc gzip mcf parser perlbmk twolf vortex vpr
2. input graphic reference cook reference 166.i random reference reference splitmail reference vortex1 reference
3. skip 4. L1miss 0, consists of k conventional lists, called segments, and k − 1 points between the segments: ½
½
¾
¾
¿
¿
½
If parameter k is irrelevant, we simply speak of a plist instead of a k-plist. Conventional lists are obviously a special case of plists. To distinguish between functions on lists and plists, we prefix the latter with the letter p, e. g. pmap. To transform between lists and plists, we use the following two functions: – list2plist k transforms a list into a plist, consisting of k segments and k −1 points. It partitions an arbitrary list into k segments: list2plist k (l1 + + [a1 ] + + ··· + + [ak−1 ] + + lk ) = [ l1 , a1 , . . . , ak−1 , lk ]
686
H. Bischof, S. Gorlatch, and E. Kitzelmann
Our further considerations are valid for arbitrary partitions but, in practice of parallelism, one tries to obtain segments of approximately the same size. – plist2list k is the inverse of list2plist k , transforming a k-plist into a conventional list: + [a1 ] + + ··· + + [ak−1 ] + + lk plist2list([ l1 , a1 , . . . , ak−1 , lk ]) = l1 + We now develop a parallel implementation for a distributed version of scanrl , function pscanrl , which computes scanrl on a plist: 1 , 2 ) = plist2list ◦ pscanrl ( 1 , 2 ) ◦ list2plist scanrl ( k
We introduce the following auxiliary skeletons as higher-order functions on plists, omitting their formal definitions and illustrating them instead graphically: – pmap l g applies function g, which operates on conventional lists, to all segments of a plist: ½
·½ ½
– pscanrl p(⊕, ⊗) applies function scanrl (⊕, ⊗), defined in (2), to the list containing only the points of the argument plist:
¨ ª [
]
– pinmap l (, ⊕, ⊗) modifies each segment of a plist depending on the segment’s neighbouring points, using operation ⊕ for the left-most segment, ⊗ for the right-most segment and operation for all inner segments, where operation is a three-adic operator, i. e. it gets a pair of points and a single point as parameters:
¨
½
½ª
¬
·½
½
·½
½
– pinmap p(⊕, ⊗) modifies each single point of a plist depending on the last element of the point’s left neighbouring segment and the first element of the point’s right neighbouring segment.
½
¨
ª
·½
·½
The implementations and time complexity analysis of these functions are provided in the next section.
Cost Optimality and Predictability of Parallel Programming with Skeletons
3.2
687
A Cost-Optimal Implementation of Double-Scan
In this section, we present a theorem that shows that the distributed version of the double-scan skeleton can be expressed using the auxiliary skeletons introduced above. We use here the following definition: a binary operation is called to be associative modulo , iff for arbitrary elements a, b, c it holds: (a b) c = (a b) (b c). Usual associativity is associativity modulo operation first, which yields the first element of a pair. 1 , 2 , 3 and 4 be binary operators, such that 1 and 3 Theorem 2. Let 1 , 2 ) = scanlr ( 3 , 4 ), and 2 is associative modulo 4 . are associative, scanrl ( 5 be a three-adic operator, such that (a, a 3 c) 5 b = a 3 (b 1 c). Moreover, let Then, the double-scan skeleton pscanrl on plists can be implemented as follows: 1 , 2 ) = pinmap l ( 5 , 1 , 3 ) ◦ pscanrl p( 1 , 2 ) pscanrl ( 2 , 4 ) ◦ (pmap l scanrl ( 1 , 2 )) ◦ pinmap p(
(4)
5 , For the theorem’s proof, see [2]. To help the user in finding the operator 5 3 we show in [2] how can be generated if (a ) is bijective for arbitrary a and if 3 distributes over 1 . Since pscanrl ( 1 , 2 ) = pscanlr ( 3 , 4 ), equality (4) holds 3 4 also for pscanlr ( , ). Let us analyze the pscanrl implementation (4) provided by Theorem 2. On a parallel machine, we partition plists so that each segment and its right “border point” are mapped to a processor. The last processor contains no extra point because there is no point to the right of the last segment in a plist. We further assume that all segments are of approximately the same size. The right-hand side of (4) consists of four stages executed from right to left, whose parallel time complexity we now study, relying on the graphical representation in Section 3.1: 1 , 2 ) can be computed by simultaneously ap1. The function pmap l scanrl ( 1 , 2 ) on all processors, provided that the argument plist is plying scanrl ( 1 , 2 ) = partitioned among p processors as described above. Because scanrl ( 3 , 4 ), we can apply either of both on all processors, so the minscanlr ( imum of their runtimes should be taken. Thus, the time complexity is + t , t + t } ∈ O(n/p), where t , . . . , t T1 = (n/p − 2) · min{t 1 2 3 4 1 4 1 4 denote the time for one computation with operators , . . . , , respectively. 2 , 4 ), each processor sends its first element to the 2. To compute pinmap p( preceding processor and receives the first element from the next processor. 2 and 4 are applied to the last element of each processor: Then operations
/* first buffers 1st element of each processor */ /* first_next receives 1st element of following processor */ MPI_Sendrecv(first, preceding,..., first_next, following,...); if ( !last_processor ) { otwo(last, point); ofour(point, first_next); /* last denotes the last element of each segment */ }
688
H. Bischof, S. Gorlatch, and E. Kitzelmann
The resulting time complexity is T2 = ts + tw + t + t ∈ O(1), where ts 2 4 is the communication startup time and tw the time needed to communicate one element of the corresponding datatype. 1 , 2 ) applies scanrl ( 1 , 2 ) on the 3. As described in Section 3.1, pscanrl p( points of the argument plist, which are distributed across the first p − 1 processors. Since DS is an instance of the DH skeleton, the generic DH implementation developed in [8] can be used for this step: for (dim=1; dim a->b to create a process, the programmer can concentrate on partitioning the algorithm into parallel subtasks, thereby taking into account issues like task granularity, topology, and data distribution. Eden processes are eagerly instantiated, and instantiated processes produce their output even if it is not demanded. These deviations from lazy evaluation aim at increasing the parallelism degree and at speeding up the
734
J. Berthold et al.
distribution of the computation. In general, a process is implemented by several threads concurrently running in the same processor, so that different values can be produced independently. The concept of a virtually shared global graph is avoided, to save the administration costs while paying the price of possibly duplicating work. Processes communicate via unidirectional channels which connect one writer to exactly one reader. When trying to access input which is not available yet threads are temporarily suspended. The type class Trans (short for transmissible) comprises all types which can be communicated and provides overloaded, implicitly used functions for sending values. All primitive types like Int, Bool, Char etc., pre- and user-defined algebraic types2 as well as function and process abstraction types belong to Trans. Example: The following function is at the heart of a simple ray-tracer program. It computes an image with y lines and x columns of a scene consisting of spheres. The sequential function body of the ray function is simply the expression map (traceLine x world) [0..y-1]. The parallel version produces the image by several processes each computing a chunk of lines: ray :: Int -> Int -> Int -> [Sphere] -> [[RGB]] ray chunk x y world = concat ([process (map (traceLine x world)) # linenumbers | linenumbers () for sending values along communication channels where the channel is known from the context of its application. The additional functions tupsize and writeDCs shown in Fig. 3 will be explained later. The context NFData (normal form data) 3 4
Note that, in GHC, primitive operations and types are distinguished from common functions and types by # as the last sign in their names. With the same name as the primitive operation, except for the #.
736
J. Berthold et al.
is needed to ensure that transmissible data can be fully evaluated (using the overloaded function rnf (reduce to normal form)) before sending it (using the primitive operation sendVal# wrapped by sendVal). Lists are transmitted in a stream-like fashion, i.e. element by element. For this, sendChan is specialized to sendStream which first evaluates each list element to normal form and transmits it using sendHead (see Fig. 3).
class NFData a => Trans a where sendChan :: a -> (); tupsize :: a -> Int; writeDCs :: ChanName a -> a -> (); --instance Trans a => Trans [a] where sendChan x = sendStream x
sendChan x = rnf x ‘seq‘ sendVal x tupsize _ = 1 writeDCs (cx:_) x = writeDC cx x default definitions, changed appropriately for tuples and lists
sendStream :: Trans a => [a] -> () sendStream [] = sendVal [] sendStream (x:xs) = (rnf x) ‘seq‘ ((sendHead x) ‘seq‘ (sendStream xs)) Fig. 3. Type Class Trans with List Instance
Before any communication can take place, a channel must be created and installed. For this, the functions shown in Fig. 4 are provided. The RTS equivalent to channels is a structure Port which contains (1) a specific Port number, (2) the process id (pid), and (3) the processor number, forming a unique port address. At module level, these port addresses (connection points of a channel to a process) are represented by the type ChanName’. Objects of this type contain exactly the port identifier triple (see Fig. 4). Note that the higher level type ChanName a is a list of port identifiers; one for each component of type a in case a is a tuple. The function createDC :: Trans a => a -> (ChanName a, a) creates a new (dynamic) input channel, i.e. a channel on which data can be received, using the corresponding primitive operation createDC#. It yields the channel name which can be sent to other processes and (a handle to) the input that will be received via the channel. If createDC is used for tuple types a, a list of port identifiers (type ChanName’) and a tuple of input handles will be created. To ensure correct typing, createDC is always applied to its second output, but will only use it to determine the needed number of channels, using the overloaded function tupsize in class Trans. Data transmission is done by the function writeDC. This function takes a port identifier and a value, connects the current thread to the given port (setChan#) and sends the value using the function sendChan. The connection by setChan# prior to evaluating and sending values guarantees that the receiver of data mes-
High-Level Process Control in Eden
737
type ChanName a = [ChanName’ a] data ChanName’ a = Chan Int# Int# Int# createDC :: a -> (ChanName a, a) createDC t = let (I# i#) = tupSize t in case createDC# i# of (# c,x #) -> (c,x) writeDC :: ChanName’ a -> b -> () writeDC chan a = setChan chan ‘seq‘ sendChan a Fig. 4. Channel Creation and Communication
sages is always defined when sendVal# or sendHead# is called. While writeDC defines the behaviour of a single thread, the overloaded function writeDCs (see Fig. 3) handles tuples in a special way. It takes a list of port identifiers (length identical to the number of tuple components) and creates a thread for each tuple component. The instance declaration of Trans for pairs is e.g. as follows: instance (Trans a, Trans b) => Trans (a,b) where tupsize = 2 writeDCs (cx:cy: ) (x,y) = writeDC cx x ‘fork‘ writeDC cy y
The Eden module contains corresponding instance declarations for tuples with up to eight components.
3.2
Process Handling
Subsequently, we will focus on the module definitions for process abstraction and instantiation shown in Fig. 5 and 6. Process creation can be defined on this level, using the internal functions to create channel names and to send data on them, plus the primitive operation createProcess# for forking a process on a remote processor. A process abstraction of type Process a b is implemented by a function f remote (see Fig. 5) which will be evaluated remotely by a corresponding child process. It takes two channel names: the first outDCs (of type ChanName b) is a channel for sending its output while the second chanDC (of type ChanName (ChanName a)) is an administrative channel to return the names of input channels to the parent process. The exact number of channels which are established between parent and child process does not matter in this context, because the operations on dynamic channels are overloaded. The definition of process shows that the remotely evaluated function, f remote, creates its input channels via the function createDC. Moreover, writeDCs is used twice: the dynamically created input channels of the child, inDCs, are sent to the parent process via the channel
738
J. Berthold et al.
data (Trans a, Trans b) => Process a b = Proc (ChanName b -> ChanName (ChanName a) -> ()) process :: (Trans a, Trans b) => (a -> b) -> Process a b process f = Proc f_remote where f_remote outDCs chanDC = let (inDCs, invals) = createDC invals in writeDCs chanDC inDCs ‘fork‘ (writeDCs outDCs (f invals)) Fig. 5. Haskell definitions of Eden process abstraction
( # ) :: (Trans a, Trans b) => Process a b -> a -> b pabs # inps = case createProcess (-1#) pabs inps of Lift x -> x data Lift a = Lift a createProcess :: (Trans a, Trans b) => Int# -> Process a b -> a -> Lift b createProcess on# (Proc f_remote) inps = let (outDCs, outvals) = createDC outvals (chanDC, inDCs ) = createDC inDCs pinst = f_remote outDCs chanDC in outDCs ‘seq‘ chanDC ‘seq‘ case createProcess# on# pinst of 1# -> writeDCs inDCs inps ‘fork‘ (Lift outvals) _ -> error "process creation failed" Fig. 6. Haskell definitions of Eden process instantiation
chanDC and the results of the process determined by evaluating the expression (f invals) are sent via the channels outDCs5 .
Process instantiation by the operator ( # ) defines process creation on the parent side. To cope with lazy evaluation and to get back control without waiting for the result of the child process, the process results are lifted to an immediately available weak head normal form using the constructor Lift. Before returning the result, the Lift is removed. The function createProcess takes the process abstraction and the input expression and yields the lifted process result. The placement parameter on# is a primitive integer (type Int#) which can be used to allocate newly created processes explicitly. The current system does not make use of this possibility, processes are allocated round-robin or randomly on the available processors. The channels are handled using createDC and writeDCs in the 5
The prefixes in and out in channel names in Fig. 5 and 6 reflect the point of view of a child process. Thus, in a process instantiation, the inputs inps for the child are written into the channels inDCs, which are outputs of the parent process.
High-Level Process Control in Eden
739
Fig. 7. Sequence Diagram for Process Instantiation
same way as on the child side (see the process abstraction). The remote creation of the child process is performed by the primitive operation createProcess#. The whole sequence of actions is shown in Fig. 7, which illustrates the interplay of the codes in Fig. 5 and 6. Process creation is preceded by the creation of new channels (one for every result) plus one additional port to receive channels for input parameters upon creation. The primitive createProcess# sends a message createProcessDC to a remote processor, which contains these channels and the process abstraction (an unevaluated Proc f remote packed as a subgraph). The remote processor (PE 2 in Fig. 7) receives this message, unpacks the graph and starts a new process by creating an initial thread. As the first thread in a process, this thread plays the role of a process driver. Evaluating the code shown in Fig. 5, it forks a first thread to communicate channels and then evaluates the results, forking one thread for each tuple component but the last, which is evaluated and sent back by the initial thread itself. Thus, one thread is only used to communicate channels, the other threads evaluate the output expressions. Threads block on the created input handles if they need their arguments. As soon as the input arrives, these threads are reactivated by the communication handler when it writes the received values into the heap. This concludes our discussion of fundamental mechanisms in the Eden system.
740
4
J. Berthold et al.
Related Work and Conclusions
Implementations of parallel functional languages are either based on a parallel abstract machine or on parallel libraries, linked to an ordinary sequential system. Eden is a typical example for the monolithic approach (parallel abstract machine). It is closely related to GpH [19], using the same framework and even sharing a lot of code. Although GpH, combined with evaluation strategies [18] provides comparable control, it generally follows the concept of implicit, or “potential” parallelism, whereas Eden passes parallelism control to the programmer. During the last decade, the extension of functional languages for parallelism moved from implicit to more and more explicit approaches, because it became clear that an effective parallelisation needs some support from the programmer. The para-functional programming approach [11], as well as Caliban [17], provide explicit control over subexpression evaluation and process placement, going essentially further than our process concept. Providing a set of parallel skeletons [16] is another way of implementing (more or less explicit) parallelisation facilities. Parallel skeletal languages, as e.g. P3 L [1], provide a fixed set of skeletons, sometimes combined with sophisticated program analysis, as e.g. in PMLS [10]. The skeleton implementation is usually out of reach for the programmer, whereas in Eden, programming a skeleton requires nothing but ordinary parallel functional programming. By lifting the implementation of explicit low-level process control out of the RTS into the functional language itself, we reach two goals: As new techniques like Grid-Computing evolve, it becomes more and more important for a parallel programming system to provide not only good performance and speed-up behaviour, but also to make parallel programming as simple as possible. We achieve this not only on the highest level, i.e. for the ordinary application programmer, but also for the advanced programmer interested in the development of skeletons or even parallel extensions. An advantage of our layered implementation is that developers can use the high-level layers of module and skeleton library which we have introduced. With rising demand for efficient compilers and better exploitation of parallel computers, compiler construction is getting much more complex. By the layer concept, we gain advantages on portability, code reuse, extensibility, maintenance, and abstraction on the implementation side. The implementation based on a few primitive operations leads to clean interfaces between the implementation layers, which makes it easier to follow version changes of the underlying sequential compiler. This is the first step to our long-term objective: the design of a generic parallel extension of sequential functional runtime systems, on top of which various parallel functional languages could be implemented. Acknowledgements. We thank Hans-Wolfgang Loidl, Simon Marlow and Fernando Rubio for their support during the development of the Eden runtime system. Additionally, we are grateful to Phil Trinder and the anonymous referees for helpful comments on this paper.
High-Level Process Control in Eden
741
References 1. B. Bacci, M. Danelutto, S. Orlando, S. Pelagatti, and M. Vanneschi. P3 L: A Structured High Level Programming Language and its Structured Support. Concurrency — Practice and Experience, 7(3):225–255, May 1995. 2. S. Breitinger, U. Klusik, and R. Loogen. From (Sequential) Haskell to (Parallel) Eden: An Implementation Point of View. In PLILP’98, LNCS 1490, pages 318–334. Springer, 1998. 3. S. Breitinger, R. Loogen, Y. Ortega-Mall´en, and R. Pe˜ na. Eden: Language Definition and Operational Semantics. Technical report, 1996. Available at http://www.mathematik.uni-marburg.de/inf/eden. 4. Glasgow Parallel Haskell. WWW page. http://www.cee.hw.ac.uk/˜dsg/gph/. 5. K. Hammond, J. Berthold, and R. Loogen. Automatic Skeletons in Template Haskell. In HLPP 2003. Paris, France, June 2003. 6. K. Hammond and G. Michaelson, editors. Research Directions in Parallel Functional Programming. Springer-Verlag, 1999. 7. U. Klusik, Y. Ortega-Mall´en, and R. Pe˜ na Mar´ı. Implementing Eden – or: Dreams Become Reality. In IFL’98, LNCS 1595, pages 103–119. Springer, 1999. 8. H.-W. Loidl, F. Rubio Diez, N. Scaife, K. Hammond, U. Klusik, R. Loogen, G. Michaelson, S. Horiguchi, R. Pena Mari, S. Priebe, A. R. Portillo, and P. Trinder. Comparing parallel functional languages: Programming and performance. Higherorder and Symbolic Computation, 16(3), 2003. 9. R. Loogen, Y. Ortega-Mall´en, R. Pe˜ na, S. Priebe, and F. Rubio. Parallelism Abstractions in Eden. In F. A. Rabhi and S. Gorlatch, editors, Patterns and Skeletons for Parallel and Distributed Computing. Springer, 2002. 10. G. Michaelson, N. Scaife, P. Bristow, and P. King. Nested Algorithmic Skeletons from Higher Order Functions. Parallel Algorithms and Appl., 16:181–206, 2001. 11. R. Mirani and P. Hudak. First-Class Schedules and Virtual Maps. In FPCA’95, pages 78–85. ACM Press, June 1995. 12. S. Peyton Jones. Implementing lazy functional languages on stock hardware: The spineless tagless g-machine. J. of Functional Programming, 2(2):127–202, 1992. 13. S. Peyton Jones, C. Hall, K. Hammond, W. Partain, and P. Wadler. The Glasgow Haskell Compiler: a Technical Overview. In JFIT’93, pages 249–257, March 1993. http://www.dcs.gla.ac.uk/fp/papers/grasp-jfit.ps.Z. 14. S. Peyton Jones and J. Hughes. Haskell 98: A Non-strict, Purely Functional Language, 1999. Available at http://www.haskell.org/. 15. PVM: Parallel Virtual Machine. WWW page. http://www.epm.ornl.gov/pvm/. 16. F. A. Rabhi and S. Gorlatch, editors. Patterns and Skeletons for Parallel and Distributed Computing. Springer, 2002. 17. F. Taylor. Parallel Functional Programming by Partitioning. PhD thesis, Department of Computing, Imperial College, London, 1997. http://www.lieder.demon.co.uk/thesis/thesis.ps.gz. 18. P. Trinder, K. Hammond, H.-W. Loidl, and S. Peyton Jones. Algorithm + Strategy = Parallelism. J. of Functional Programming, 8(1):23–60, 1998. 19. P. Trinder, K. Hammond, J. Mattson Jr., A. Partridge, and S. Peyton Jones. GUM: a Portable Parallel Implementation of Haskell. In PLDI’96, pages 78–88. ACM Press, May 1996.
Using Skeletons in a Java-Based Grid System Martin Alt and Sergei Gorlatch Technische Universit¨ at Berlin, Germany {mnalt|gorlatch}@cs.tu-berlin.de Abstract. Grid systems connect high-performance servers via the Internet and make them available to application programmers. This paper addresses the challenge of software development for Grids, by means of reusable algorithmic patterns called skeletons. Skeletons are generic program components, which are customizable for a particular application and can be executed remotely on high-performance Grid servers. We present an exemplary repository of skeletons and show how a particular application, FFT (Fast Fourier Transform), can be expressed using skeletons and then executed using RMI (Remote Method Invocation). We describe a prototypical Java-based Grid system, present its optimized RMI mechanism, and report experimental results for the FFT example.
1
Introduction
Grid systems connect high-performance computational servers via the Internet and make them available to application programmers. While the enabling infrastructures for Grid computing are fairly well developed [1], initial experience has shown that entirely new approaches are required for Grid programming [2]. A particular challenge is the phase of algorithm design: since the type and configuration of the servers on which the program will be executed is not known in advance, it is difficult to make the right design decisions, to perform program optimizations and estimate their impact on performance. We propose to address Grid programming by providing the application programmers with two kinds of software components on the server side: (1) traditional library functions, and (2) reusable, high-level patterns, called skeletons. Skeletons are generic algorithmic components, customizable for particular applications by means of their functional parameters. Time-intensive skeleton calls are executed remotely on high-performance Grid servers, where architecture-tuned, efficient parallel implementations of the skeletons are provided. The contributions and organization of the paper are as follows: We present the structure of our Grid system based on Java and RMI and explain the advantages of skeletons on the Grid (Sect. 2). We introduce a repository of skeletons for expressing parallel and distributed aspects of Grid applications (Sect. 2.1), discuss the inefficiency of standard Java RMI on the Grid and propose using future-based RMI (Sect. 2.2). We show how a mathematical specification of FFT is expressed using our skeletons, develop a skeleton-based Java program and report experimental performance results for it (Sect. 3). We conclude by discussing our results in the context of related work. H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 742–749, 2003. c Springer-Verlag Berlin Heidelberg 2003
Using Skeletons in a Java-Based Grid System
2
743
Our Grid Prototype
Our prototypical Grid environment (Fig. 1) consists of two university LANs – one at the Technical University of Berlin and the other at the University of Erlangen. They are connected by the German academic internet backbone (WiN), covering a distance of approx. 500 km. There are three high-performance servers in our Grid: a shared-memory multiprocessor SunFire 6800, a MIMD supercomputer of type Cray T3E, and a Linux cluster with SCI network. Application programmers work from clients (PCs and workstations). A central entity called “lookup service” is used for resource discovery. The reader is referred to [3] for details of the system architecture and the issues of resource discovery and management. Internet
Clients
a=skeleton1(); b=skeleton2(); c=localMethod(b); d=skeleton3(a,c); ...
1
Servers 111 000 01 1 0 000 111 0 1 0 skeleton1 01 1 0 1 0000 1111 0 1 0 1 0000 1111 0 1 0 01 1 0 1 0 1 0 0000 1111 01 1 0 1 0000 0 1 0 2 1111 0000 1111 01 1 0 1 0000 1111 0 1 0 0000 1111 01 1 0 1
SunFire 6800
skeleton2 WAN Shared Network Links
skeleton3 1 0 1 0
Linux Cluster
Lookup Service
Cray T3E
Fig. 1. System architecture and interaction of its parts
We propose developing application programs for such Grid systems using a set of reusable, generic components, called skeletons. As shown in the figure, a program on a client is expressed as a sequential composition of skeleton calls and local calls. The servers in the Grid provide architecture-tuned implementations of the skeletons: multithreaded, MPI-based, etc. Applications composed of skeletons can thus be assigned for execution to particular servers in the Grid with a view to achieving better performance. Time-intensive skeleton calls are executed remotely on servers which provide 1 in the figure). If two implementations for the corresponding skeleton (arrow subsequent skeleton calls are executed on different servers, then the result of the 2 ). first call must be communicated as one of inputs for the second call (arrow This situation is called composition of skeletons. Using skeletons for programming on the Grid has the following advantages: – Skeletons’ implementations on the server side are usually highly efficient because they can be carefully tuned to the particular server architecture. – The once-developed, provably correct implementation of a skeleton on a particular server can be reused by different applications. – Skeletons hide the details about the executing hardware and the server’s communication topology from the application programmer. – Skeletons provide a reliable model for performance prediction, providing a sound information base for selecting servers.
744
2.1
M. Alt and S. Gorlatch
A Repository of Skeletons
In the following, we describe a (by no means exhaustive) collection of skeletons. Since at least some of the skeletons’ parameters are functions, skeletons can be formally viewed as higher-order functions. In practice, functional parameters are provided as program codes, in our system as Java bytecodes. We begin our presentation with simple skeletons that express data parallelism: Map: Apply a unary function f to all elements of a list: map(f, [x1 , . . . , xn ]) = [f (x1 ), . . . , f (xn )] Scan: Compute prefix sums of a list by traversing the list from left to right and applying a binary associative operator ⊕: scan(⊕, [x1 , . . . , xn ]) = [ x1 , (x1 ⊕ x2 ), . . . , (· · ·(x1 ⊕ x2 )⊕ x3 )⊕· · ·⊕ xn ) ] A more complex data-parallel skeleton, DH (Distributable Homomorphism) [4], expresses a divide-and-conquer pattern with parameter operators ⊕ and ⊗: DH: Formally, dh(⊕, ⊗, x) transforms a list x = [x1 , . . . , xn ] of size 2l into a result list y = [y1 , . . . , yn ], whose elements are computed as follows: if i ≤ n/2 ui ⊕ vi , yi = (1) ui−n/2 ⊗ vi−n/2 , otherwise where u = dh(⊕, ⊗, [x1 , . . . , xn/2 ]) and v = dh(⊕, ⊗, [xn/2+1 , . . . , xn ]). In addition to these data-parallel skeletons, we provide two auxiliary skeletons, whose aim is efficient communication between client and server: Replicate: Create a new list containing n times element x: repl (x, n) = [x, . . . , x]. The repl skeleton can be called remotely on a server to create there a list of n identical elements, instead of sending the whole list over the network. Apply: Applies a unary function f to a parameter x: apply(f, x) = f (x). The apply skeleton is used to remotely execute a function f by shipping its code to the server, rather than moving the data to the client, executing the function locally and then sending the result to the server again. Our skeleton-based Grid programmimg environment for the system shown in Fig. 1 is built on top of Java and RMI. We chose the Java platform mostly for reasons of portability (see [5] for “10 reasons to use Java in Grid computing”). In the system, skeletons are offered as Java (remote) interfaces, which can be implemented in different ways on different servers. To be as flexible as possible, all skeletons operate on Objects or arrays of Object. For example, the interface for the scan skeleton contains a single method public Object [] invoke ( Object [], BinOp oplus );
To use the scan skeleton, the client first finds a server for execution, using the lookup service (see [3] for details). After obtaining an RMI reference to the scan implementation on the server, the skeleton is executed via RMI by calling the invoke method with appropriate parameters.
Using Skeletons in a Java-Based Grid System
2.2
745
Future-Based RMI for the Grid
Using the RMI mechanism in Grid programs has the important advantage that the outsourcing of skeleton calls to remote servers is transparent for the programmer: remote calls are coded in exactly the same way as local calls. However, since the RMI mechanism was developed for client-server systems, it is not optimal for the Grid. We illustrate this using the following example: a composition of two skeleton calls, with the result of the first call being used as an argument of the second call (skeleton1 and skeleton2 are remote references): result1 = skeleton1 . invoke (...); result2 = skeleton2 . invoke ( result1 ,...);
Executing such a composition of methods using standard RMI, the result of a remote method invocation is always sent back directly to the client. This is exemplified for the above example in Fig. 2 (left). When skeleton1 is invoked 1 ), the result is sent back to the client ( 2 ), then to skeleton2 ( 3 ). Finally, ( 4 ). For typical applications consisting of the result is sent back to the client ( many composed skeletons, this feature of RMI results in very high time overhead. Server1 1
Server1 1
2
Client
2 5
Client 3 4
3
4
6
Server2
Server2
Fig. 2. Skeleton composition using plain RMI (left) and future-based RMI (right)
To eliminate this overhead, we have developed so-called future-based RMI : an invocation of a skeleton on a server initiates the skeleton’s execution and then returns immediately, without waiting for the skeleton’s completion (see Fig. 2, right). As a result of the skeleton call, a future reference is returned to the client 2 ) and can be used as a parameter for invoking the next skeleton ( 3 ). When ( 4 the future reference is dereferenced (), the dereferencing thread on the server is blocked until the result is available, i. e. the first skeleton actually completes. The result is then sent directly to the server dereferencing the future reference 5 ). After completion of skeleton2, the result is sent to the client ( 6 ). ( Compared with plain RMI, our future-based mechanism substantially reduces the amount of data sent over the network, because only a reference to the data is sent to the client; the result itself is communicated directly between the servers. Moreover, communications and computations overlap, effectively hiding latencies of remote calls. We have implemented future-based RMI on top of SUN Java RMI and report experimental results in Sect. 3.3. (see [6] for further details). Future references are available to the user through a special Java interface RemoteReference. There are only few differences when using future-based RMI compared with the use of plain RMI: (1) instead of Objects, all skeletons return values of type RemoteReference, and (2) skeletons’ interfaces are extended by invoke methods, accepting RemoteReferences as parameters.
746
3
M. Alt and S. Gorlatch
Case Study: Programming FFT Using Skeletons
By way of an example application, we consider the Fast Fourier Transformation (FFT). The FFT of a list x = [x0 , . . . , xn−1 ] of length n = 2l yields a list whose n−1 ki i-th element is defined as (FFT x)i √= k=0 xk ωn , where ωn denotes the n-th 2π −1/n complex root of unity, i. e. ωn = e . 3.1
Expressing FFT with Skeletons
We now outline how the FFT can be expressed as a composition of skeletons (see [4] for details). The FFT can be written in divide-and-conquer form as follows, where u = [x0 , x2 , . . . , xn−2 ] and v = [x1 , x3 , . . . , xn−1 ]: ˆ i,n (FFTv)i if i < n/2 (FFTu)i ⊕ (FFTx)i = (2) ˆ i−n/2,n (FFTv)i−n/2 else (FFTu)i−n/2 ⊗ j j ˆ j,m b = a − ωm ˆ j,m b = a + ωm b , and a ⊗ b. where a ⊕ The formulation (2) is close to the dh skeleton format from Sect. 2.1, except ˆ and ⊗ ˆ being parameterized with the position i of the list element and the for ⊕ length n of the input list. Therefore we express the FFT as instance of the dh skeleton, applied to a list of triples (xi , i, n), with operator ⊕ defined on triples as ˆ i1 ,n1 x2 , i1 , 2n1 ). Operator ⊗ is defined similarly. (x1 , i1 , n1 ) ⊕ (x2 , i2 , n2 ) = (x1 ⊕
Computing FFT using skeletons: As skeletons are higher-order functions, we first provide a functional program for FFT, which is then transformed to Java in a straightforward manner. The FFT function on an input list x can be expressed using skeletons by transforming the input list into a list of triples, applying the dh skeleton and finally taking the first elements of the triples for the result list: FFT = map(π1 ) ◦ dh (⊕, ⊗) ◦ apply(triple) where triple is a user-defined function that transforms a list [x1 , . . . , xn ] to the list of triples [(x1 , 1, 1), . . . , (xi , i, 1), . . . , (xn , n, 1)], and ◦ denotes function composition from right to left, i. e. (f ◦ g) (x) = f (g(x)). ˆ and ⊗ ˆ in (2) repeatedly compute the roots of unity ωni . Both operators ⊕ Instead of computing these for every call, they can be computed once a priori and n/2 stored in a list Ω = [ωn1 , . . . , ωn ], accessible by both operators, thus reducing n/m i ˆ ⊗ ˆ in ⊕/ computations. Using the relation ωm = ωn , the computation of ωm can be replaced with π(ni/m, Ω), where π(k, Ω) selects the k-th entry of Ω. ˆ ˆ can be expressed as a ⊕ ˆ j,m,Ω b = a + π(nj/m, Ω)b . Operator ⊗ Therefore, ⊕ can be expressed using Ω analogously. Thus, ⊕/⊗ are parameterized with Ω; we express this by writing ⊕(Ω)/ ⊗ (Ω) in the following. Now, we can express the computation of Ω using the repl and the scan skeletons, and arrive at the following skeleton-based program for the FFT: Ω = scan(∗) ◦ repl (n/2, ωn ) FFT = map(π1 ) ◦ dh (⊕(Ω), ⊗(Ω)) ◦ apply(triple) where ∗ denotes complex number multiplication.
(3)
Using Skeletons in a Java-Based Grid System
3.2
747
Skeleton-Based Java Program for FFT
The Java code for the FFT, obtained straightforwardly from (3), is as follows: // repl , scan , map , dh are remote refs to skeletons on servers // compute roots of unity RemoteReference r = repl . invoke ( length , omega_n ); RemoteReference omegas = scan . invoke ( r , new ScanOp ()); // instantiate operators for dh oplus = new FFTOplus ( omegas ); otimes = new FFTOtimes ( omegas ); // fft r = apply . invoke ( inputList , new TripleOp ()); r = dh . invoke ( oplus , otimes , r ); r = map . invoke ( r , new projTriple ()); // get result result = r . getValue ();
At first, the roots of unity are computed, using the repl and scan skeletons. Both repl and scan are RMI references to the skeletons’ implementation on a remote server, obtained from the lookup service. Execution of the skeletons is started using the invoke methods. Variable OmegaN passed to the repl skeleton corresponds to ωn and omegas corresponds to Ω. As a binary operator for scan, complex multiplication is used, implemented in class ComplexMult. The operators ⊕(Ω) and ⊗(Ω) for the dh-skeleton are instantiated as objects of classes FFTOplus and FFTOtimes on the client side. The constructor for the parameters receives the list Ω as an argument. Each operator stores a reference to the list in a private variable in order to access it later for computations. Next, the FFT itself is computed in three steps. First the input list is transformed to a list of triples, using the apply skeleton with a user-defined function. Then the dh-skeleton is called on the list of triples, using the two customizing operators defined earlier. Finally, the list of result values is retrieved from the list of triples using the map skeleton with an instance of the user-defined class projTriple. In a preliminary step (omitted in the code presented above), the program obtains from the lookup service a remote reference for each skeleton used in the program (repl, scan, map and dh). The program is executed on the client side; all calls to the invoke method of the involved skeletons are executed remotely on servers via RMI. 3.3
Experimental Results
We measured the performance of the skeleton-based FFT program using the testbed of Fig. 1. We used a SunFire 6800 with 12 US-III+ 900 MHz processors in Berlin as our server and an UltraSPARC-IIi 360 MHz as client in Erlangen, both using SUN’s JDK1.4.1 (HotSpot Client VM in mixed mode). Because there were several other applications running on the server machine during our experiments, a maximum of 8 processors was available for measurements.
748
M. Alt and S. Gorlatch
Fig. 3 shows the runtimes for different problem sizes (ranging from 215 to 218 ) and four different versions of the program: the first running locally on the client (“local FFT”), second using plain RMI, third version using future-based RMI, and the fourth version where the FFT is executed as a single server sided method called from the client (“ideal remote”). We consider the fourth version as ideal, as there is no overhead for remote composition of skeletons for that version: it corresponds to copying the whole program to the server and executing it there. For the plain RMI version, only the scan and dh skeletons are executed on the server, because all parameters and results are transmitted between client and server for each method call using plain RMI, so that executing the repl , apply and map skeleton remotely would slow down the program unnecessarily. For the future-based RMI version, all skeletons are executed on the server. 16000
local FFT plain RMI ideal remote future-based RMI
14000
Time [ms]
12000 10000 8000 6000 4000 2000 0 15
16
17
18
List Size (log)
Fig. 3. Measured runtimes for the FFT programs
The figure shows ten measurements for each program version, with the average runtimes for each parameter size connected by lines. The plain RMI version is much (three to four times) slower than the future-based RMI version and unable to outperform the local, client sided FFT. Thus, the communication overhead outweighs the performance gain for execution on the server. By contrast, the future-based RMI version eliminates most of the overhead and is three to four times faster than the local version. Compared with the “ideal remote” case the runtimes are almost identical. For large input lists (217 and 218 ), the future-based version is even slightly faster than the remote version. This is due to the fact, that the future-based version invokes skeletons asynchronously, so the apply skeleton is already called while the scan skeleton is still running. Thus, using future-based RMI allows an efficient execution of programs with compositions of remote methods, in particular compositions of skeletons.
4
Conclusions and Related Work
In this paper, we have addressed the challenging problem of software design for heterogeneous Grids, using a repository of reusable algorithmic patterns called
Using Skeletons in a Java-Based Grid System
749
skeletons, that are executed remotely on high-performance Grid servers. While the use of skeletons in the parallel setting is an active research area, their application for the Grid is a new, intriguing problem. We have described our prototypical Grid system. Java and RMI were chosen to implement our system in order to obtain a highly portable solution. Other promising opportunities include, e. g. the Lithium system [7] for executing mainly task-parallel skeletons in Java. We have proposed a novel, future-based RMI mechanism, which substantially reduces communication overhead for compositions of skeleton calls. It differs from comparable approaches because it combines hiding network latencies using asynchronous methods (as in [8,9]) and reducing network dataflow by allowing server/server communication (e. g. found in [10]). We have proposed an exemplary (and by no means exhaustive) repository of skeletons, which includes several elementary data-parallel functions, the divideand-conquer skeleton DH, and two auxiliary skeletons which are helpful in a Grid environment. We have demonstrated how a mathematical description of FFT (Fast Fourier Transform) can be expressed using our skeletons, leading to an efficient Java program with remote calls for skeletons. At present, each skeleton call is executed on a single Grid node. We plan to allow distribution of skeletons across several nodes in the future, at least for task-parallel and simple data-parallel skeletons.
References 1. Foster, I., Kesselmann, C., eds.: The Grid: Blueprint for a New Computing Infrastructure. Morgan Kaufmann (1998) 2. Kennedy, K., et al.: Toward a framework for preparing and executing adaptive grid programs. In: Proc. of NSF Next Generation Systems Program Workshop (2002) 3. Alt, M., et al.: Algorithm design and performance prediction in a Java-based Grid system with skeletons. In: Proc. of Euro-Par 2002. LNCS Vol. 2400, Springer (2002) 4. Gorlatch, S., Bischof, H.: A generic MPI implementation for a data-parallel skeleton: Formal derivation and application to FFT. Parallel Processing Letters 8 (1998) 5. Getov, V., et al.: Multiparadigm communications in Java for Grid computing. Communications of the ACM 44 (2001) 118–125 6. Alt, M., Gorlatch, S.: Optimizing the use of Java RMI for Grid application programming. Technical Report 2003/08, TU Berlin (2003) ISSN 1436-9915. 7. Danelutto, M., Teti, P.: Lithium: A structured parallel programming enviroment in Java. In: Proc. of Computational Science. ICCS. LNCS Vol. 2330, Springer (2002) 8. Raje, R., Williams, J., Boyles, M.: An asynchronous remote method invocation (ARMI) mechanism in Java. Concurrency: Practice and Experience 9 (1997) 9. Falkner, K.K., Coddington, P., Oudshoorn, M.: Implementing asynchronous remote method invocation in Java. In: Proc. of Parallel and Real Time Systems (1999) 10. Yeung, K.C., Kelly, P.H.J.: Optimising Java RMI programs by communication restructuring. In: Middleware 2003, Springer (2003)
Prototyping Application Models in Concurrent ML David Johnston, Martin Fleury, and Andy Downton Dept. Electronic Systems Engineering, University of Essex, Wivenhoe Park, Colchester, Essex, CO4 3SQ, UK {djjohn, fleum, acd}@essex.ac.uk http://www.essex.ac.uk/ese/research/mma lab/index.htm
Abstract. We report on the design of parallel applications using harnesses, which are abstractions of concurrent paradigms. A split/combine harness is prototyped using the functional language Concurrent ML (CML). The event model simplifies harness development by unifying the structure of serial and parallel algorithms. The portability of the harness between different hardware (both serial and parallel) is planned through the compact and elegant CML programming model.
1
Introduction
Concurrent Meta-Language (CML) is a version of the well-established functional language ML with extensions for concurrency by Reppy [16]. Concurrent processes cooperate through CSP [10] channels i.e. unbuffered communication. The instantiation of CSP formalism within CML is more powerful than that of the classical version of the parallel language occam2 [6], specifically: 1. dynamic recursion, process and channel creation (c.f. no recursion, static processes and static channels in occam2) 2. selection between channels based on input and output events (c.f. just input selection in occam2 for implementation reasons) 3. functions can be sent over channels (side effect of ML treating code and data the same) 4. events and event functions (c.f. only synchronised communication in occam2) Modern occam [1] has absorbed some of these capabilities e.g. recursion and the ability to send processes over channels. It is worth also comparing CML with the occam2-like Handel-C [17], a silicon compiler language targeted at FPGAs, where 2 and 3 are also present. Modern occam and CML are general purpose in nature rather than static in the tradition of embedded languages. Parallel designs in both languages could map onto the fine-grained parallelism of Handel-C or onto systems composed of distributed processor nodes. This paper demonstrates, through example, how the features of CML allow the rapid prototyping of software infrastructures (or harnesses) that support H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 750–759, 2003. c Springer-Verlag Berlin Heidelberg 2003
Prototyping Application Models in Concurrent ML
751
generic application-level parallelism. Such harnesses have been available within occam2 for some time, but following through the consequences of the CML event model and CML’s functional nature results in a different and somewhat surprising outcome. Section 2 presents the context of the work within a larger development method. CML is introduced in Section 3 though some small illustrative examples. The main technical content within Section 4 builds upon these earlier examples to show how support for parallel execution is prototyped within CML. Section 5 presents related work, and the paper ends with conclusions and a description of the current status of the work.
2
Development Method
The key idea behind our project1 is that the same paradigms of parallelism occur in many different applications. The execution of a single application is characterised by a dynamically changing set of such paradigms. Accordingly, a number of customised harnesses for each paradigm detected dynamically within the executing application can be transparently launched. To illustrate, consider the simple split/combine application model for parallelism where the input can be split recursively; the parts processed in parallel; and the outputs combined. Figure 1 shows how this and other generic structures can be implemented as a harness containing the parallelism (HCM L or HC++ ) in conjunction with user’s application-specific code (AM L or AC++ respectively) which does not. The Figure presents a variety of software engineering routes because there is a choice of working in CML or a more conventional language such as C++. The use of CML is not an end in itself (though there is useful software output as a side effect) but a means to rapidly prototype a C++ version of the system, which a C++ developer can use off-the-shelf. The splitting of an application into a harness and serial application code is shown both for CML and C++. However, the solid arrows represent which activities are intended exclusively for systems developers i.e. they are capturing the real parallelisations and then abstracting them within general harnesses so users need not be concerned with the parallelisation of specific applications. If the user is prototyping in ML, then the translation to C++ is possible either as sequential code or after fitting into a CML harness.
3
CML Examples
3.1
Example 1: Communication-Based Remote Procedure Call
To illustrate the use of CML, the following code performs a remote procedure call (RPC). The server applies a function (f) to the input argument (arg) and sends the result down a channel (ch) to the RPC client. For its part, the client 1
Rapid Parallel Prototyping in the Image/Multimedia Domain, EPSRC Contract: GR/N20980
752
D. Johnston, M. Fleury, and A. Downton
A
pseudocode
implementation
A
implementation ML
translation parallelisation parallelisation
A’
A
CML
abstraction
C++
parallelisation
abstraction
H
A’’
CML
ML
translation
system transformation
H
A’
C++
C++
translation
A
L
H
L
application in language L
user transformation third party transformation
harness in language L
Fig. 1. Overall development methodology
creates the channel; spawns the server process with its arguments and receives a result from the channel. The client is placed within a main process so it can be run in the CML environment using RunCML.doit. fun rpc_server ( ch, f, arg ) = send( ch, f(arg) ); fun rpc_client f arg = let val ch = channel(); val tid = spawnc rpc_server ( ch, f, arg ); in recv( ch ) end; fun main () = print ( Real.toString( rpc_client Math.sqrt 2.0 ) ˆ "\n" ); RunCML.doit ( main, NONE );
3.2
Example 2: Event-Based Remote Procedure Call
The second example performs the same actions as the first, only it is written in terms of the primitive event functions of CML: sendEvt and recvEvt instead of the derived send and recv. The communication only happens after the event returned by sendEvt and the corresponding event returned by recvEvt are both explicitly synchronised using the sync operator. In the server, the send event is directly synchronised; whereas the client actually returns a receive event which is synchronised at a higher level in the code. These useful client and server routines will be used later in Sections 4.3, 4.4 and 4.5.
Prototyping Application Models in Concurrent ML
753
fun rpc_server( ch, f, arg ) = sync( sendEvt( ch, f(arg) ) ); fun rpc_client f arg = let val ch = channel(); val tid = spawnc rpc_server ( ch, f, arg ); in recvEvt( ch ) end; fun main () = print ( Real.toString( sync ( rpc_client Math.sqrt 2.0 ) ) ˆ "\n" ); RunCML.doit ( main, NONE );
3.3
Example 3
A guard event operator implements a delay in terms of the absolute temporal event primitive atTimeEvt. The guard operator performs pre-synchronisation actions or may be informally considered to “push” an action back [12]. The absolute time for the output event is only evaluated when the sync is called and not when timeout is called. The result is that synchronising on event e always causes a delay of exactly one second. fun timeout t = guard ( fn () => atTimeEvt (Time.+ (t, Time.now())) ); fun main () = let val e = timeout (Time.fromSeconds 1); in sync e end; RunCML.doit ( main, NONE );
4
Split/Combine Application Harness
CML allows the rapid prototyping of a parallel harness which implements the split/combine model described in Section 2. This is shown below in five stages starting with a serial harness. 4.1
Serial Hierarchical Harness – SH harness
The harness calls itself twice recursively if the problem is still to be split. Otherwise the input data is processed normally. Recursion is limited by simple recursion depth, though in a practical implementation this would be dynamic recursion through monitoring of performance as a function of granularity. Note the combine function is an output of the split function, so it can be a customised inverse operation.
754
D. Johnston, M. Fleury, and A. Downton
(* * argument * -------* depth * max_depth * f * split * * input *)
description ----------current recursion depth maximum recursion depth allowed function that does processing of input to output function that splits input AND provides matching function which combines the corresponding outputs input data to application
fun SH_harness ( depth, max_depth ) ( f, split ) input = if ( depth < max_depth ) then let val ( combine, input1, input2 ) = split input; val daughter = SH_harness ( depth+1, max_depth ) ( f, split ); val output1 = daughter input1; val output2 = daughter input2; in combine output1 output2 end else f input;
h s
harness
split
in1 in
h11 f in11 s h1 in12 h12 f
out11 c1 out12
in2
h21 f in21 out21 s h2 c2 in22 out22 h22 f
process
c
combine
out1
h
s
f
c
out
out2 data flow channel spawn
Fig. 2. Communicating parallel hierarchical split/combine harness
4.2
Communicating Parallel Hierarchical Harness – CPH harness
An extra channel argument is supplied for the harness to send its result rather than returning it. Parallel execution is obtained by spawning two further versions of the harness instead of calling them sequentially in turn. To achieve parallelism
Prototyping Application Models in Concurrent ML
755
the results must be received after both processes have been spawned. Note that this code will have to na¨ively wait for the first harness spawned to finish even though the second could have finished earlier. Corresponding sends and receives occur confusingly between different recursion levels of the harness. Figure 2 shows the process diagram. The subscript sequences indicate where each item is in the binary tree hierarchy. Each harness is spawned with its input parameters, while the output results are communicated by channel. fun CPH_harness ( depth, max_depth ) ( f, split ) ch input = if ( depth < max_depth ) then let val ch1 = channel(); val ch2 = channel(); val ( combine, input1, input2 ) = split input; val daughter = CPH_harness ( depth+1, max_depth ) ( f, split ); val _ = spawnc (daughter ch1) input1; val _ = spawnc (daughter ch2) input2; val output1 = val output2 =
recv(ch1); recv(ch2);
in send( ch, combine output1 output2 ) end else send( ch, f input );
4.3
Event-Explicit Parallel Hierarchical Harness – EEPH harness
Explicit communication is now done within the RPC functions, so the harness is abstracted from channel use. recvEvt and sendEvt appear together in the RPC calls using the same channel, so less channel management is required. If the recvEvt in the client were a recv the harness would spawn the daughter harnesses sequentially - not what is required! fun EEPH_harness ( depth, max_depth ) ( f, split ) input = if ( depth < max_depth ) then let val ( combine, input1, input2 ) = split input; val daughter = EEPH_harness ( depth+1, max_depth ) ( f, split ); val event1 = rpc_client daughter input1; val event2 = rpc_client daughter input2; val output1 = sync(event1); val output2 = sync(event2); in combine output1 output2 end else f input;
756
4.4
D. Johnston, M. Fleury, and A. Downton
Event-Implicit Parallel Hierarchical Harness – EIPH harness
If the harness purely deals in events, synchronisation can be done at the last possible moment just before the data is about to be sent on to provide maximum decoupling. The event and binary event operator “fires” only when both individual events would fire. The alwaysEvt operator is necessary to turn a value into an (always enabled) event of that value. fun event_and combine event1 event2 = let val event1’ = rpc_client sync event1; val event2’ = rpc_client sync event2; in guard ( fn () => alwaysEvt ( combine ( sync event1’ ) ( sync event2’ ) ) ) end; fun EIPH_harness ( depth, max_depth ) ( f, split ) input = if ( depth < max_depth ) then let val ( combine, input1, input2 ) = split input; val daughter = EIPH_harness ( depth+1, max_depth ) ( f, split ); val event1 = rpc_client daughter input1; val event2 = rpc_client daughter input2; in event_and combine event1 event2 end else alwaysEvt( f input );
4.5
Unified Harness – U harness
The previous parallel harness is identical to the serial harness apart from the use of three higher order functions in three places to abstract the parallelism. Both the serial and parallel harnesses can thus be instanced as parameterised versions of the same generic harness as below. It is difficult to visualise an event-based solution compared to a message-passing one, but the communication patterns are still those shown in Figure 2. However, the timing and software engineering characteristics are considerably better. fun U_harness ( exec, merge_using, return ) ( depth, max_depth ) ( f, split ) input = if ( depth < max_depth ) then let val ( combine, input1, input2 ) = split input; val daughter = U_harness ( exec, merge_using, return ) ( depth+1, max_depth ) ( f, split ); val output1 = exec daughter input1; val output2 = exec daughter input2; in merge_using combine output1 output2 end else return ( f input );
Prototyping Application Models in Concurrent ML
757
val id = fn a => a; (* identity operator *) val val
SH_harness = U_harness EIPH_harness = U_harness
5
Related Work
( id , id , id ); ( rpc_client, event_and, alwaysEvt );
The concept of parallel processing paradigms is almost as old as parallel processing itself. For example, in [11] the ‘divide-and-conquer’ paradigm is characterized as a suitable for architectures in which dynamic process creation is possible. The concept of a harness which encapsulates or abstracts the underlying parallel paradigm or communication pattern seems to have emerged from occam and the transputer, for example in the TINY communications harness [2, 9]. It also seems to have preceded the pattern movement in conventional software [7]. The parallel paradigm is expressed in its clearest form, sometimes called an ‘algorithmic skeleton’ [3], in functional languages. Arising from the algorithmic skeleton approach has come the claim that parallel processing can be executed by functional composition of a restricted number of paradigms [5], such as the pipeline and the task queue, which claim we remain neutral on. Unfortunately, attempts to implement a parallel architecture suitable for processing functional languages seem to have floundered [15]. Therefore it has been suggested [4], that functional languages are best used as a way of expressing parallel decomposition in a non-parallel language. In [14], this approach was indeed taken, but the prototype was in the sequential (i.e Standard) version of ML, before transferring to occam. In this paper, the approach has been taken one stage further, in that a concurrent functional language, CML, has been used to express parallelism. Hammond & Michaelson [8] give an overview of recent work in the parallel functional field.
6
Conclusion and Future Work
CML has proved an effective tool for the development of parallel harnesses: the split/combine harness shown is only one of a number of harnesses that have been developed in this way. The CML model of parallelism is small, clean and elegant providing an ideal porting layer that should enable the harnesses to be moved to a variety of parallel architectures in future. Sadly, execution within the current CML environment is strictly serial. Modern occam in the form of kroc [1] was considered as an implementation layer, and this would have had the advantage of execution in a multi-process environment though prototyping would have been less rapid. In the absence of a genuinely distributed implementation of CML, current research is looking at mixed language solutions (C++ & CML) to establish whether the harnesses can remain in CML or need to be ultimately implemented directly in C++. Some of the harnesses have been implemented in C++ for a distributed processor architecture and have been evaluated for performance on
758
D. Johnston, M. Fleury, and A. Downton
pre-existing image processing applications [13] but this is outside the scope of this paper. The developmental method shifts the ball-park of parallelisation significantly. Instead of writing parallel mechanism code to some external system specification, the user writes application-oriented sequential code, leaving the parallelisation to be described by a separate harness. By freeing the user from the concerns and difficulties of parallelisation, time is available to evolve existing software to match harness templates. The split/combine harness presented can clearly dynamically adapt to the appropriate granularity and scalability for the parallel hardware on which it finds itself running. All the user has to supply are suitable application-specific split and combine routines for the input and output datatypes respectively of a particular function, for that function to be transparently executed in parallel. It was a surprise that the parallel split/combine harness developed under CML emerged homologous to the serial harness, and it is the expressive power of CML that permits such strong unifications. With the appropriate high level constructs it may be that supporting parallel execution is no more complex than writing serial code to the same pattern.
References 1. F. Barnes and P. Welch. Prioritised dynamic communicating processes – part 1. In Communicating Process Architectures – 2002, pages 331–362, 2002. 2. L. Clarke. TINY: Discussion and user guide, 1989. Newsletter 7, Edinburgh Concurrent Supercomputer Project. 3. M. Cole. Algorithmic Skeletons: Structured Management of Parallel Computation. MIT, Cambridge, MA, 1989. 4. M. Cole. Writing parallel programs in non-parallel languages. In R. Perrott, editor, Software for Parallel Computers: Exploiting Parallelism through Software Environments, Tools, Algorithms and Application Libraries, pages 315–325. Chapman & Hall, London, 1992. 5. J. Darlington, A. J. Field, P. G. Harrison, P. H. J. Kelly, and R. L. While. Parallel processing using skeleton functions. In PARLE93, Parallel Architectures and Languages Europe, 1993. LNCS 694. 6. Hoare C.A.R. Ed. Occam 2 Reference Manual. Prentice-Hall, 1988. 7. E. Gamma, R. Helm, R. Johnson, and J. Vlissides. Design patterns: Abstraction and reuse of object-oriented design. In ECOOP’93, pages 406–431, 1993. 8. K. Hammond and G. Michelson. Research Directions in Parallel Functional Programming. Springer Verlag, first edition, 1999. ISBN: 1852330929. 9. A. J. G. Hey. Experiments in MIMD parallelism. In PARLE89, Parallel Architectures and Languages Europe, pages 28–42, 1989. LNCS 366. 10. C. A. R. Hoare. Communicating Sequential Processes. Prentice Hall, first edition, 1985. ASIN: 0131532715. 11. E. Horowitz and A. Zorat. Divide and conquer for parallel computing. IEEE Transactions on Computers, 32(6):582–585, 1983. 12. D Johnston. privatewww.essex.ac.uk/∼djjohn/rappid A Concurrent ML Tutorial, 2002. Essex University.
Prototyping Application Models in Concurrent ML
759
13. D. J. Johnston, M. Fleury, and A. C. Downton. A functional methodology for parallel image processing development. In Visual Information Engineering 2003, 2003. 14. G. Michaelson and N. Scaife. Prototyping a parallel vision system in Standard ml. Functional Programming, Concurrency, Simulation, and Automated Reasoning, 5(3):345–382, 1995. 15. S. L. Peyton Jones and Lester D. Implementing Functional Languages. Prentice Hall, New York, 1992. 16. J. H. Reppy. Concurrent Programming in ML. Cambridge University Press, Cambridge, UK, 1999. 17. M. Spivey, I. Page, and W. Luk. How to program in Handel, 1995. Oxford University Hardware Compilation Unit.
THROOM – Supporting POSIX Multithreaded Binaries on a Cluster Henrik L¨ of, Zoran Radovi´c, and Erik Hagersten Uppsala University, Department of Information Technology P.O. Box 337, SE-751 05 Uppsala, Sweden
[email protected] Abstract. Today, most software distributed shared memory systems (SW-DSMs) lack industry standard programming interfaces which limit their applicability to a small set of shared-memory applications. In order to gain general acceptance, SW-DSMs should support the same look-andfeel of shared memory as hardware DSMs. This paper presents a runtime system concept that enables unmodified POSIX (Pthreads) binaries to run transparently on clustered hardware. The key idea is to extend the single process model of multi-threading to a multi-process model where threads are distributed to processes executing in remote nodes. The distributed threads execute in a global shared address space made coherent by a fine-grain SW-DSM layer. We also present THROOM, a proof-ofconcept implementation that runs unmodified Pthread binaries on a virtual cluster modeled as standard UNIX processes. THROOM runs on top of the DSZOOM fine-grain SW-DSM system with limited OS support.
1
Introduction
Clusters built from high-volume compute nodes, such as workstations, PCs, and small symmetric multiprocessors (SMPs), provide powerful platforms for executing large-scale parallel applications. Software distributed shared memory (SW-DSM) systems can create the illusion of a single shared memory across the entire cluster using a software run-time layer, attached between the application and the hardware. In spite of several successful implementation efforts [1], [2], [3], [4], [5], SW-DSM systems are still not widely used today. In most cases, this is due to the relatively poor and unpredictable performance demonstrated by the SW-DSM implementations. However, some recent SW-DSM systems have shown that this performance gap can be narrowed by removing the asynchronous protocol overhead [5], [6], and demonstrate a performance overhead of only 3040 percent in comparison to hardware DSMs (HW-DSM) [5]. One obstacle for SW-DSMs is the fact that they often require special constructs and/or impose special programming restrictions in order to operate properly. Some SW-DSM systems further alienate themselves from HW-DSMs by relying heavily on very weak memory models in order to hide some of the false sharing created by their page-based coherence strategies. This often leads to large performance variations when comparing the performance of the same applications run on HW-DSMs. H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 760–769, 2003. c Springer-Verlag Berlin Heidelberg 2003
THROOM – Supporting POSIX Multithreaded Binaries on a Cluster
761
We believe that, SW-DSMs should support the same look-and-feel of shared memory as the HW-DSMs. This includes support for POSIX [7] threads running on some standard memory model and a performance footprint similar to that of HW-DSMs, i.e., the performance gap should remain approximately the same for most applications. Our goal is that binaries that run on HW-DSMs could be run on SW-DSMs, without modifications. In contrast to a HW-DSM system, where the whole address space of all processes are kept coherent by hardware, most SW-DSMs only keep coherence for specified segments in the user-level part of the virtual address space. This segment, which we call G MEM, is mapped shared across the DSM nodes using the interconnect hardware. Furthermore, the text (program code), data and stack segments of the UNIX process abstraction are private to the parent process and its children on each node of the cluster. This creates a SW-DSM programming model where special constructs are needed to separate shared data, which must be allocated in G MEM, from private data, which is allocated in the data and stack segments of the UNIX process at program loading. This is often done by creating a separate heap space in G MEM with an associated primitive for doing allocation. In a standard multi-threaded world, there exist only one process and one address space which is shared among all threads. There is no distinction between shared and private data. Consider the following example: An application allocates a shared global array for its threads to operate on. This is often done by a single thread in an initialization phase. In a typical SW-DSM system such as TreadMarks [3], a special malloc()-type call has to be implemented to allocate the memory for the shared array inside the G MEM. Also, the pointer variable holding the address, which is allocated in the static data segment of the process, has to be propagated to all remote nodes. This is often done by introducing a special propagation primitive. In this paper, we present THROOM which is a runtime system concept that creates the illusion of a single process shared memory abstraction on a cluster. In essence, we want to make the static data and heap segments globally accessible by threads executing in remote nodes without introducing special DSM constructs in the application code. In the light of the example above, the application should use a standard malloc() call and the pointer variable should be replicated automatically. The rest of this paper is organized as follows: First, the THROOM concept is presented. Second, we give a brief presentation of the SW-DSM used. We also specify the requirements of THROOM on the SW-DSM. Third, we present a proof-of-concept implementation of THROOM on a single system image cluster and finally we discuss the performance of this implementation as well as the steps needed to take THROOM to a real cluster.
2
The THROOM Concept
In many implementations of SW-DSMs, the different nodes of the cluster all run some daemon process to maintain the G MEM mappings and to deal with requests for coherency actions. In this paper we use the term user node to refer
762
H. L¨ of, Z. Radovi´c, and E. Hagersten
to the cluster node in which the user executes the binary (the user process). All other nodes are called remote nodes and their daemon processes will be called shadow processes. In execution, THROOM consists of one process per node of the cluster. The fundamental idea of THROOM is to distribute threads from the user process to shadow processes executing on remote nodes running different instances of a standard UNIX OS kernel. As discussed earlier, such systems exists, but they require non-standard programming models. To support a standard model such as POSIX, it is required that the whole address space of the user process can be accessed by all of the distributed threads. To accomplish this, we can simply place the text, data and stack segments inside a G MEM-type segment made coherent by a SW-DSM. This will create the illusion of a large scale shared memory multiprocessor built out of standard software and hardware components.
3
DSZOOM – A Fine-Grained SW-DSM
Our prototype implementation is based on the sequentially consistent DSZOOM SW-DSM [5]. Each DSZOOM node can either be a uniprocessor, a SMP, or a CC-NUMA cluster. The node’s hardware keeps coherence among its caches and its memory. The different cluster nodes run different kernel instances and do not share memory with each other in a hardware-coherent way. DSZOOM assumes a cluster interconnect with an inexpensive user-level mechanism to access memory in other nodes, similar to the remote put/get semantics found in the cluster version of the Scalable Coherent Interface (SCI), or the emerging InfiniBand standard that supports RDMA READ/WRITE as well as the atomic operations CmpSwap and FetchAdd [8].1 Another example is the Sun Fire (TM) Link interconnect hardware [9]. While traditional page-based SW-DSMs rely on TLB traps to detect coherence “violations”, fine-grained SW-DSMs like Shasta [10], Blizzard-S [11], Sirocco-S [12] and DSZOOM [5] insert the coherence checks in-line. In DSZOOM, this is done by replacing each load and store that may reference shared data of the binary with a code snippet (short sequence of machine code). In terms of THROOM, the only requirement on the SW-DSM system is that it uses binary instrumentation. THROOM will also inherit the memory consistency model of the SW-DSM system.
4
Implementing THROOM
This section discuss how we can implement the THROOM concept using standard software components and the DSZOOM SW-DSM. 1
Atomic operations are needed to support a blocking directory protocol [5].
THROOM – Supporting POSIX Multithreaded Binaries on a Cluster
4.1
763
Achieving Transparency
The most important aspect of THROOM is that it is totally transparent to the application code, no recompilation is allowed. To achieve this, we use a technique called library interposition or library pre-loading [13], which allow us to change the default behavior of a shared library call without recompiling the binary. Many operating systems implement the core system libraries such as libc, libpthread and libm as shared libraries. Using inter-positioning, we can catch a call, to any shared library and redirect it to our own implementations. In practice, this is done by redefining a symbol in a separate shared library to be pre-loaded at runtime. When an application calls the function represented by the symbol, the runtime linker searches its path for a match. Pre-loading simply means that we can insert an alternate implementation before the standard implementation in the search path of the linker. Pre-loading also allow us to reuse the native implementation. Original arguments can be modified in the interposer before the call to the native implementation is made.2 4.2
Distributing Threads
To distribute threads, the pthread create() call is redefined in a pre-loaded library. The interposed implementation, first schedules the thread for execution in a remote shadow process. Second, the chosen shadow process is told to create a new thread, by calling the native pthread create() from within the interposing library. The new distributed thread will start to execute in the shadow process, with arguments pointing to the software context of its original user process. 4.3
Creating a Global Shared Address Space
A minimal requirement for a distributed thread to execute correctly in a shadow process is that it must share the whole address space of the user process. To accomplish this, the malloc() call is redefined in a pre-loaded library to allocate memory from G MEM instead of the original data segment of the user process. This will make all dynamically allocated data accessible from the shadow processes. Code and static data are made globally accessible by copying the segments containing code and static data from the user process to the G MEM. The application code is then modified, using binary instrumentation, to access the G MEM copy instead of the original segments. This will make the application execute entirely in the global shared memory segment. Hence, no special programming constructs are needed to propagate writes to static data. The whole process is also transparent in the sense that a user does not need access to the application source code, as binary instrumentation modifies the binary itself. All references to the G MEM must also be made coherent as the hardware only support remote reads and writes. This is taken care of by a fine-grain SWDSM. If the SW-DSM use binary instrumentation to insert snippets for access 2
To our knowledge, Linux, Solaris, HP-UX, IRIX and Tru64 all support library preloading.
764
H. L¨ of, Z. Radovi´c, and E. Hagersten
control, we can simply add instructions needed for the static data access diversion to these snippets. In all cases, a maximum of four instructions were added to the existing snippets of DSZOOM. To lower the overheads associated with binary instrumentation, the present implementation does not instrument accesses to the stack. Hence stacks are considered thread private. Although this is not in full compliance with the POSIX model of multi-threading, it is sufficient to support a large set of pthread applications. 4.4
Cluster-Enabled Library Calls
Most applications use system calls and/or calls to standard shared libraries such as libc. If the arguments refer to static data, the accesses must be modified to use the G MEM in order for memory operations to be coherent across the cluster. This can be done in at least two ways. We either instrument all library code or we overload the library calls to copy any changes from the user process original data segments to the G MEM copies at each library call. Remember that un-instrumented code referencing static data of the application will operate in the original data segments of the user process. Hence, copying is needed to make any modifications visible to other nodes. Instrumenting all library code is in principle, the best way to cluster-enable library calls. However, our instrumentation tool, EEL [14], was not able to instrument all of the libraries. Instead, we had to use the library interposition method for our prototype implementation. An obvious disadvantage of this method is that we have to redefine a large amount of library calls, especially if we want complete POSIX support. Another disadvantage is the runtime overhead associated with data copying, especially for I/O operations. A better solution would be to generate the coherence actions on the original arguments before the call is made in the application binary, see Scales et. al. [1]. This requires a very sophisticated instrumentation tool, which is outside the scope of this work.
5
Implementation Details
We have implemented the THROOM system on a 2-node Sun WildFire prototype SMP cluster [15], [16]. The cluster is running a single-system image version of Solaris 2.6 and the hardware is configured as a standard CC-NUMA architecture. Although, this system already supports a global shared address space, we can still use it to emulate a future THROOM architecture. The runtime system is implemented as a shared library. A user simply sets the LD PRELOAD environment variable to the path of the THROOM runtime library, and then executes the instrumented binary. As the system is a single system image we can use standard Inter Process Communication (IPC) primitives to emulate a real distributed cluster. The DSZOOM address space is set up during initialization using the .init section. This makes the whole initialization transparent. Control is then given to the application. The user process issues
THROOM – Supporting POSIX Multithreaded Binaries on a Cluster
765
a fork(2) call to create a shadow process, which will inherit its parents mappings by the copy-on-write semantics of Solaris. The two processes are bound to the two nodes using the WildFire first-touch memory initialization and the pset bind() call. The home process then reads its own /proc file system to locate the .text, .data, and .bss segments and copies them to the G MEM. The shadow process waits on a process shared POSIX conditional variable to create remote threads for execution in the G MEM. Parameters are passed through a shared memory mapping separated from the G MEM. Since the remote thread is created in another process, thread IDs are no longer unique. To fix this, the remote node ID is copied into the most significant eight bits of the thread type, which in the Solaris 2.6 implementation is an unsigned integer. Similar techniques are used for other pthread calls. Also, the synchronization primitives of the application were overloaded using pre-loading to pre-prepared PROCESS SHARED POSIX primitives to allow for multi-process synchronization. More details on the implementation are available in L¨ of et al. [17]. Table 1. Problem sizes and replacement ratios for the 10 SPLASH-2 applications studied. Instrumented loads and stores are showed as a percentage of the total amount of load or store instructions. The number in parenthesis shows the replacement ratio for the DSZOOM SW-DSM without THROOM. Program Problem size, Iterations Replaced Loads (%) Replaced Stores (%) FFT 1 048 576 points (48.1 MByte) 44.6(19.0) 32.8(16.5) LU-C 1024x1024, block 16 (8.0 MByte) 48.3(15.5) 23.0(9.4) LU-NC 1024x1024, block 16 (8.0 MByte) 49.2(16.7) 27.7(11.1) RADIX 4 194 304 items (36.5 MByte) 54.4(15.6) 31.4(11.6) Barnes 16 384 bodies (8.1 MByte) 56.6(23.8) 55.4(31.1) Ocean-C 514x514 (57.5 MByte) 50.6(27.0) 31.2(23.9) Ocean-NC 258x258 (22.9 MByte) 51.0(11.6) 39.0(28.0) Radiosity room (29.4 MByte) 41.1(26.3) 35.1(27.1) Water-NSQ 2197 mol, 2 steps (2.0 Mbyte) 50.4(13.4) 38.0(16.2) Water-SQ 2197 mol, 2 steps (1.5 Mbyte) 48.5(15.7) 32.5(13.9)
6
Performance Study
First a set a test pthread programs were run to verify the correctness of the implementation. To produce a set of pthread programs to be used as a comparison to DSZOOM, ten SPLASH-2 applications [18] were compiled using the GCC v2.95.2 compiler without optimization (-O0)3 and a standard Pthread PARMACS macro implementation (c.m4.pthreads.condvar barrier) was employed. No modifications was made to the PARMACS run-time system or the applications. To exclude the initialization time for the THROOM runtime system, timings are started at the beginning of the parallel phase. All timings have been performed on the 2-node Sun WildFire [15] configured as a traditional CCNUMA architecture. Each node has 16 UltraSPARCII processors running at 250 3
The code is compiled without optimization to eliminate any delay slots, which EEL cannot handle correctly.
766
H. L¨ of, Z. Radovi´c, and E. Hagersten
MHz. The access time to node-local memory is about 330ns. Remote memory is accessed in about 1800ns. In Table 1, we see that more instructions are replaced in the case of THROOM since all references to static data have to be instrumented. This large difference in replacement ratio compared to DSZOOM is explained by the fact that DSZOOM can exploit the PARMACS programming model and use program slicing to remove accesses to static data that are not shared. Figure 1 and 2 shows execution times in seconds for 8- and 16-processor runs for the following THROOM configurations: THROOM RR. THROOM runtime system using library pre-loading. Roundrobin scheduling of threads between the two nodes. All references to static data are instrumented. DSZOOM. Used as reference. Aggressive slicing and snippet optimizations. Optimized for a two-node fork-exec native PARMACS environment, see [5]. CC-NUMA. Uses the same runtime system as DSZOOM but without any instrumentations. Coherence is kept by the WildFire hardware [5]
Fig. 1. Runtime performance of the THROOM runtime system. Two nodes with 4 CPUs each.
A study of Figures 1 and 2 reveals that the present implementation is slower than a state-of-the-art SW-DSM such as DSZOOM. The average runtime overhead compared to DSZOOM for THROOM RR is 65% on 8 CPUs and 78% on 16 CPUs. In order to put these numbers into the context of total SW overhead compared to a HW-DSM, the average slowdown comparing the CC-NUMA and the DSZOOM cases is only 26%. The most significant contribution to the
THROOM – Supporting POSIX Multithreaded Binaries on a Cluster
767
Fig. 2. Runtime performance of the THROOM runtime system. Two nodes with 8 CPUs each.
high overhead when comparing DSZOOM to THROOM is the increased number of instrumentations needed to support the POSIX thread model. Another source of overhead is the inefficient implementation of locks and barriers. This can be observed by comparing the performance of Barnes, Ocean-C, Ocean-NC and Radiosity from Figures 1 and 2. The performance of these four applications drops when increasing the number of threads as they spend a significant amount of time executing in synchronization primitives. The DSZOOM runtime system uses its own implementations of spin-locks and barriers which are more scalable.
7
Related Work
To our knowledge, no SW-DSM system has yet been built that enables transparent execution of an unmodified POSIX binary. The Shasta system [1], [2] come closest to our work and this system has showed that it is possible to run an Oracle database system on a cluster using a fine-grain SW-DSM. Shasta has solved the OS functionality issues in a similar way as is done in THROOM although they support a larger set of system calls and process distribution. THROOM differs from Shasta in that it supports sharing of static data. THROOM also supports thread distribution. Shasta motivates the lack of multi-threading support by claiming that the overhead associated with access checks lead to lower performance [1]. Another system announced recently is the CableS system [19] built on the GeNIMA page-based DSM [6]. This system support a large set of system calls,
768
H. L¨ of, Z. Radovi´c, and E. Hagersten
but they have not been able to achieve binary transparency. Some source code modifications must be made and the code must be recompiled for the system to operate. Another work related to THROOM is the OpenMP interface to the TreadMarks page-based DSM [20] [3], where a compiler front-end translates the OpenMP pragmas into TreadMark fork-join style primitives. The DSM-Threads system [21] provide a page-based DSM interface similar to the Pthreads standard without binary transparency.
8
Conclusions
We have showed that it is possible to extend a single process address space to a multi-process model. Even though the current THROOM implementation relies on some of the WildFire’s single system image properties, we are convinced that the THROOM concept can be implemented on a real cluster. In a pure distributed setting, additional issues need to be addressed. One way of initializing the system could be to use a standard MPI runtime system for process creation and handshaking. The address space mappings must also be set up using the RDMA features of the interconnect hardware. Also, synchronization needs to be handled more efficiently (see Radovi´c et. al. [22]), and we need to create more complete and more efficient support for I/O and other library calls. For complete POSIX compliance, we also need to address the problem of threads sharing data on the stack.
References 1. Scales, D.J., Gharachorloo, K.: Towards Transparent and Efficient Software Distributed Shared Memory. In: Proceedings of the 16th ACM Symposium on Operating System Principles, Saint-Malo, France. (1997) 2. Dwarkadas, S., Gharachorloo, K., Kontothanassis, L., Scales, D.J., Scott, M.L., Stets, R.: Comparative Evaluation of Fine- and Coarse-Grain Approaches for Software Distributed Shared Memory. In: Proceedings of the 5th International Symposium on High-Performance Computer Architecture. (1999) 260–269 3. Keleher, P., Cox, A.L., Dwarkadas, S., Zwaenepoel, W.: TreadMarks: Distributed Shared Memory on Standard Workstations and Operating Systems. In: Proceedings of the Winter 1994 USENIX Conference. (1994) 115–131 4. Stets, R., Dwarkadas, S., Hardavellas, N., Hunt, G., Kontothanassis, L., Parthasarathy, S., Scott, M.: Cashmere-2L: Software Coherent Shared Memory on a Clustered Remote-Write Network. In: Proceedings of the 16th ACM Symposium on Operating System Principle. (1997) 5. Radovi´c, Z., Hagersten, E.: Removing the Overhead from Software-Based Shared Memory. In: Proceedings of Supercomputing 2001. (2001) 6. Bilas, A., Liao, C., Singh, J.P.: Using Network Interface Support to Avoid Asynchronous Protocol Processing in Shared Virtual Memory Systems. In: Proceedings of the 26th Annual International Symposium on Computer Architecture (ISCA’99). (1999)
THROOM – Supporting POSIX Multithreaded Binaries on a Cluster
769
7. IEEE Std 1003.1-1996, ISO/IEC 9945-1: Portable Operating System Interface (POSIX)–Part1: System Application Programming Interface (API) [C Language]. (1996) 8. InfiniBand(SM) Trade Association: InfiniBand Architecture Specification, Release 1.0. (2000) Available from: http://www.infinibandta.org. 9. Sistare, S., Jackson, C.J.: Ultra-High Performance Communication with MPI and the Sun Fire(TM) Link Interconnect. In: Proceedings of the IEEE/ACM SC2002 Conference. (2002) 10. Scales, D.J., Gharachorloo, K., Thekkath, C.A.: Shasta: A Low-Overhead SoftwareOnly Approach to Fine-Grain Shared Memory. In: Proceedings of the 7th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS-VII). (1996) 174–185 11. Schoinas, I., Falsafi, B., Lebeck, A.R., Reinhardt, S.K., Larus, J.R., Wood, D.A.: Fine-grain Access Control for Distributed Shared Memory. In: Proceedings of the 6th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS-VI). (1994) 297–306 12. Schoinas, I., Falsafi, B., Hill, M., Larus, J.R., Wood, D.A.: Sirocco: Cost-Effective Fine-Grain Distributed Shared Memory. In: Proceedings of the 6th International Conference on Parallel Architectures and Compilation Techniques. (1998) 13. Thain, D., Livny, M.: Multiple Bypass, Interposition Agents for Distributed Computing. In: Cluster Computing. (2001) 14. Larus, J.R., Schnarr, E.: EEL: Machine-Independent Executable Editing. In: Proceedings of the SIGPLAN ’95 Conference on Programming Language Design and Implementation. (1995) 291–300 15. Hennessy, J.L., Patterson, D.A.: Computer Architecture: A Quantative Approach 3:rd edition. Morgan Kaufman (2003) 16. Hagersten, E., Koster, M.: WildFire: A Scalable Path for SMPs. In: Proceedings of the 5th IEEE Symposium on High-Performance Computer Architecture. (1999) 172–181 17. L¨ of, H., Radovi´c, Z., Hagersten, E.: THROOM — Running POSIX Multithreaded Binaries on a Cluster. Technical Report 2003-026, Department of Information Technology, Uppsala University (2003) 18. Woo, S.C., Ohara, M., Torrie, E., Singh, J.P., Gupta, A.: The SPLASH-2 Programs: Characterization and Methodological Considerations. In: Proceedings of the 22nd Annual International Symposium on Computer Architecture (ISCA’95). (1995) 24–36 19. Jamieson, P., Bilas, A.: CableS: Thread Control and Memory Mangement Extentions for Shared Virtual Memory Clusters. In: 8th International Symposium on High-Performance Computer Architeture, HPCA-8. (2002) 20. A. Scherer, H. Lu, T.G., Zwaenepoel, W.: Transparent Adaptive Parallelism on NOWs using OpenMP. In: Principles Practice of Parallel Programming. (1999) 21. Mueller, F.: Distributed Shared-Memory Threads:DSM-Threads. In: Proc. of the Workshop on Run-Time Systems for Parallel Programming. (1997) 22. Radovi´c, Z., Hagersten, E.: Efficient Synchronization for Nonuniform Communication Architectures. In: Proceedings of Supercomputing 2002. (2002)
An Inter-entry Invocation Selection Mechanism for Concurrent Programming Languages Aaron W. Keen1 and Ronald A. Olsson2 1
Computer Science Department, California Polytechnic State University, San Luis Obispo, CA 93407 USA
[email protected] 2 Department of Computer Science, University of California, Davis, Davis, CA 95616 USA
[email protected] Abstract. Application-level message passing is supported by many concurrent programming languages. Such languages allow messages to be generated by invoking an entry. Messages are removed from an entry by an invocation selection mechanism. Such mechanisms may allow selection from a set of entries (multi-way receive). Many concurrent languages provide support for multi-way receives, but they are limited in their expressive power. This paper presents three proposed inter-entry selection mechanisms. The proposed mechanisms overcome the limitations of the existing mechanisms. Each of these mechanisms allows an invocation selection algorithm to examine the entire set of pending invocations (and their parameters) as part of the invocation selection process. These mechanisms are analyzed and compared both qualitatively and quantitatively.
1
Introduction
Many concurrent programming languages support application-level message passing as a form of thread synchronization. A message, henceforth called an invocation, is generated when a process invokes an operation (i.e., an entry or port), optionally passing it parameters (such message passing includes synchronous and asynchronous invocation of an operation). Once generated, the invocation is added to the set of pending invocations associated with the invoked operation. An invocation is removed from an operation’s set by a servicing process using an invocation servicing mechanism. Such a mechanism, depending on the language, may allow selection of an invocation from a single operation (a receive) or from one of potentially many operations (a multi-way receive). This paper focuses on multi-way receives and the invocation selection support provided. Ada [12], Concurrent C [9], CSP [11,15,10], Erlang [3], Hermes [17], JR [14], Limbo [8], occam [5,16], Orca [4], SR [2,1], and SRR [7] each provide support for multi-way receives, but with differing expressive power. None of these languages provides general support for inter-entry invocation selection. H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 770–780, 2003. c Springer-Verlag Berlin Heidelberg 2003
An Inter-entry Invocation Selection Mechanism
771
Specifically, these languages do not provide simultaneous access to all pending invocations (both within an operation’s set and over the set of operations) during selection. Inter-entry selection facilitates the implementation of, for example, lottery scheduling for resource management [18] (in which a manager process picks a request randomly) and preferential job scheduling [7] (in which preference is given to normal interactive jobs unless there is a superuser batch job). This paper presents three proposed inter-entry invocation selection mechanisms that address the limitations discussed above. The designs combine aspects of functional, object-oriented, and concurrent programming languages. The result of this study is the addition of the most balanced (in terms of abstraction and performance) mechanism to the JR concurrent programming language [14], which provides threads and message passing. The rest of this paper is organized as follows. Section 2 discusses the limitations of multi-way receives provided by other languages. Section 3 describes the details of the proposed selection mechanisms. Section 4 analyzes the proposed mechanisms. Section 5 compares the proposed mechanisms. Section 6 concludes. Further discussion and details appear in Reference [13].
2
Background
Many concurrent languages provide support for multi-way receives, but with limited expressive power. For example, Ada provides the select statement, which allows multiple accept statements, each servicing an operation. When a select statement is executed, an operation is nondeterministically selected from those with pending invocations, and an invocation from the selected operation is serviced. Figure 1 demonstrates, using JR syntax (where an inni statement specifies a multi-way receive), a simple two-way receive that services invocations from two operations. Each accept statement may be preceded by a boolean guard to restrict the operations considered for servicing. A guard, however, cannot access an invocation’s arguments and, thus, cannot use the values of these arguments in deciding which invocation to select. These guards provide a very coarse-grained control over selection. Other languages have similar limitations. For example, occam supports boolean guards to restrict the candidate invocations, but does not allow selection based on an invocation’s parameters or on the entire set of pending invocations. Erlang allows a boolean guard that specifies which messages are acceptable for servicing, but does not allow selection based on all pending invocations. SR provides a more flexible invocation selection statement called the input (in) statement. A guard on an arm of an input statement can contain a synchronization expression and a scheduling expression. The former specifies which invocations are acceptable for servicing; the latter specifies the order in which to service acceptable invocations. Figure 2 shows a modification of the two-way receive from the earlier example, lines 7–8 in Figure 1. This modified two-way receive uses scheduling expressions (specified via by clauses) to service each operation’s invocations in order of highest priority (lowest integer value). Unlike
772
A.W. Keen and R.A. Olsson 1 public class Server { 2 public op void entryOne(int priority, int b); 3 public op void entryTwo(int priority, float b); 4 protected void server() { 5 /* repeatedly service an invocation from entryOne or entryTwo */ 6 while (true) { 7 inni void entryOne(int priority, int b) { /* service */ } 8 [] void entryTwo(int priority, float b) { /* service */ } 9 } 10 } 11 }
Fig. 1. Simple Two-way Receive.
the guards in Ada, SR’s synchronization and scheduling expressions can access an invocation’s arguments and use their values to determine an invocation’s acceptability and to order invocations. Such expressions, however, cannot simultaneously access the arguments of multiple invocations, and cannot be used to order invocations between multiple operations. 7 8
inni void entryOne(int priority, int b) by -priority [] void entryTwo(int priority, float b) by -priority
{ /* service */ } { /* service */ }
Fig. 2. Two-way Receive with Scheduling Expression.
SRR extends SR to provide the rd statement, which allows the examination of all pending invocations of a set of operations. The rd statement, however, cannot directly service an invocation. An invocation is marked for service by a mark statement and is serviced by a subsequent take statement. Though this approach allows selection based on the pending invocations, the separation of selection into three non-atomic stages (rd, mark, and take) can complicate solutions. For example, one thread might mark an invocation and then attempt to take it, only to find that it was serviced by another thread in the intervening time between marking and taking. None of the aforementioned mechanisms provides support for implementing selection algorithms that require atomic access to the entire set of pending invocations. This class of selection algorithms includes debugging, visualization, and scheduling (e.g., lottery scheduling). Extending the example in Figure 2, these mechanisms cannot directly enforce the priority ordering between operations; an invocation in entryTwo may be serviced even if entryOne has a higher priority invocation pending.
3
Proposed Invocation Selection Mechanisms
An acceptable inter-entry invocation selection mechanism must: – allow atomic selection of any pending invocation in the set of operations serviced. – provide access to each invocation’s actual arguments.
An Inter-entry Invocation Selection Mechanism
773
– disallow removal of multiple invocations from the set of pending invocations. – disallow insertion into the set of pending invocations as a side-effect. We devised three approaches that satisfy these criteria: Invocation Enumeration, Functional Reduction, and Hybrid. Invocation Enumeration and Functional Reduction are discussed below. The Hybrid approach combines aspects of the other approaches, but mixes programmer abstractions and performs poorly, so it is not discussed further (though the experiments do include Hybrid). 3.1
Invocation Enumeration
The Invocation Enumeration approach provides an enumeration of the currently pending invocations. This enumeration is passed to a programmer-specified method that selects an individual invocation. The inni statement services the invocation returned from this method. We designed two approaches to Invocation Enumeration: View Enumeration and Named Enumeration. These variants differ in how they name invocations and invocation parameters. View Enumeration names each invocation’s parameters as invocations are extracted from an enumeration. Named Enumeration names invocations as they are placed into an enumeration. 1 2 3 4 5
Invocation select_method(ArmEnumeration arm) { ... } ... inni with select_method over void entryOne(int i, float f) { ... } [] void entryTwo(int i, float f, float g) { ... }
Fig. 3. Invocation Enumeration: Specification of selection method.
Both approaches use a with-over clause to specify the selection method (line 3 of Figure 3). The method must have type signature ArmEnumeration → Invocation. An ArmEnumeration enumerates InvocationEnumerations, each corresponding to the operation serviced by an arm of the inni statement. Each InvocationEnumeration is an enumeration of the respective operation’s pending invocations. Figure 4 gives a pictorial example of the enumeration structure. arm enumeration
a1 a 200 11 11 00 00 a n−1 a n entryOne 11
invocation enumeration
b1 b200 11 11 00 00 bm−1 bm entryTwo 11
Fig. 4. Arm and Invocation Enumeration Structure.
View Enumeration. An abridged implementation of the selection method used in Figure 3 is given in Figure 5. This example demonstrates accesses to individual invocations and their arguments. Line 3 shows the extraction of
774
A.W. Keen and R.A. Olsson
an InvocationEnumeration from the ArmEnumeration. An Invocation is extracted from an InvocationEnumeration on line 6. Under View Enumeration, the class Invocation is viewed as a variant type [6] of all invocation types and each invocation as a value of this variant type. A view statement is used to determine the underlying type of a specific invocation and to provide access to the arguments of the invocation. The view statement on lines 7–8 compares invocation invoc to the underlying invocation types int×f loat and int×f loat×f loat. The statement associated with the matching as clause is executed with the invocation’s arguments bound to the respective parameters. 1 2 3 4 5 6 7 8 9 10 11
Invocation select_method(ArmEnumeration arm) { while (arm.hasMoreElements()) { InvocationEnumeration invoc_enum = arm.nextElement(); if (invoc_enum == null) continue; while (invoc_enum.hasMoreElements()) { Invocation invoc = invoc_enum.nextElement(); view invoc as (int i, float f) { ... } // block using i and f as (int i, float f, float g) { ... } // block using i, f, and g } } }
Fig. 5. View Enumeration: Implementation of selection method.
Named Enumeration. Named Enumeration allows the programmer to “name” the invocation type used within an invocation enumeration. These types are specified via as clauses (Figure 6). Each “named” type (e.g., nameOne and nameTwo) is a class that extends the Invocation class and provides a constructor with type signature matching the operation being serviced. The “named” types are used within the selection method to access each invocation’s parameters. 1 2 3
inni with select_method over void entryOne(int i, float f) as nameOne [] void entryTwo(int i, float f, float g) as nameTwo
{ ... } { ... }
Fig. 6. Named Enumeration: Specification of selection method and invocation types.
3.2
Functional Reduction
The Functional Reduction approach splits invocation selection into two phases to provide a simple means for accessing the invocation parameters. Figure 7 gives a pictorial example of the two phases. The first phase (depicted by the horizontal arrows), which has direct access to the parameters of the invocations, selects a single invocation from each arm of the inni statement; this invocation is called the representative for its respective arm. The second phase (depicted by the vertical arrow), which can access invocation parameters through user-defined invocation types, selects the actual invocation to service from the representatives. Each phase selects an invocation through a reduction, using a programmerspecified method, over the appropriate set of invocations.
An Inter-entry Invocation Selection Mechanism arm reduction
representative reduction
775
a1 a 2 0 10 10 1 a n−1 a n entryOne b1 b2 0 10 10 1 bm−1 bm entryTwo
Fig. 7. Arm Reduction and Representative Reduction Structure.
4
Analysis of Mechanisms
Though the proposed mechanisms all satisfy the criteria for acceptable invocation selection mechanisms, they differ in terms of abstraction and performance. This section discusses solutions to representative examples from three problem domains. In the interest of space, and since it is ultimately adopted, only the View Enumeration solutions are shown. The problem domains are: – Selection Independent of Arguments: Does not require access to the arguments of the invocations. The representative, RANDOM, selects a random invocation from the pending invocations. – Single Comparison Required: A comparison between two invocations is sufficient to eliminate one from consideration. The representative, PRIORITY, selects an invocation based on a “priority” argument. – Multiple Comparisons Required: Simultaneous examination and comparison of multiple invocations is required. The representative, MEDIAN, selects the invocation with the median first argument of all pending invocations.
1 Invocation select_random(ArmEnumeration arm) { 2 int num = 0; 3 while (arm.hasMoreElements()) { // tally invocations 4 InvocationEnumeration invoc_enum = arm.nextElement(); 5 if (invoc_enum != null) num += invoc_enum.length(); 6 } 7 int rand = RandomInvoc.rand.nextInt(num); 8 arm.reset(); // return to beginning of enumeration 9 while (arm.hasMoreElements()) { // find invocation 10 InvocationEnumeration invoc_enum = arm.nextElement(); 11 if (invoc_enum != null) { 12 if (rand >= invoc_enum.length) rand -= invoc_enum.length; 13 else { 14 while (rand > 0) { // find invocation in this "arm" 15 invoc_enum.nextElement(); rand--; 16 } 17 return (Invoc) (invoc_enum.nextElement()); 18 } } } 19 return null; // Shouldn’t get here 20 }
Fig. 8. View Enumeration: RANDOM.
4.1
Selection Independent of Arguments
Invocation Enumeration. Figure 8 gives the selection method used in both Invocation Enumeration solutions to RANDOM. The select_random method
776
A.W. Keen and R.A. Olsson
calculates the total number of pending invocations in the enumeration, generates a random number in the range, and, finally, returns the “random” invocation. Functional Reduction. A Functional Reduction solution to RANDOM first selects a representative invocation for each arm by randomly determining if the “current” invocation should be the representative based on the number of invocations examined thus far (i.e., the “previous” invocation carries a weight based on the number of invocations that it has beaten out). Finally, the invocation to service is selected from the representatives in the same manner. 4.2
Single Comparison Required
Invocation Enumeration. Figure 9 gives the selection method used in a View Enumeration solution to PRIORITY. The algorithm used is very simple: loop through the invocations for each arm and record the invocation with the highest priority. A view statement (line 9) is used to access each invocation’s arguments. 1 Invocation prio_select(ArmEnumeration arm_enum) { 2 Invocation cur = null; 3 int best_prio = 0, cur_prio; 4 while (arm_enum.hasMoreElements()) { 5 InvocationEnumeration invoc_enum =arm_enum.nextElement(); 6 if (invoc_enum == null) continue; 7 while (invoc_enum.hasMoreElements()) { 8 Invocation invoc = invoc_enum.nextElement(); 9 view invoc as (int prio_tmp, int i) //params of invoc 10 cur_prio = prio_tmp; 11 ... // for each invocation type 12 if ((cur == null) || (cur_prio < best_prio)) { 13 cur = invoc; best_prio = cur_prio; 14 } } } 15 return cur; 16 }
Fig. 9. View Enumeration: PRIORITY.
Functional Reduction. The Functional Reduction solution to PRIORITY compares the “current” invocation with the “previous” invocation and returns the one with highest priority. This general reduction is used to first select the representative invocations and then to select the invocation to service. 4.3
Multiple Comparisons Required
Invocation Enumeration. Figure 10 outlines the selection method used in a View Enumeration solution to MEDIAN. The algorithm used gathers the invocations into a Vector, converts the Vector into an array, sorts the array, and, finally, selects the median invocation. Functional Reduction. The Functional Reduction solution to MEDIAN highlights the drawbacks of the approach. The arm reduction method gathers each invocation into a Vector stored within a user-defined object. It is necessary to gather the invocations in this manner because the Functional Reduction approach does not provide direct simultaneous access to all of the invocations.
An Inter-entry Invocation Selection Mechanism
777
1 Invocation median_select(ArmEnumeration arm) { 2 Vector v = new Vector(); 3 while (arm.hasMoreElements()) { 4 InvocationEnumeration invoc_enum = arm.nextElement(); 5 if (invoc_enum == null) continue; 6 view invoc as (int firstarg, int i) // params of invoc 7 v.add(new Element(invoc, firstarg)); 8 ... // for each invocation type 9 } 10 ... // convert vector, sort, and return median invocation 11 }
Fig. 10. View Enumeration: MEDIAN.
With the invocations for each arm gathered into objects, a reduction over the arms selects the final invocation to service. Unfortunately, since a specific call of the reduction method cannot determine if it is the last, each call of the reduction method must select the “final” invocation to service (i.e., the median invocation of those examined thus far). As such, each call of the representative reduction method performs the costly median invocation selection. 4.4
Performance
A number of experiments were run to evaluate the performance of each of the proposed mechanisms. The experiments were conducted on a 850 MHz Intel Pentium III with 128 MB of RAM running Linux kernel 2.2.19 and IBM’s JRE 1.3.0. The experiments apply the solutions to an inni statement servicing 10 distinct operations with pending invocations initially evenly distributed. The results report, for each solution, the time to select and service all pending invocations. (Other experiment sizes were also run with similar results.)
70
View Enumeration Named Enumeration Functional Reduction Hybrid
60
View Enumeration Named Enumeration Functional Reduction Hybrid
5000
40
Time (s)
Time (s)
50
30
3000
20 1000
10 0 0
2 4 6 8 Number of Pending Invocations (in thousands)
10
0
20 40 60 80 Number of Pending Invocations (in thousands)
100
Fig. 11. RANDOM: Time to service all pending invocations.
RANDOM. Figure 11 plots the number of initially pending invocations versus time for each solution to RANDOM. In this experiment, Functional Reduction does not compare favorably to Invocation Enumeration. It must be noted, however, that the Invocation Enumeration solution to RANDOM implements a more efficient algorithm than does Functional Reduction. The Invocation Enumeration
778
A.W. Keen and R.A. Olsson
solutions use a divide-and-conquer approach to examine a fraction of the pending invocations. An operation’s entire set of pending invocations is skipped if the random invocation is not a member of the set (see lines 12-16 in Figure 8). The same divide-and-conquer algorithm cannot be implemented using the Functional Reduction approach because the reduction examines every invocation. PRIORITY. Figure 12 plots the number of initially pending invocations versus time for each solution to PRIORITY. This problem requires access to invocation arguments, which is the reason that the Named Enumeration solution performs poorly. To support the abstraction provided by Named Enumeration, an object (of the programmer “named” type) must be created for each pending invocation. This accounts for the performance difference between View and Named Enumeration. MEDIAN. Figure 13 plots the number of initially pending invocations versus time for each solution to MEDIAN. Functional Reduction performs poorly because the expensive selection computation (selecting the median element) is executed for each call of the arm reduction method (equal to the number of arms).
140
11000
View Enumeration Named Enumeration Functional Reduction Hybrid
120
View Enumeration Named Enumeration Functional Reduction Hybrid
9000
100 7000 Time (s)
Time (s)
80 60 40
5000
3000
20 1000 0 0
2 4 6 8 Number of Pending Invocations (in thousands)
10
0
20 40 60 80 Number of Pending Invocations (in thousands)
100
Fig. 12. PRIORITY: Time to service all pending invocations.
8
13000
View Enumeration Named Enumeration Functional Reduction Hybrid
View Enumeration Named Enumeration Functional Reduction Hybrid
11000
6
Time (s)
Time (s)
9000
4
7000 5000
2
3000 1000
0 0
0.2 0.4 0.6 0.8 Number of Pending Invocations (in thousands)
1
0
20 Number of Pending Invocations (in thousands)
Fig. 13. MEDIAN: Time to service all pending invocations.
40
An Inter-entry Invocation Selection Mechanism
5
779
Discussion
View Enumeration performs the best. The view statement incurs little overhead over direct access to the underlying operation implementation, while providing sufficient abstraction of that implementation. Unfortunately, the structure of a view statement must closely match that of the associated inni statement. This structural dependence potentially limits selection method reuse. Named Enumeration reduces this dependence, but incurs object creation overhead. Functional Reduction is elegant, but suffers greatly in terms of performance. This strategy’s performance penalties can be categorized into method call overhead, repeated selection, and state maintenance. A method call overhead is incurred for each invocation, even when the invocation to service has already been found. In such a case, the number of method calls could be minimized if it were possible to abort reduction. Repeated selection is the repeated execution of selection code to satisfy an invariant (as in MEDIAN). A predicate indicating the last method call of a reduction could reduce the repeated selection penalty. State maintenance is the need to explicitly carry extra state through a reduction (as in MEDIAN). This, unfortunately, makes it necessary to create extra objects.
6
Conclusion
This paper discussed different candidate selection mechanisms considered for addition to the JR programming language. For consideration, a candidate had to satisfy criteria requiring selection of any invocation from the set of pending invocations, access to each invocation’s actual arguments, and the prevention of selection side-effects. We have extended the JR programming language with support for the View Enumeration approach because of its high-level of abstraction and performance. Because View Enumeration does not rely on subclassing, this approach can also be used in concurrent languages that are not object-oriented.
References 1. G. R. Andrews and R. A. Olsson. The SR Programming Language: Concurrency in Practice. The Benjamin/Cummings Publishing Co., Redwood City, CA, 1993. 2. G. R. Andrews, R. A. Olsson, M. Coffin, I. Elshoff, K. Nilsen, T. Purdin, and G. Townsend. An overview of the SR language and implementation. ACM Transactions on Programming Languages and Systems, 10(1):51–86, January 1988. 3. J. Armstrong, R. Virding, and M. Williams. Concurrent Programming in Erlang. Prentice Hall, Englewood Cliffs, New Jersey, 1993. 4. H. E. Bal, M. F. Kaashoek, and A. S. Tanenbaum. Orca: A language for parallel programming of distributed systems. IEEE Transactions on Software Engineering, 18(3):190–205, March 1992. 5. A. Burns. Programming in Occam. Addison Wesley, 1988. 6. L. Cardelli and P. Wegner. On understanding types, data abstraction, and polymorphism. ACM Computing Surveys, 17(4):471–522, 1985.
780
A.W. Keen and R.A. Olsson
7. M. Chung and R. A. Olsson. New mechanisms for invocation handling in concurrent programming languages. Computer Languages, 24:254–270, December 1998. 8. S. Dorward and R. Pike. Programming in Limbo. In Proceedings of the IEEE Compcon 97 Conference, pages 245–250, 1997. 9. N. Gehani and W.D. Roome. The Concurrent C Programming Language. Silicon Press, Summit, NJ, 1989. 10. G. Hilderink, J. Broenink, W. Vervoort, and A. Bakkers. Communicating Java Threads. In WoTUG 20, pages 48–76, 1997. 11. C. A. R. Hoare. Communicating sequential processes. Communications of the ACM, 21(8):666–677, 1978. 12. Intermetrics, Inc., 733 Concord Ave, Cambridge, Massachusetts 02138. The Ada 95 Annotated Reference Manual (v6.0), January 1995. 13. A. W. Keen. Integrating Concurrency Constructs with Object-Oriented Programming Languages: A Case Study. PhD dissertation, University of California, Davis, Department of Computer Science, June 2002. http://www.csc.calpoly.edu/˜akeen/papers/thesis.ps. 14. A. W. Keen, T. Ge, J. T. Maris, and R. A. Olsson. JR: Flexible distributed programming in an extended Java. In Proceedings of the 21st IEEE International Conference on Distributed Computing Systems, pages 575–584, April 2001. 15. University of Kent. Communicating sequential processes for Java. http://www.cs.kent.ac.uk/projects/ofa/jcsp/. 16. University of Kent. Kent retargetable occam compiler. http://www.cs.kent.ac.uk/projects/ofa/kroc/. 17. R. E. Strom et al. Hermes: A Language for Distributed Computing. Prentice Hall, Englewood Cliffs, New Jersey, 1991. 18. C.A. Waldspurger and W.E. Weihl. Lottery Scheduling: Flexible ProportionalShare Resource Management. In Proceedings of the First Symposium on Operating System Design and Implementation, pages 1–11, Monterey, CA, November 1994.
Parallel Juxtaposition for Bulk Synchronous Parallel ML Fr´ed´eric Loulergue Laboratory of Algorithms, Complexity and Logic 61, avenue du g´en´eral de Gaulle – 94010 Cr´eteil cedex – France
[email protected] Abstract. The BSMLlib library is a library for Bulk Synchronous Parallel (BSP) programming with the functional language Objective Caml. It is based on an extension of the λ-calculus by parallel operations on a parallel data structure named parallel vector. An attempt to add a parallel composition to this approach led to a non-confluent calculus and to a restricted form of parallel composition. This paper presents a new, simpler and more general semantics for parallel composition.
1
Introduction
Declarative parallel languages are one possible way that may ease the programming of massively parallel architectures. Those languages do not enforce in the syntax itself an order of evaluation, and thus appear more suitable to automatic parallelization. Functional languages are often considered. Nevertheless, even if some problems encountered in the parallelization of sequential imperative languages are avoided, some still remain (for example two different but denotationally equivalent programs may lead to very different parallel programs) and some are added, for example the fact that in those languages data-structures are always dynamic ones. It makes the amount and/or the grain of parallelism often too low or difficult to control in case of speculation. An opposite direction of research is to give the programmer the entire control over parallelism. Message passing facilities are added to functional languages. But in this case, the obtained parallel languages are either non-deterministic, [10] or non-functional (i.e. referential transparency is lost) [3]. The design of parallel programming languages is, therefore, a tradeoff between the possibility of expressing parallel features necessary for predictable efficiency, but which make programs more difficult to write, to prove and to port ; the abstraction of such features that are necessary to make parallel programming easier, but which must not hinder efficiency and performance prediction. An intermediate approach is to offer only a set of algorithmic skeletons [1, 11] that are implemented in parallel. Among researchers interested in declarative parallel programming, there is a growing interest in execution cost models
This work is supported by the ACI Grid program from the French Ministry of Research, under the project Caraml (www.caraml.org).
H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 781–788, 2003. c Springer-Verlag Berlin Heidelberg 2003
782
F. Loulergue
taking into account global hardware parameters like the number of processors and bandwidth. The Bulk Synchronous Parallel [12] execution and cost model offers such possibilities and with similar motivations we have designed BSP extensions of the λ-calculus [8] and a library for the Objective Caml language, called BSMLlib, implementing those extensions. This framework is a good tradeoff for parallel programming for two main reasons. First, we defined a confluent calculus so we can design purely functional parallel languages from it. Without side-effects, programs are easier to prove, and to re-use (the semantics is compositional) and we can choose any evaluation strategy for the language. An eager language allows good performances. Secondly, this calculus is based on BSP operations, so programs are easy to port, their costs can be predicted and are also portable because they are parametrized by the BSP parameters of the target architecture. A BSP algorithm is said to be in direct mode [5] when its physical process structure is made explicit. Such algorithms offer predictable and scalable performance and BSML expresses them with a small set of primitives taken from the confluent BSλ calculus [8]: a constructor of parallel vectors, asynchronous parallel function application, synchronous global communications and a synchronous global conditional. Those operations are flat: it is impossible to express directly parallel divide-and-conquer algorithms. Nevertheless many algorithms are expressed as parallel divide-and-conquer algorithms [13] and it is difficult to transform them into flat algorithms. In a previous work, we proposed an operation called parallel composition [7], but is was limited to the composition of two terms whose evaluations require the same number of BSP super-steps. In this paper we present a new version of this operation and a new operation which allow to juxtapose (this is the reason why we renamed this operation parallel juxtaposition) on subsets of the parallel machine the evaluations of two terms even if they require a different number of BSP super-steps. We first present the flat BSMLlib library (section 2). Section 3 addresses the semantics of the parallel juxtaposition. We also give a small example in section 4, discuss related and future work in section 5 and conclude in section 6.
2
Functional Bulk Synchronous Parallelism
There is currently no implementation of a full Bulk Synchronous Parallel ML language but rather a partial implementation as a library for Objective Caml. The so-called BSMLlib library is based on the following elements. It gives access to the BSP parameters of the underling architecture. In particular, it offers the function: bsp p:unit->int such as the value of bsp p() is p, the static number of processes of the parallel machine. This value does not change during execution, for “flat” programming only. It is no longer true when parallel juxtaposition is added to the language. There is also an abstract polymorphic type ’a par which represents the type of p-wide parallel vectors of objects of type ’a, one per process. The nesting of par types is prohibited. We have a type system [4] which enforces this restriction.
Parallel Juxtaposition for Bulk Synchronous Parallel ML
783
The BSML parallel constructs operates on parallel vectors. Those parallel vectors are created by: mkpar: (int -> ’a) -> ’a par so that (mkpar f) stores (f i) on process i for i between 0 and (p − 1). We usually write f as fun pid->e to show that the expression e may be different on each processor. This expression e is said to be local. The expression (mkpar f) is a parallel object and it is said to be global. A BSP algorithm is expressed as a combination of asynchronous local computations and phases of global communication with global synchronization. Asynchronous phases are programmed with mkpar and with: apply: (’a -> ’b) par ->’a par -> ’b par apply (mkpar f) (mkpar e) stores (f i) (e i) on process i. The communication and synchronization phases are expressed by: put: (int->’a option) par -> (int->’a option) par where ’a option is defined by: type ’a option = None | Some of ’a. Consider the expression: put(mkpar(fun i->fsi ))(∗). To send a value v from process j to process i, the function fsj at process j must be such that (fsj i) evaluates to Some v. To send no value from process j to process i, (fsj i) must evaluate to None. Expression (∗) evaluates to a parallel vector containing a function fdi of delivered messages on every process. At process i, (fdi j) evaluates to None if process j sent no message to process i or evaluates to Some v if process j sent the value v to the process i. A global conditional operation: ifat : (bool par) ∗ int ∗ ’a * ’a -> ’a would also be contained in the full language. It is such that ifat (v,i,v1,v2) will evaluate to v1 or v2 depending on the value of v at process i. But Objective Caml is an eager language and this synchronous conditional operation can not be defined as a function. That is why the core BSMLlib contains the function: at:bool par -> int -> bool to be used only in the construction: if (at vec pid) then... else... where (vec:bool par) and (pid:int). if at expresses communication and synchronization phases. Without it, the global control cannot take into account data computed locally. Global conditional is necessary of express algorithms like: Repeat Parallel Iteration Until Max of local errors < Due to lack of space, in the following sections we will omit it.
3
Semantics of Parallel Juxtaposition
Parallel juxtaposition allows the network to be divided in two subsets. If we add a notion of parallel juxtaposition to the BSλ-calculus, the confluence is lost [8]. Parallel juxtaposition has to preserve the BSP execution structure. Moreover the number of processors is no longer a constant and depends on the context of the term. “Flat” terms. The local terms (denoted by lowercase characters) represents “sequential” values held locally by each processor. They are classical λexpressions. They are given by the following grammar:
784
F. Loulergue
e ::= n integer constant | λx. ˙ e lambda abstraction | (e → e, e) conditional | b boolean constant | e e application | x˙ local variable | ⊕ operations
The set of processor names N is a set of p closed local terms in normal form. In the following will we consider that N = {0, . . . , p − 1}. The “flat” global terms (denoted by uppercase characters) represent parallel values. They are given by the grammar: Ef ::= x ¯ | Ef Ef | Ef e | λ¯ x. Ef | λx. ˙ E f | π e | E f # E f | Ef ? E f
π e is the intentional parallel vector which contains the local term e i at processor i. In the BSλp -calculus, the number of processors is constant. In this case, π e is syntactic sugar for e 0 , . . . , e (p − 1) , where the width of the vector is p. In the pratical programming language we will no longer use the enumerated form of parallel vectors. Nevertheless in the evaluation semantics, we will use enumerated vectors as values. Their widths can range from 1 to p. E1 # E2 is the point-wise parallel application. E1 must be a vector of functions and E2 a vector of values. On each process, the function is applied to the corresponding value. E1 ? E2 is the get operation used to communicate values between processors. E2 is a parallel vector of processor names. If at processor i the value held by E2 is j then the result of the get operation (a parallel vector) will contain the value held by parallel vector E1 at processor j. The execution of this operation corresponds to a communication and synchronization phase of a BSP super-step. With the get operation, a processor can only obtain data from one other processor. There exists a more general communication operation put, which allows any BSP communication schema using only one super-step. With this operation a processor can send at most p messages to the p processors of the parallel machine, each messages can have a different content. The put operation is of course a bit more complicated and its interaction with parallel juxtaposition is the same than the get operation. For the sake of simplicity we will only consider the get operation in this section. Terms with parallel juxtaposition. E1 m E2 is the parallel juxtaposition, which means that the m first processors will evaluate E1 and the others will evaluate E2 . From the point of view of E1 the network will have m processors named 0, . . . , m − 1. From the point of view of E2 the network will have p − m processors (where p is the number of processors of the current network) named 0, . . . , (p − m − 1) (processor m is renamed 0, etc.). We want to preserve the flat BSP super-step structure and avoid subset synchronization [6]. The proposition of this paper relies on the sync operation. Consider the term parjux m E1 E2 . The evaluation of E1 and E2 will be as described below for flat terms but respectively on a network of m and p − m processors except that the synchronization barriers of put and if at operations will concern the whole network. If for example E2 has less super-steps and thus less synchronization barriers than E1 then we need additional, useless in the point of view of E2 ,
Parallel Juxtaposition for Bulk Synchronous Parallel ML
785
synchronization barriers to preserve the BSP execution model. This is why we have added the sync operation. Each operation using synchronization barrier labels its calls to the synchronization barrier. With this label each processor knows what kind of operation the other processors have called to end the superstep. The sync operation repeatedly requests synchronization barriers. When each processor requests a synchronization barrier labeled by sync, it means all the processors reached the sync and the evaluation of sync ends. The sync operation introduce an additional synchronization barrier, but it allows a more flexible use of parallel juxtaposition. The complete syntax for global terms E is E ::= Ef | sync(Es ) where Es ::= x ¯ | Es Es | Es e | λ¯ x. Es | λx. ˙ Es | Es # Es | Es ? Es | π e | Es e Es
It is important to notice that this syntax does not allow nesting of parallel objects: a local term cannot contain a global term. Thus a parallel vector can not contain a parallel vector. Semantics. The semantics starts from a term, and returns a value. The local values are the constants and the local λ-abstractions. The global values are the enumerated vectors of local values and the λ-abstractions. The evaluation semantics for local terms e loc v is a classical call-by-value evaluation (omitted here for the sake of conciseness). The evaluation of global terms with parallel juxtaposition s uses an environment for global variables: a parallel vector can be bound to a variable in a subnetwork with p processors and this variable can be used in a subnetwork with p processors (p ≤ p). In this case we are only interest in p of the p values of the vector. So the global evaluation depends on the number of processors in the current subnetwork and the name (in the whole network) of the first processor of this subnetwork. Judgments of this semantics have the form Es pf E s V which means “in global environment Es on a subnetwork with p processors whose first processor has name f in the whole network, the term Es evaluates to value V ”. The environment for the evaluation of global terms with parallel juxtaposition Es can be seen as an association list which associates global variables to pairs composed of an integer and a global value. In the following we will use the notation [v0 ; . . . ; vn ] for lists and in particular environments. We will also note E(¯ x) for the first value associated with the global variable x ¯ in E. The two first rules are used to retrieve a value in the global environment. It is to notice that there are two different rules: one for parallel vectors and one for other values, i.e. global λ-abstractions. In this environment, a variable is not only bound to its value but to a pair whose first component is the name of the first processor of the network on which the value was produced. This information is only useful for the first rule. This rule says that if a parallel vector was bound to a variable in a subnetwork with p1 processors and with first processor f1 , then this variable can be used in a subnetwork with p2 processors (p2 ≤ p1 ) and with first processor f2 (f2 ≥ f1 ). In this case we are only interested by p2 components of the vector, starting at component numbered f2 − f1 :
786
F. Loulergue x) = (f1 , v0 , . . . , vp1 −1 ) Es (¯ Es pf22 x ¯ s vf2 −f1 , . . . , vf2 −f1 +p2 −1
x) = (f1 , λx.Es ) Es (¯ Es pf22 x ¯ s λx.Es
The next rule says that to evaluate an intentional parallel vector term π f , one has to evaluate the expression f i on each processor i of the subnetwork. This evaluation corresponds to a pure computation phase of a BSP super-step: ∀i ∈ 0 . . . p − 1, e i loc vi
Es pf π e s v0 , . . . , vp −1
The three next rules are for application and point-wise parallel application. Observe that in the rule for application, the pair (f, V2 ), rather than a single value, is bound to the variable x ¯:
Es pf Es2 s V2 (x ¯ → (f, V2 ) ) :: Es pf Es1 s V1
e loc v Es pf Es [x ← v] s V
Es pf (λ¯ x.Es1 ) Es2 s V1
Es pf (λx.E ˙ s ) e s V
Es pf Es1 s λx.e0 , . . . , λx.ep −1 Es pf Es2 s v0 , . . . , vp −1 ∀i ∈ 0 . . . p − 1, (λx.ei )vi loc wi
Es pf Es1 # Es2 s w0 , . . . , wp −1
The next rule is for the operation that need communications:
Es pf Es1 s v0 , . . . , vp −1
Es pf Es2 s n0 , . . . , np −1
Es pf Es1 ? Es2 s vn0 , . . . , vnp −1
The last important rule is the one for parallel juxtaposition. The left side of parallel juxtaposition is evaluated on a subnetwork with m processors, the right side is evaluated on the remaining subnetwork containing p − m processors.
−m 1 Es m Es fp +m Es2 s vm , . . . , vp −1 f Es s v0 , . . . , vm−1 e loc m and 0 < m < p Es pf Es1 e Es2 s v0 , . . . , vp −1
(1)
The remaining rule is a classical one: Es pf λx.Es s λx.Es , where x is either x˙ or x ¯. We now need to define the evaluation of global terms E (with parallel juxtaposition enclosed in a sync operation). The environment E for the evaluation of global terms E can be seen as an association list which associates global variables to global values. This evaluation has judgments of the form E E V which means “in global environment E on the whole network the term E evaluates to value V ”. The first rule is the rule used to evaluate a parallel composition in the scope of a sync: Es p0 E s V E sync(E) V
where Es (¯ x) = (0, V ) if and only if E(¯ x) = V . The remaining rules are similar to the previous ones except that the number of processes and the name of the first processor of the sub-network are not longer useful. Due to the lack of space we omit these rules.
Parallel Juxtaposition for Bulk Synchronous Parallel ML
4
787
Example
Objective Caml is an eager language. To express parallel juxtaposition as a function we have to “freeze” the evaluation of its parallel arguments. Thus parallel juxtaposition must have the following type: parjux: int -> (unit->’a par) -> (unit->’a par) -> ’a par The sync operation has in the BSMLlib library the type ’a -> ’a even if it is only useful for global terms. The following example is a divide-and-conquer version of the scan program which is defined by scan ⊕ v0 , . . . , vp−1 = v0 , . . . , v0 ⊕ v1 ⊕ . . . ⊕ vp−1 : let rec scan op vec = if bsp_p()=1 then vec else let mid = bsp_p()/2 in let vec’=parjux mid (fun ()->scan op vec) (fun ()->scan op vec) in let msg vec=apply(mkpar(fun i v-> if i=mid-1 then fun dst-> if dst>=mid then Some v else None else fun dst-> None)) vec and parop=parfun2(fun x y->match x with None->y|Some v->op v y) in parop (apply(put(msg vec’))(mkpar(fun i->mid-1))) vec’
The network is divided into two parts and the scan is recursively applied to those two parts. The value held by the last processor of the first part is broadcast to all the processors of the second part, then this value and the value held locally are combined together by the operator op on each processor of the second part. To use this function at top-level, it must be put into a sync operation. For example : let this = mkpar (fun pid->pid) in sync (scan (+) this).
5
Related Work
[14] presents another way to divide-and-conquer in the framework of an objectoriented language. There is no formal semantics and no implementation from now on. Furthermore, the proposed operation is rather a parallel superposition, several BSP threads use the whole network, than a parallel juxtaposition. The same author advocates in [9] a new extension of the BSP model in order to ease the programming of divide-and-conquer BSP algorithms. It adds another level to the BSP model with new parameters to describe the parallel machine. [15] is an algorithmic skeletons language based on the BSP model and offers divide-and-conquer skeletons. Nevertheless, the cost model is not really the BSP model but the D-BSP model [2] which allows subset synchronization. We follow [6] to reject such a possibility.
6
Conclusions and Future Work
Compared to a previous attempt [7], the parallel juxtaposition has not the two main drawbacks of its predecessor : (a) the two sides of spatial parallel composi-
788
F. Loulergue
tion may not have the same number of synchronization barriers. This enhancement comes to the price of an additional synchronization barrier, but only one barrier even in the case of nested parallel compositions ; (b) the cost model is a compositional one. The remaining drawback is that the semantics is strategydependant. The use of the spatial parallel composition changes the number of processors: it is an imperative feature. Thus there is no hope in this way to obtain a confluent calculus with a parallel juxtaposition. The next released implementation of the BSMLlib library will include parallel juxtaposition. Its ease of use will be experimented by implementing BSP algorithms described as divide-and-conquer algorithms in the literature.
References 1. M. Cole. Algorithmic Skeletons: Structured Management of Parallel Computation. MIT Press, 1989. 2. P. de la Torre and C. P. Kruskal. Submachine locality in the bulk synchronous setting. In Euro-Par’96, number 1123–1124. Springer Verlag, 1996. 3. C. Foisy and E. Chailloux. Caml Flight: a portable SPMD extension of ML for distributed memory multiprocessors. In A. W. B¨ ohm and J. T. Feo, editors, Workshop on High Performance Functionnal Computing, 1995. 4. F. Gava and F. Loulergue. A Polymorphic Type System for Bulk Synchronous Parallel ML. In Seventh International Conference on Parallel Computing Technologies (PaCT 2003), LNCS. Springer Verlag, 2003. to appear. 5. A. V. Gerbessiotis and L. G. Valiant. Direct Bulk-Synchronous Parallel Algorithms. Journal of Parallel and Distributed Computing, 22:251–267, 1994. 6. G. Hains. Subset synchronization in BSP computing. In H.R.Arabnia, editor, PDPTA’98, pages 242–246. CSREA Press, 1998. 7. F. Loulergue. Parallel Composition and Bulk Synchronous Parallel Functional Programming. In S. Gilmore, editor, Trends in Functional Programming, Volume 2, pages 77–88. Intellect Books, 2001. 8. F. Loulergue, G. Hains, and C. Foisy. A Calculus of Functional BSP Programs. Science of Computer Programming, 37(1-3):253–277, 2000. 9. J. Martin and A. Tiskin. BSP Algorithms Design for Hierarchical Supercomputers. submitted for publication, 2002. 10. P. Panangaden and J. Reppy. The essence of concurrent ML. In F. Nielson, editor, ML with Concurrency, Monographs in Computer Science. Springer, 1996. 11. S. Pelagatti. Structured Development of Parallel Programs. Taylor & Francis, 1998. 12. D. B. Skillicorn, J. M. D. Hill, and W. F. McColl. Questions and Answers about BSP. Scientific Programming, 6(3):249–274, 1997. 13. A. Tiskin. The Design and Analysis of Bulk-Synchronous Parallel Algorithms. PhD thesis, Oxford University Computing Laboratory, 1998. 14. A. Tiskin. A New Way to Divide and Conquer. Parallel Processing Letters, (4), 2001. 15. A. Zavanella. Skeletons and BSP : Performance Portability for Parallel Programming. PhD thesis, Universita degli studi di Pisa, 1999.
Parallelization with Tree Skeletons Kiminori Matsuzaki1 , Zhenjiang Hu1,2 , and Masato Takeichi1 1
2
Graduate School of Information Science and Technology, University of Tokyo
[email protected] {hu,takeichi}@mist.i.u-tokyo.ac.jp PRESTO21, Japan Science and Technology Corporation.
Abstract. Trees are useful data structures, but to design efficient parallel programs over trees is known to be more difficult than to do over lists. Although several important tree skeletons have been proposed to simplify parallel programming on trees, few studies have been reported on how to systematically use them in solving practical problems; it is neither clear how to make a good combination of skeletons to solve a given problem, nor obvious even how to find suitable operators used in a single skeleton. In this paper, we report our first attempt to resolve these problems, proposing two important transformations, the tree diffusion transformation and the tree context preservation transformation. The tree diffusion transformation allows one to use familiar recursive definitions to develop his parallel programs, while the tree context preservation transformation shows how to derive associative operators that are required when using tree skeletons. We illustrate our approach by deriving an efficient parallel program for solving a nontrivial problem called the party planning problem, the tree version of the famous maximum-weight-sum problem. Keywords: Parallel Skeletons, Tree Algorithms, Parallelization, Program Transformation, Algorithm Derivation.
1
Introduction
Trees are useful data types, widely used for representing hierarchical structures such as mathematical expressions or structured documents like XML. Due to irregularity (imbalance) of tree structures, developing efficient parallel programs manipulating trees is much more difficult than developing efficient parallel programs manipulating lists. Although several important tree skeletons have been proposed to simplify parallel programming on trees [4,5,13], few studies have been reported on how to systematically use them in solving practical problems. Although many researchers have devoted themselves to constructing systematic parallel programming methodology using list skeletons [1,2,6,8], few have reported the methodology with tree skeletons. Unlike linear structure of lists, trees do not have a linear structure, and hence the recursive functions over trees are not linear either (in the sense that there are more than one recursive call in H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 789–798, 2003. c Springer-Verlag Berlin Heidelberg 2003
790
K. Matsuzaki, Z. Hu, and M. Takeichi
the definition body). It is this nonlinearity that makes the parallel programming on trees complex and difficult to solve. In this paper, we aim at a systematic method for parallel programming using tree skeletons, by proposing two important transformations, the tree diffusion transformation and the tree context preservation transformation. – The tree diffusion transformation is an extension of the list version [8]. It shows how to decompose familiar recursive programs into equivalent parallel ones in terms of tree skeletons. – The tree context preservation transformation is an extension of the list version [1]. It shows how to derive associative operators that are required when using tree skeletons. In addition, to show the usefulness of these theorems, we demonstrate a derivation of an efficient parallel program for solving the party planning problem, using tree skeletons defined in Section 2. The party planning problem is an interesting tree version of the well-known maximum-weight-sum problem [2], which appeared as an exercise in [3]. Professor Stewart is consulting for the president of a corporation that is planning a company party. The company has a hierarchical tree structure; that is, the supervisor relation forms a tree rooted at the president. The personnel office has ranked each employee with a conviviality rating, which is a real number. In order to make the party fun for all attendees, the president does not want both an employee and his or her immediate supervisor to attend. The problem is to design an algorithm making the guest list, and the goal is to maximize the sum of the conviviality rating of the guest. It is not easy to decide which tree skeletons to use and how to combine them properly so as to solve this problem. Moreover, skeletons impose restriction (such as associativity) on the functions and operations, and it is not straightforward to find such ones. The rest of the paper is as follows. After reviewing the tree skeletons in Section 2, we explain our two parallelization transformations for trees: the diffusion transformation in Section 3, and the context preservation transformation in Section 4. We show the experimental results in Section 5, and give conclusion in Section 6.
2
Parallel Skeletons on Trees
To simplify our presentation, we consider binary trees in this paper. The primitive parallel skeletons on binary trees are map, zip, reduce, upwards accumulate and downwards accumulate [13,14], and their formal definitions using the notation of the Haskell language [9] are described in Fig 1. We will use the Haskell notation for the rest of this paper.
Parallelization with Tree Skeletons
791
data BTree α β = Leaf α | Node (BTree α β) β (BTree α β) map :: (α → γ, β → δ) → BTree α β → BTree γ δ = Leaf (fL n) map (fL , fN ) (Leaf n) map (fL , fN ) (Node l n r) = Node (map (fL , fN ) l) (fN n) (map (fL , fN ) r) zip :: BTree α β → BTree γ δ → BTree (α, γ) (β, δ) zip (Leaf n) (Leaf n ) = Leaf (n, n ) zip (Node l n r) (Node l n r ) = Node (zip l l ) (n, n ) (zip r r ) reduce :: (α → γ, γ → β → γ → γ) → BTree α β → γ reduce (fL , fN ) (Leaf n) = fL n reduce (fL , fN ) (Node l n r) = fN (reduce (fL , fN ) l) n (reduce (fL , fN ) r) uAcc :: (α → γ, γ → β → γ → γ) → BTree α β → BTree γ γ uAcc (fL , fN ) (Leaf n) = Leaf (fL n) uAcc (fL , fN ) (Node l n r) = let l = uAcc (fL , fN ) l r = uAcc (fL , fN ) r in Node l (fN (root l ) n (root r )) r dAcc :: (γ → γ → γ) → (β → γ, β → γ) → BTree α β → γ → BTree γ γ dAcc (⊕) (fL , fR ) (Leaf n) c = Leaf c dAcc (⊕) (fL , fR ) (Node l n r) c = Node (dAcc (⊕) (fL , fR ) l (c ⊕ fL n)) c (dAcc (⊕) (fL , fR ) r (c ⊕ fR n)) Fig. 1. Definitions of five primitive skeletons
The map skeleton map (fL , fN ) applies function fL to each leaf and function fN to each internal node. The zip skeleton accepts two trees of the same shape and returns a tree whose nodes are pairs of corresponding two nodes of the original two trees. The reduce skeleton reduce (fL , fN ) reduces a tree into a value by applying fL to each leaf, and fN to each internal node upwards. Similar to reduce, the upwards accumulate skeleton uAcc (fL , fN ) applies fL to each leaf and fN to each internal node in a bottom-up manner, and returns a tree of the same shape as the original tree. The downwards accumulate skeleton dAcc (⊕) (fL , fR ) c computes by propagating accumulation parameter c downwards, and the accumulation parameter is updated by ⊕ and fL when propagated to left child, or updated by ⊕ and fR when propagated to right child. To guarantee the existence of efficient implementation for the parallel skeletons, we have requirement on the operators and functions used in the above skeletons. Definition 1 (Semi-Associative). A binary operator ⊗ is said to be semiassociative if there is an associative operator ⊕ such that for any a, b, c, (a ⊗ b) ⊗ c = a ⊗ (b ⊕ c). 2 Definition 2 (Quasi-Associative). A binary operator ⊕ is said to be quasiassociative if there is a semi-associative operator ⊗ and a function f such that for any a, b, a ⊕ b = a ⊗ f b. 2
792
K. Matsuzaki, Z. Hu, and M. Takeichi
Definition 3 (Bi-Quasi-Associative). A ternary operator f is said to be biquasi-associative if there is a semi-associative operator ⊗ and two functions fL , fR such that for any l, n, r, f l n r = l ⊗ fL n r = r ⊗ fR n l. We can fix a bi-quasi-associative operator f by providing ⊗, ⊕ (associative operator for ⊗), 2 fL and fR , therefore, we will write f with 4-tuple as f ≡ [[⊗, ⊕, fL , fR ]]. Based on the tree contraction technique [12], we require the fN used in the reduce and upwards accumulate be bi-quasi-associative, and ⊕ in downwards accumulate be associative. We omit the detailed description of the cost for each skeleton. Informally, if all the operators used in the skeletons use constant time, all skeletons can be implemented in at most O(log N ) parallel time using enough processors, where N denotes the number of nodes in the tree.
3
Tree Diffusion Theorem
Hu et al. proposed the diffusion theorem (on lists) [8], with which we can directly derive efficient combinations of skeletons from recursive programs. In this section, we start by formalizing a very general tree diffusion theorem, then discuss three practical cases, and finally derive a combination of skeletons for the party planning problem. Theorem 1 (Tree Diffusion). Let f be defined in the following recursive way over binary trees: f (Leaf n) c = gL (n, c) f (Node l n r) c = gN (f l (c ⊗ hL n)) (n, c) (f r (c ⊗ hR n)) where gN is a bi-quasi-associative operator, ⊗ is an associative operator, and gL , hL , hR are user-defined functions. Then f can be equivalently defined in terms of the tree skeletons as follows. f xt c = let ct = dAcc (⊗) (hL , hR ) xt c in reduce (gL , gN ) (zip xt ct) Proof Sketch: This can be proved by induction on the structure of xt. Due to the limitation of space, the proof is given in the technical report [11]. 2 This theorem is very general. Practically, It is often the case that the function f returns a tree with the same shape as the input. If we naively apply this diffusion theorem, we will have a costly reduce skeleton for combining all sub-trees. To remedy this situation, we propose the following two useful specializations, in which we use appropriate skeletons rather than reduce. The first specialization deals with the function whose computation of new values for each node depends on the original value and the accumulation parameter. For each internal node, such a function f can be defined as f (Node l n r) = Node (f l (c ⊗ hL n)) (gN (n, c)) (f r (c ⊗ hR n)), and this function can be efficiently computed by map rather than reduce.
Parallelization with Tree Skeletons
793
ppp xt = ppp xt True ppp (Leaf n) c = Leaf c ppp (Node l n r) c = let (lm , lu ) = mis l (rm , ru ) = mis r in Node (ppp l (if c then False else (lm > lu ))) c (ppp r (if c then False else (rm > ru ))) mis (Leaf n) = (n, 0) mis (Node l n r) = let (lm , lu ) = mis l (rm , ru ) = mis r in (lu + n + ru , (lm ↑ lu ) + (rm ↑ ru )) Fig. 2. A sequential program for party planning program
The second specialization deals with the function whose computation of new values for each node depends on the original value, the accumulation parameter and the new value of its children. For each internal node, such a function f can be defined as f (Node l n r) c = Node l (gN (root l ) (n, c) (root r )) r where l = f l (c ⊗ hL n) and r = f l (c ⊗ hR n). This function can be efficiently computed by upwards accumulate rather than reduce. Let us discuss another practical matter for the case where the function f calls an auxiliary function k to compute over the sub-trees. Such a function can be defined as follows. f (Leaf n) c = Leaf (gL (( , n, ), c)) f (Node n l r) c = let n = (k l, n, k r) in Node (f l (c ⊗ hL n )) (gN (n , c)) (f r (c ⊗ hR n )) k (Leaf n) = kL n k (Node l n r) = kN (k l) n (k r) It is a little difficult to efficiently parallelize this recursive function into the combination of primitive skeletons, because there are multiple traversals over the trees, and naive computation of f will make redundant function calls of k. By making use of the tupling transformation and the fusion transformation [7], we can parallelize the function efficiently. In the following, we use a function gather ch, which accepts two trees of the same shape and makes a triple for each node. The triple consists of a node of the first tree and two immediate children of the second tree. Detailed discussions are referred to [11]. Corollary 1 (Paramorphic Diffusion). The function f defined above can be diffused into the following combination of skeletons if kN is a bi-quasi-associative operator, and ⊗ is associative. f xt c = let yt = gather ch xt (uAcc (kL , kN ) xt) in dAcc (⊗) (hL , hR ) yt c
2
Having shown the diffusion theorem and its corollaries, we now try to derive a parallel program for the party planning problem. By making use of dynamic
794
K. Matsuzaki, Z. Hu, and M. Takeichi
programming technique, we can obtain an efficient sequential program as shown in Fig 2. Here, the function mis accepts a tree, and returns a pair of values which are the maximum independent sums when the root of the input is marked or unmarked. The recursive function ppp is defined with an accumulation parameter, which represents a node to be marked or unmarked. The recursive function ppp is a paramorphic function because it calls an auxiliary function mis on each sub-tree, therefore, let us use paramorphic diffusion theorem to derive the following program in terms of skeletons. ppp xt = ppp xt True ppp xt c = let yt = gather ch xt (uAcc (mis L , mis N ) xt) in dAcc (⊗) (hL , hR ) yt c However, we have not yet parallelized the underlined parts successfully. First, from the definition of the sequential program, we can derive mis L n = (n, 0) and mis N (lm , lu ) n (rm , uu ) = (lu + n + ru , (lm ↑ lu ) + (rm ↑ ru )), however, we have still to show the bi-quasi-associativity of mis N . Second, we have to derive an associative operator ⊗ and two functions hL and hR such that c ⊗ hL ((lm , lu ), n, (rm , ru )) = if c then False else (lm > lu ) and almost the same equation for hR hold. In the following section, we will see how to derive those operators.
4
Tree Context Preservation
The parallel skeletons require the operators used in them to be (bi-quasi)associative, however, it is not straightforward to find such ones for many practical problems. For linear self-recursive programs, Chin et al. proposed the context preservation transformation [1], with which one can systematically derive such operators based on the associativity of function composition. In this section, we will extend the transformation theorem for tree skeletons. Our main idea is to resolve the non-linear functions over trees into two linear recursive functions, and then we can consider the context preservation on these two linear functions. We start by introducing the basic notations and concepts about contexts. Definition 4 (Context Extraction [1]). Given an expression E and subterms e1 , . . . , en , we shall express its extraction by: E =⇒ E e1 , . . . , en . The context E has a form of λ 1 , . . . , n .[ei → ––i ]ni=1 E, where ––i denotes a new –– –– n hole and [ei → ––i ]i=1 E denotes a substitution notation of ei in E to ––i . 2 Definition 5 (Skeletal Context [1]). A context E is said to be a skeletal context if every sub-term in E contains at least one hole. Given a context E, we can make it into a skeletal one ES by extracting all sub-terms that do not 2 contain holes. This process shall be denoted by E =⇒S ES ei i∈N Definition 6 (Context Transformation [1]). A context may be transformed (or simplified) by either applying laws or unfolding. We shall denote this process 2 as E =⇒T E .
Parallelization with Tree Skeletons
795
Definition 7 (Context Preservation Modulo Replication [1]). A context E with one hole is said to be preserved modulo replication if there is a skeletal context ES , E =⇒S ES ti and ES αi ◦ ES βi = ES γi hold, where αi and 2 βi are variables, and γi are sub-terms without holes. Now, we will discuss about the functions which can be transformed into a program with reduce. We can show the case of uAcc in the same manner, however, due to the limitation of space, we give it in the technical report [11]. Definition 8 (Simple Upwards Recursive Function). A function is said to be a simple upwards recursive function (SUR-function for short) if it has the following form. f (Leaf n) = fL n 2 f (N ode l n r) = fN (f l) n (f r) The inductive case of an SUR-function has two recursive calls, f l and f r, therefore, we cannot apply the Chin’s theorem. To resolve this non-linearity, we define the extraction of two linear recurring contexts from an SUR-function, and extended context preservation for these two contexts as shown in the following. Definition 9 (Left(Right)-Recurring Context). For the inductive case of an SUR-function, we can extract the left(right)-recurring context E L (E R ) by abstracting either of the recurring terms: f (N ode l n r) = E L f l = E R f r. 2 Definition 10 (Mutually Preserved Contexts). Two linear recurring contexts E L , E R are said to be mutually preserved if there exists a skeletal context ES such that E L =⇒S ES g l n r, E R =⇒S ES g r n l and ES α ◦ ES β = ES γ hold. Here, γ is a sub-terms computed only with variables α and β. 2 Based on the idea of tree contraction algorithm, we can parallelize the SURfunction as shown in the following theorem. Due to the limitation of space we omit the proof, which is given in the technical report [11]. Theorem 2 (Context Preservation for SUR-function). The SUR-function function f can be parallelized to f = reduce (fL , fN ) if there exist a skeletal context ES such that E L =⇒S ES g l n r, E R =⇒S ES g r n l and ES α ◦ ES β = ES γ hold. Here, fN is a bi-quasi-associative operator such as fN ≡ [[⊕, ⊗, g l , g r ]] where x ⊕ α = ES αx and β ⊗ α = γ. 2 Next, we discuss about the functions which can be transformed into a program with dAcc. As is the case of reduce, based on the tree contraction algorithm, we can parallelize a non-linear function by extracting two linear contexts and showing these contexts to be mutually preserved. Due to the limitation of space, we only show the definitions and theorem for this. Definition 11 (Simple Downwards Recursive Function). A function is said to be a simple downwards recursive function (SDR-function for short) if it has the following form. f (Leaf n) c = Leaf c f (Node l n r) c = Node (f l (fL c n)) c (f r (fR c n))
2
796
K. Matsuzaki, Z. Hu, and M. Takeichi
Definition 12 (Recurring Contexts for SDR-function). For the inductive case of an SDR-function f , we can obtain two recurring contexts DL , DR by abstracting the recursive calls on the accumulative parameter respectively, 2 f (Node l n r) c = Node (f l DL c) c (f r DR c). Theorem 3 (Context Preservation for SDR-function). The SDR-function f can be parallelized to f xt c = map ((c⊗), (c⊗)) (dAcc (⊕) (g l , g r ) ι⊕ ) if there exist a skeletal context ES such that DL =⇒S DS g l n, DR =⇒S DS g r n and DS α ◦ DS β = DS γ hold. Here, the operators are defined as β ⊕ α = γ and c ⊗ α = DS αc, and ι⊕ is the unit of ⊕. 2 Having shown the context preservation theorems for trees, we now demonstrate how these theorems work by deriving an associative operator ⊗ and functions hR , hL in the diffused program in Section 3. The corresponding part is defined recursively as follows. ppp (Node l ((lm , lu ), n, (rm , ru )) r) c = Node (ppp l (if c then False else (lm > lu ))) c (ppp r (if c then False else (rm > ru ))) From this definition, we can obtain the following two linear recurring contexts by abstracting recursive calls. DL = λc.if c then False else (lm > lu ) DR = λc.if c then False else (rm > ru ) We can show that these two contexts are mutually recursive because the skeletal context DS = λ 1 , 2 .λc.if c then 1 else 2 satisfies our requirement. –– –– –– –– L l R D = DS g ((lm , lu ), n, (rm , ru )), D = DS g r ((lm , lu ), n, (rm , ru )) where g l ((lm , lu ), n, (rm , ru )) = (f alse, (lm > lu )) g r ((lm , lu ), n, (rm , ru )) = (f alse, (rm > ru )) DS α1 , α2 ◦ DS β1 , β2 = λc.if c then (if β1 then α1 else α2 ) else (if β2 then α1 else α2 ) = DS if β1 then α1 else α2 , if β2 then α1 else α2 From the derivations above, we can apply theorem 3 to obtain an efficient parallel program with map and downwards accumulate. The whole parallel program for the party planning problem is shown in Fig 3. Detailed derivations are referred to [11].
5
An Experiment
We have conducted an experiment on the party planning problem. We have coded our algorithm using C++, the MPI library and our implementation of tree skeletons [10]. We have used a tree of 999,999 nodes for our experiment. Fig 4 shows the result of the program executed on our PC-Cluster using 1 to 12 processors. This result is shown in the speedup excluding partitioning and flattening of the tree. The almost linear speedup shows the effectiveness of the program derived by our theorems.
Parallelization with Tree Skeletons
797
ppp xt = let yt = gather ch xt (uAcc (mis L , mis N ) xt) in map (fst, fst) (dAcc () (hL , hR ) yt ι ) where mis L = (n, 0) mis N ≡ [[⊕, ⊗, f L , f R ]] (β1 , β2 , β3 , β4 ) ⊕ (α1 , α2 , α3 , α4 ) = ((β1 + α1 ) ↑ (β3 + α2 ), (β2 + α1 ) ↑ (β4 + α2 ), (β1 + α3 ) ↑ (β3 + α4 ), (β2 + α3 ) ↑ (β4 + α4 )) (xm , xu ) ⊗ (α1 , α2 , α3 , α4 ) = ((xm + α1 ) ↑ (xu + α2 ), (xm + α3 ) ↑ (xu + α4 )) f L n (rm , ru ) = (−∞, n + ru , rm ↑ ru , rm ↑ ru ) f R n (lm , lu ) = (−∞, n + lu , lm ↑ lu , lm ↑ lu ) (β1 , β2 ) (α1 , α2 ) = (if β1 then α1 else α2 , if β2 then α1 else α2 ) ι = (True, False) hL ((lm , lu ), n, (rm , ru )) = (False, (lm > lu )) hR ((lm , lu ), n, (rm , ru )) = (False, (rm > ru )) Fig. 3. Parallel program for party planning problem 5.5
result
5 4.5
Speedup
4 3.5 3 2.5 2 1.5 1 0
2
4 6 8 Number of Processors
10
12
Fig. 4. Experiment result
6
Conclusion
In this paper, we have proposed two parallelization transformations, the tree diffusion transformation and the context preservation transformation, for helping programmers to systematically derive efficient parallel programs in terms of tree skeletons from the recursive programs. The list versions of these two theorems have been proposed and shown important in skeletal parallel programming, which once in fact motivated us to see if we could generalize them for trees. Due to the non-linearity of the tree structures, it turns out to be more difficult than we had expected. Although the usefulness of our theorems await more evidence, our successful derivation of the first skeletal parallel program for solving the party planning problem and the good experiment result have indicated that this is a good start and is worth further investigation.
798
K. Matsuzaki, Z. Hu, and M. Takeichi
We are currently working on generalizing the context preservation theorem so that we can relax conditions of the skeletons. In addition, we are figuring out whether we can automatically parallelize the recursive programs on trees.
References 1. W.N. Chin, A. Takano, and Z. Hu. Parallelization via context preservation. IEEE Computer Society International Conference on Computer Languages (ICCL’98), pages 153–162, May 1998. 2. M. Cole. Parallel programming, list homomorphisms and the maximum segment sum problems. Report CSR-25-93, Department of Computing Science, The University of Edinburgh, May 1993. 3. T. H. Cormen, C. E. Leiserson, R. L. Rivest, and C. Stein. Introduction to Algorithms. MIT Press, second edition, 2001. 4. J. Gibbons. Algebras for Tree Algorithms. PhD thesis, Programming Research Group, Oxford University, 1991. Available as Technical Monograph PRG-94. 5. J. Gibbons. Computing downwards accumulations on trees quickly. In G. Gupta, G. Mohay, and R. Topor, editors, Proceedings of 16th Australian Computer Science Conference, volume 15 (1), pages 685–691. Australian Computer Science Communications, February 1993. 6. S. Gorlatch. Systematic efficient parallelization of scan and other list homomorphisms. In Annual European Conference on Parallel Processing, LNCS 1124, pages 401–408, LIP, ENS Lyon, France, August 1996. Springer-Verlag. 7. Z. Hu, H. Iwasaki, and M. Takeichi. Construction of list homomorphisms by tupling and fusion. In 21st International Symposium on Mathematical Foundation of Computer Science, LNCS 1113, pages 407–418, Cracow, September 1996. SpringerVerlag. 8. Z. Hu, M. Takeichi, and H. Iwasaki. Diffusion: Calculating efficient parallel programs. In 1999 ACM SIGPLAN Workshop on Partial Evaluation and SemanticsBased Program Manipulation (PEPM ’99), pages 85–94, San Antonio, Texas, January 1999. BRICS Notes Series NS-99-1. 9. S. Peyton Jones and J. Hughes, editors. Haskell 98: A Non-strict, Purely Functional Language. Available online: http://www.haskell.org, February 1999. 10. K. Matsuzaki, Z. Hu, and M. Takeichi. Implementation of parallel tree skeletons on distributed systems. In Proceedings of The Third Asian Workshop on Programming Languages And Systems, pages 258–271, Shanghai, China, 2002. 11. K. Matsuzaki, Z. Hu, and M. Takeichi. Parallelization with tree skeletons. Technical Report METR 2003-21, Mathematical Informatics, Graduate School of Information Science and Technology, University of Tokyo, 2003. 12. M. Reid-Miller, G. L. Miller, and F. Modugno. List ranking and parallel tree contraction. In John H. Reif, editor, Synthesis of Parallel Algorithms, chapter 3, pages 115–194. Morgan Kaufmann Publishers, 1996. 13. D. B. Skillicorn. Foundations of Parallel Programming. Cambridge University Press, 1994. 14. D. B. Skillicorn. Parallel implementation of tree skeletons. Journal of Parallel and Distributed Computing, 39(2):115–125, 1996.
Topic 11 Numerical Algorithms and Scientific Engineering Problems Iain S. Duff, Luc Giraud, Henk A. van der Vorst, and Peter Zinterhof Topic Chairs
Following the traditions of previous Euro-Par Conferences, Euro-Par 2003 includes the topic “Numerical Algorithms”. This topic has been scheduled this year for a moring and afternoon sessions on 2003, 29 August. Current research and its applications in nearly all areas of natural sciences involves, with increasing importance, mathematical modelling and numerical simulation. Indeed, such a basis for computation has recently been extended to other disciplines like engineering and economics. At the same time, there is a continuously increasing number of requests to solve problems with growing complexity. It is thus nearly superfluous to say that the efficient design of parallel numerical algorithms becomes more and more essential. As for previous Euro-Par Conferences, we have accepted only papers in which fundamental numerical algorithms are addressed and which are therefore of a wide range of interest, including parallel numerical algebra with new developments for Sylvester-type matrix equations and the one-sided Jacobi method, parallel work in solving ODEs and PDEs including applications to global ocean circulation. Out of 14 submitted papers, we accepted only 5 as regular presentations and 2 as short. According to this selection, we have divided the presentations into two sessions. One is mainly devoted to methods of parallel linear algebra and the second to parallel methods for PDEs. We hope and we believe again that the sessions contain a highly interesting mix of parallel numerical algorithms. We would like to take this opportunity of thanking all contributing authors as well as all reviewers for their work.
H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, p. 799, 2003. c Springer-Verlag Berlin Heidelberg 2003
Parallel ScaLAPACK-Style Algorithms for Solving Continuous-Time Sylvester Matrix Equations Robert Granat, Bo K˚ agstr¨om, and Peter Poromaa Department of Computing Science and HPC2N, Ume˚ a University, SE-901 87 Ume˚ a, Sweden. {granat,bokg,peterp}@cs.umu.se
Abstract. An implementation of a parallel ScaLAPACK-style solver for the general Sylvester equation, op(A)X − Xop(B) = C, where op(A) denotes A or its transpose AT , is presented. The parallel algorithm is based on explicit blocking of the Bartels-Stewart method. An initial transformation of the coefficient matrices A and B to Schur form leads to a reduced triangular matrix equation. We use different matrix traversing strategies to handle the transposes in the problem to solve, leading to different new parallel wave-front algorithms. We also present a strategy to handle the problem when 2 x 2 diagonal blocks of the matrices in Schur form, corresponding to complex conjugate pairs of eigenvalues, are split between several blocks in the block partitioned matrices. Finally, the solution of the reduced matrix equation is transformed back to the originally coordinate system. The implementation acts in a ScaLAPACK environment using 2-dimensional block cyclic mapping of the matrices onto a rectangular grid of processes. Real performance results are presented which verify that our parallel algorithms are reliable and scalable. Keywords: Sylvester matrix equation, continuous-time, Bartels– Stewart method, blocking, GEMM-based, level 3 BLAS, SLICOT, ScaLAPACK-style algorithms.
1
Introduction
We present a parallel ScaLAPACK-style solver for the Sylvester equation (SYCT) op(A)X − Xop(B) = C, (1) where op(A) denotes A or its transpose AT . Here A of size M × M , B of size N × N and C of size M × N are arbitrary matrices with real entries. Equation (1) has a unique solution X of size M × N if and only if op(A) and op(B) have disjoint spectra. The Sylvester equation appears naturally in several applications. Examples include block-diagonalizing of a matrix in Schur form and condition estimation of eigenvalue problems (e.g., see [15,10,16]). Our method for solving SYCT (1) is based on the Bartels–Stewart method [1]: H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 800–809, 2003. c Springer-Verlag Berlin Heidelberg 2003
Parallel ScaLAPACK-Style Algorithms
801
1. Transform A and B to upper (quasi)triangular form TA and TB , respectively, using orthogonal similarity transformations: QT AQ = TA ,
P T BP = TB .
2. Update the matrix C with respect to the transformations done on A and B: = QT CP. C 3. Solve the reduced (quasi)triangular matrix equation: − Xop(T op(TA )X B ) = C. back to the original coordinate system: 4. Transform the solution X T. X = QXP The quasitriangular form mentioned in Step 1 is also called the real Schur form, which means that the matrix is upper block triangular with 1 × 1 and 2 × 2 diagonal blocks, corresponding to real and complex conjugate pairs of eigenvalues, respectively. To carry out Step 1 we use the QR-algorithm [2]. The updates in Step 2 and the back-transformation in Step 4 are carried out using ordinary GEMM-operations C ← βC + αop(A)op(B), where α and β are scalars [5,13,14]. Our focus is on Step 3. Using the Kronecker product notation, ⊗ , we can rewrite the triangular Sylvester equation as a linear system of equations Zx = y,
(2)
where Z = IN ⊗op(A)−op(B)T ⊗IM is a matrix of size M N ×M N , x = vec(X) and y = vec(C). As usual, vec(X) denotes an ordered stack of the columns of the matrix X from left to right starting with the first column. The linear system (2) can be solved to the cost of O(M 3 N 3 ) using ordinary LU factorization with pivoting. This is a very expensive operation, even for moderate-sized problems. Since A and B are (quasi)triangular, the triangular Sylvester equation can indeed be solved to the cost O(M 2 N + M N 2 ) using a combined backward/forward substitution process [1]. In blocked algorithms, the explicit Kronecker matrix representation Zx = y is used in kernels for solving small-sized matrix equations (e.g., see [11,12,15]). The rest of the paper is organized as follows: In Section 2, we give a brief overview of blocked algorithms for solving the triangular SYCT equation. Section 3 is devoted to parallel algorithms focusing on the solution of the reduced triangular matrix equations. Finally, in Section 4, we present experimental results and discuss the performance of our general ScaLAPACK-style solver. Our parallel implementations mainly adopt to the ScaLAPACK software conventions [3]. The P processors (or virtual processes) are viewed as a rectangular processor grid Pr ×Pc , with Pr ≥ 1 processor rows and Pc ≥ 1 processor columns such that P = Pr · Pc . The data layout of dense matrices on a rectangular grid is assumed to be done by the two-dimensional (2D) block-cyclic distribution scheme.
802
2
R. Granat, B. K˚ agstr¨ om, and P. Poromaa
Blocked Algorithms
Blocking is a powerful tool in Numerical Linear Algebra to restructure wellknown standard algorithms in level 3 operations with the potential to reuse data already stored in cache or registers. This will make things faster and more efficient on one processor or in a shared memory environment. Blocking is also useful for parallelizing tasks in distributed memory environments. 2.1
The Non-transposed Case
We start by reviewing the serial block algorithm proposed in [15] for the nontransposed triangular Sylvester equation AX − XB = C.
(3)
Here A and B have already been transformed to real Schur form. Let MB and NB be the block sizes used in the partitioning of A and B, respectively. Then MB is the row-block size and NB is the column-block size of C and X (which overwrites C). Now, the number of diagonal blocks of A and B can be expressed as Da = M/MB and Db = N/NB, respectively. Then Equation (3) can be rewritten in block-partitioned form: Aii Xij − Xij Bjj = Cij − (
Da k=i+1
Aik Xkj −
j−1
Xik Bkj ),
(4)
k=1
where i = 1, 2, . . . , Da and j = 1, 2, . . . , Db . Based on this summation formula, a serial blocked algorithm can be formulated, see Figure 1.
for j=1, Db for i=Da , 1, -1 {Solve the (i, j)th subsystem} Aii Xij − Xij Bjj = Cij for k=1, i − 1 {Update block column j of C} Ckj = Ckj − Aki Xij end for k=j + 1, Db {Update block row i of C} Cik = Cik + Xij Bjk end end end Fig. 1. Block algorithm for solving AX − XB = C, A and B in upper Schur form.
Parallel ScaLAPACK-Style Algorithms
2.2
803
The Transposed Cases
The three other cases, namely, AX − XB T = C,
AT X − XB = C,
and AT X − XB T = C,
can be treated in the same way. Each of these matrix equations correspond to a summation formula based on the same block partitioning of the matrices: T = Cij − ( Aii Xij − Xij Bjj
Da
Aik Xkj −
k=i+1
ATii Xij − Xij Bjj = Cij − (
i−1
ATki Xkj −
k=1 T = Cij − ( ATii Xij − Xij Bjj
i−1 k=1
Db
T Xik Bjk ),
k=j+1 j−1
Xik Bkj ),
k=1
ATki Xkj −
Db
T Xik Bjk ).
k=j+1
For each of these summation formulas a serial block algorithm is formulated. In Figure 2, we present the one corresponding to the reduced triangular equation AT X − XB T = C.
for j= Db , 1, -1 for i=1, Da {Solve the (i, j)th subsystem} T = Cij ATii Xij − Xij Bjj for k=i + 1, Da {Update block column j of C} Ckj = Ckj − ATik Xij end for k=1, j − 1 {Update block row i of C} T Cik = Cik + Xij Bkj end end end Fig. 2. Block algorithm for solving AT X − XB T = C, A and B in upper Schur form.
Notice that each summation formula sets a starting point in the matrix C/X where we start to compute the solution. For example, while solving subsystems and updating C/X with respect to these subsolutions in Figure 1, we traverse the matrix C/X along its block diagonals from South-East to North-West (or vice versa). This “wavefront” starts in the South-West corner of C/X, as depicted in Figure 3, and moves in the North-Eastern direction. Along the way, each computed Xij will be used to update block-row i and block-column j of C.
804
R. Granat, B. K˚ agstr¨ om, and P. Poromaa
Fig. 3. Traversing the matrix C/X when solving AX − XB = C.
3
Parallel Block Algorithms
We assume that the matrices A, B and C are distributed using 2D block-cyclic mapping across a Pr × Pc processor grid. To carry out Steps 1, 2 and 4 of the Bartels–Stewart method in parallel we use the ScaLAPACK library-routines PDGEHRD, PDLAHQR and PDGEMM [3]. The first two routines are used in Step 1 to compute the Schur decompositions of A and B (reduction to upper Hessenberg form followed by the parallel QR algorithm [9,8]). PDGEMM is the parallel implementation of the level 3 BLAS DGEMM operation and is used in Steps 2 and 4 for doing the two-sided matrix multiply updates. To carry out Step 3 in parallel, we traverse the matrix C/X along its block diagonals, starting in the corner that is decided by the data dependencies. To be able to compute Xij for certain values of i and j, we need Aii and Bjj to be owned by the same process that owns Cij . We also need to have the blocks used in the updates in the right place at the right time. The situation is illustrated in Figure 4, where all the data lie on the right processors. This will however not be the general case. In general, we have to communicate for some blocks during the solves and updates. For example, while traversing C/X for the triangular AX − XB T = C, we solve the small subsystems T Aii Xij − Xij Bjj = Cij ,
(5)
associated with the current block diagonal in parallel. Then we do the GEMMupdates, Ckj = Ckj − Aki Xij , k = 1, . . . , j − 1 (6) T Cik = Cik + Xij Bkj , k = 1, . . . , i − 1, of block-row i and block-column j, which can also be done in parallel. In equation (6), the submatrix Xij has been computed in the preceeding step and broadcasted (see Figure 4) to the processors involved in the GEMM-updates. It can be
Parallel ScaLAPACK-Style Algorithms
805
C 1
2
1
2
1
2
1
3
4
3
4
3
4
3
cu
1
2
1
2
1
2
4
3
4
3
1
4
2
1
2
1
3
2
4
3
4
3
1
4
4 2 4 2
3
4 S
1
2
3
4 S
S = needed for solution of subsystem ru = needed for row update in C cu = needed for column update in C broadcast directions
2
1
2
3
4
3
4
3
4
1
2
1
2
1
2
3
4
3
4
3
4
1
2
1
2
1
2
3
4
3
4
cu
S
A
1
cu
cu
3
2
cu
cu
1
1
cu
cu
3
2 cu
cu
ru
ru
ru
ru
3
4
ru
ru
1
2
1
2
1
2
3
4
3
4
3
4
ru
ru
ru
ru
ru
ru
1
2
1
2
1
2
1
2
3
4
3
4
3
4
3
4
1
2
1
2
1
2
1
2
3
4
3
4
3
4
3
4
1
2
1
2
1
2
1
2
3
4
3
4
3
4
3
4
B
Fig. 4. Data dependencies and mapping for AX − XB = C on a 2 × 2 processor grid.
shown that this gives a theoretical limit for the speedup of the triangular solver as max(Pr , Pc ) for solving the subsystems, and as Pr · Pc for the GEMM-updates [7,16]. A high-level parallel block algorithm for the solving the general triangular SYCT equation (1) is presented in Figure 5. 3.1
The 2 × 2 Diagonal Block Split Problem
When entering the triangular solver we already have transformed A, B and C distributed across the processor grid (2D block cyclic mapping). Now, we have to assure that no 2 × 2 diagonal block of A or B in Schur form, corresponding to conjugate pairs of complex eigenvalues is being split between two blocks (processors). This happens when any element on the first subdiagonal of the real Schur form is not equal to zero and does not belong to the same block (submatrix) as the closest elements to the North, East and North-East. We solve this problem by extending one block in the real Schur form of the matrix such that the lost element is included in that block. At the same time we have to diminish some other neighboring blocks. An explicit redistribution of the matrices would cause to much overhead. Instead, we do the redistribution implicitly, that is, we only exchange elements in one row and one column, which are stored in local workarrays. Somehow we must keep track of the extensions/reductions done. As we can see, they are completely determined by the looks of Aii and Bjj . Therefore, we can use two 1D-arrays, call them INFO ARRAY A and INFO ARRAY B
806
R. Granat, B. K˚ agstr¨ om, and P. Poromaa
for k=1, # block diagonals in C {Solve subsystems on current block diagonal in parallel} if(mynode holds Cij ) if(mynode does not hold Aii and/or Bjj ) Communicate for Aii and/or Bjj Solve for Xij in op(Aii )Xij − Xij op(Bij ) = Cij Broadcast Xij to processors that need Xij for updates elseif(mynode needs Xij ) Receive Xij if(mynode does not hold needed block in A for updating block column j) Communicate for requested block in A Update block column j of C in parallel if(mynode does not hold needed block in B for updating block row i) Communicate for requested block in B Update block row i of C in parallel endif end Fig. 5. Parallel block algorithm for op(A)X − Xop(B) = C, A and B in Schur form.
of length M/MB and N/NB, respectively, which store information of the extensions as integer values as follows: 0 if Aii is unchanged IN F O ARRAY A(i) =
1 if Aii is extended
2 if Aii is diminished
3 if Aii is extended and diminished
The first thing to do in our triangular solver is traversing the first subdiagonal of A and B and assigning values to their INFO ARRAY:s. Here, we are forced to do some broadcasts since the information must be global, but since this is an O(M/MB) or O(N/NB) operation they only effect the overall performance marginally. Then, using the data in the global arrays, we can carry out an implicit redistribution by exchanging data between the processors and build up local arrays of double precision numbers holding the extra rows and/or columns for each block of A, B and C. These local arrays can then be used to form the “correct” submatrices for our solves and updates in the parallel triangular solver.
4
Performance Results and Analysis
We present measured performance results of our ScaLAPACK-style algorithms using up to 64 processors on the IBM Scalable POWERparallel (SP) system at High Performance Computing Center North (HPC2N). A theoretical scalability analysis ongoing work and is not included in this paper. We vary P = Pr · Pc between 1 and 64 in multiples of 2. Speedup Sp and efficiency Ep are computed with respect to the run for the current problem size
Parallel ScaLAPACK-Style Algorithms
807
Table 1. Performance of PDTRSY solving AX − XB = C and AX − XB T = C. M = N MB Pr Pl Time A, B 1024 64 1 1 58 1024 64 2 1 14 1024 64 2 2 7.8 1024 64 2 4 7.2 1024 64 4 4 7.3 2048 64 2 2 133 2048 64 4 2 39 2048 64 4 4 30 2048 64 8 4 25 2048 64 8 8 22 4096 64 4 4 281 4096 64 8 4 168 4096 64 8 8 117
(sec.) Sp A, B T A, B A, B T 46 1.0 1.0 13 4.1 3.4 9.4 7.5 4.9 8.2 8.1 5.6 6.0 8.0 7.7 143 1.0 1.0 59 3.4 2.4 32 4.4 4.5 28 5.3 5.1 20 6.0 7.2 301 1.0 1.0 188 1.4 1.6 97 2.4 3.1
Ep A, B A, B T 1.00 1.00 2.06 1.68 1.87 1.22 1.01 0.70 0.50 0.48 1.00 1.00 1.69 1.21 1.11 1.12 0.67 0.64 0.38 0.45 1.00 1.00 0.84 0.80 0.60 0.78
Table 2. Performance results of PDGESY solving AX − XB T = C. M =N 1024 1024 1024 1024 1024 2048 2048 2048 2048 2048 4096 4096 4096
MB 64 64 64 64 64 64 64 64 64 64 64 64 64
Pr 1 2 2 4 4 2 4 4 4 8 4 8 8
Pl Time (sec.) 1 696 1 397 2 260 2 183 4 140 2 2057 2 1061 4 553 8 384 8 364 4 5158 4 2407 8 1478
Sp 1.0 1.7 2.7 3.8 5.0 1.0 1.9 3.7 5.4 5.7 1.0 2.1 3.5
Ep 1.00 0.85 0.68 0.48 0.31 1.00 0.97 0.93 0.67 0.35 1.00 1.10 0.87
#Ext 179 160 160 148 146 663 664 704 604 663 3400 3376 3360
Abs. residual 0.10E − 10 0.98E − 11 0.96E − 11 0.89E − 11 0.89E − 11 0.27E − 10 0.26E − 10 0.26E − 10 0.24E − 10 0.24E − 10 0.72E − 10 0.67E − 10 0.68E − 10
that we were able to solve with as few processors as possible. Therefore, the results for the speedup and efficiency must be understood from the context. All timings are performed on random generated problems which typically are pretty ill-conditioned and have large-normed solutions X. In Table 1, we present performance results for the triangular solver PDTRSY when solving AX − XB = C and AX − XB T = C, and A and B are in upper real Schur form. In Table 2, we present performance results for the general solver PDGESY when solving AX − XB T = C. Here, the timings include all four steps. Moreover, we display the number of 2 × 2 diagonal split problems that were involved and the absolute residual of the solutions. The sizes of the residuals are due to the fact that the random problems are rather ill-conditioned, i.e., the separation between
808
R. Granat, B. K˚ agstr¨ om, and P. Poromaa Table 3. Execution time profile of PDGESY solving AX − XB T = C. M = N MB Pr Pc Step 1 Steps 2+4 Step 3 Total time (%) (%) (%) (sec.) 1024 64 1 1 83 12 4 696 1024 64 2 1 90 6 3 397 1024 64 2 2 92 4 4 260 1024 64 4 2 89 5 4 183 1024 64 4 4 89 2 4 140 2048 64 2 2 81 12 7 2057 2048 64 4 2 85 6 9 1061 2048 64 4 4 90 3 6 553 2048 64 4 8 87 4 7 384 2048 64 8 8 86 3 5 364 4096 64 4 4 75 17 8 5158 4096 64 8 4 79 12 8 2407 4096 64 8 8 89 4 7 1478
A and B are quite small resulting in near to singular systems to solve. Finally, in Table 3, we present the execution profile of the results of Table 2. As expected, it is the transformations to Schur form in Step 1 that dominate the execution time. However, it is still important to have a scalable and efficient solver for the triangular SYCT equations, since in condition estimation we typically have to call PDTRSY several (about five) times [15,11,12]. Our software is designed for integration in state-of-the-art software libraries such as ScaLAPACK [3] and SLICOT [17,6]. Acknowledgements. This research was conducted using the resources of the High Performance Computing Center North (HPC2N). Financial support has been provided by the Swedish Research Council under grant VR 621-2001-3284 and by the Swedish Foundation for Strategic Research under grant A3 02:128.
References 1. R.H. Bartels and G.W. Stewart Algorithm 432: Solution of the Equation AX + XB = C, Comm. ACM, 15(9):820–826. 2. E. Anderson, Z. Bai, C. Bischof. J. Demmel, J. Dongarra, J. DuCroz, A. Greenbaum, S. Hammarling, A. McKenny, S. Ostrouchov and D. Sorensen. LAPACK User’s Guide. Third Edition. SIAM Publications, 1999. 3. S. Blackford, J. Choi, A. Clearly, E. D’Azevedo, J. Demmel, I. Dhillon, J. Dongarra, S. Hammarling, G. Henry, A. Petitet, K. Stanley, D. Walker, and R.C. Whaley. ScaLAPACK Users’ Guide. SIAM Publications, Philadelphia, 1997.
Parallel ScaLAPACK-Style Algorithms
809
4. K. Dackland and B. K˚ agstr¨ om. An Hierarchical Approach for Performance Analysis of ScaLAPACK-based Routines Using the Distributed Linear Algebra Machine. In Wasniewski et.al., editors, Applied Parallel Computing in Industrial Computation and Optimization, PARA96, Lecture Notes in Computer Science, Springer, Vol. 1184, pages 187–195, 1996. 5. J. J. Dongarra, J. Du Croz, I. S. Duff, and S. Hammarling, A set of Level 3 Basic Linear Algebra Subprograms, ACM Trans. Math. Soft., 16(1):1–17, 1990. 6. E. Elmroth, P. Johansson, B. K˚ agstr¨ om, and D. Kreissner, A Web Computing Environment for the SLICOT Library, In P. Van Dooren and S. Van Huffel, The Third NICONET Workshop on Numerical Control Software, pp 53–61, 2001. 7. R. Granat, A Parallel ScaLAPACK-style Sylvester Solver, Master Thesis, UMNAD 435/03, Dept. Computing Science, Ume˚ a University, Sweden, January, 2003. 8. G. Henry and R. Van de Geijn. Parallelizing the QR Algorithm for the Unsymmetric Algebraic Eigenvalue Problem: Myths and Reality. SIAM J. Sci. Comput. 17:870–883, 1997. 9. G. Henry, D. Watkins, and J. Dongarra, J. A Parallel Implementation of the Nonsymmetric QR Algorithm for Distributed Memory Architectures. Technical Report CS-97-352 and Lapack Working Note 121, University of Tennessee, 1997. 10. N.J. Higham. Perturbation Theory and Backward Error for AX − XB = C, BIT, 33:124–136, 1993. 11. I. Jonsson and B. K˚ agstr¨ om. Recursive Blocked Algorithms for Solving Triangular Matrix Equations—Part I: One-Sided and Coupled Sylvester-Type Equations, ACM Trans. Math. Software, Vol. 28, No. 4, pp 393–415, 2002. 12. I. Jonsson and B. K˚ agstr¨ om. Recursive Blocked Algorithms for Solving Triangular Matrix Equations—Part II: Two-Sided and Generalized Sylvester and Lyapunov Equations, ACM Trans. Math. Software, Vol. 28, No. 4, pp 416–435, 2002. 13. B. K˚ agstr¨ om, P. Ling, and C. Van Loan. GEMM-based level 3 BLAS: Highperformance model implementations and performance evaluation benchmark. ACM Trans. Math. Software, 24(3):268–302, 1998. 14. B. K˚ agstr¨ om, P. Ling, and C. Van Loan. GEMM-based level 3 BLAS: Portability and optimization issues. ACM Trans. Math. Software, 24(3):303–316, 1998. 15. B. K˚ agstr¨ om and P. Poromaa. Distributed and shared memory block algorithms for the triangular Sylvester equation with Sep−1 estimators, SIAM J. Matrix Anal. Appl., 13 (1992), pp. 99–101. 16. P. Poromaa. Parallel Algorithms for Triangular Sylvester Equations: Design, Scheduling and Scalability Issues. In K˚ agstr¨ om et al. (eds), Applied Parallel Computing. Large Scale Scientific and Industrial Problems, Lecture Notes in Computer Science, Vol. 1541, pp 438–446, Springer-Verlag, 1998. 17. SLICOT library in the Numerics in Control Network (NICONET) website: www.win.tue.nl/niconet/index.html
RECSY — A High Performance Library for Sylvester-Type Matrix Equations Isak Jonsson and Bo K˚ agstr¨om Department of Computing Science and HPC2N, Ume˚ a University, SE-901 87 Ume˚ a, Sweden. {isak,bokg}@cs.umu.se
Abstract. RECSY is a library for solving triangular Sylvester-type matrix equations. Its objectives are both speed and reliability. In order to achieve these goals, RECSY is based on novel recursive blocked algorithms, which call high-performance kernels for solving small-sized leaf problems of the recursion tree. In contrast to explicit standard blocking techniques, our recursive approach leads to an automatic variable blocking that has the potential of matching the memory hierarchies of today’s HPC systems. The RECSY library comprises a set of Fortran 90 routines, which uses recursion and OpenMP for shared memory parallelism to solve eight different matrix equations, including continuous-time as well as discrete-time standard and generalized Sylvester and Lyapunov equations. Uniprocessor and SMP parallel performance results of our recursive blocked algorithms and corresponding routines in state-of-the-art libraries LAPACK and SLICOT are presented. The performance improvements of our recursive algorithms are remarkable, including 10-fold speedups compared to standard algorithms. Keywords: Sylvester-type matrix equations, recursion, automatic blocking, superscalar, GEMM-based, level 3 BLAS, LAPACK, SLICOT, RECSY
1
Introduction
In [8,9], we describe recursive blocked algorithms for solving different Sylvestertype matrix equations. We differentiate between one-sided and two-sided matrix equations. The notation one-sided matrix equations is used when the solution is only involved in matrix products of two matrices, e.g., op(A)X or Xop(A), where op(A) can be A or AT . In two-sided matrix equations, the solution is involved in matrix products of three matrices, both to the left and to the right, e.g., op(A)Xop(B). Table 1 lists eight different types of matrix equations considered together with the acronyms used (one-sided equations in the top and two-sided equations in the bottom part).
H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 810–819, 2003. c Springer-Verlag Berlin Heidelberg 2003
RECSY — A High Performance Library
811
Table 1. One-sided (top) and two-sided (bottom) matrix equations. Name Standard Sylvester (CT) Standard Lyapunov (CT) Generalized Coupled Sylvester Standard Sylvester (DT) Standard Lyapunov (DT) Generalized Sylvester Generalized Lyapunov (CT) Generalized Lyapunov (DT)
1.1
Matrix equation AX − XB = C AX + XAT = C (AX − Y B, DX − Y E) = (C, F ) AXB T − X = C AXAT − X = C AXB T − CXDT = E AXE T + EXAT = C AXAT − EXE T = C
Acronym SYCT LYCT GCSY SYDT LYDT GSYL GLYCT GLYDT
Triangular Matrix Equations
The classical method of solution of the Sylvester-type matrix equations is based on the Bartels–Stewart method [2], which includes three major steps. First, the matrix (or matrix pair) is transformed to a Schur (or generalized Schur) form. This leads to a reduced triangular matrix equation. For example, the coefficient matrices A and B in the Sylvester equation AX − XB = C are in upper triangular or upper quasi-triangular form. Finally, the solution of the reduced matrix equation is transformed back to the originally coordinate system. Reliable and efficient algorithms for the reduction step can be found in LAPACK [1] for the standard case, and in [3] for the generalized case, where a blocked variant of the QZ method is presented. Triangular matrix equations also appear naturally in estimating the condition numbers of matrix equations and different eigenspace computations, including block-diagonalization of matrices and matrix pairs and computation of functions of matrices. Related applications include the direct reordering of eigenvalues in the real (generalized) Schur form and the computation of additive decompositions of a (generalized) transfer function (see [8,9] for more information and references). 1.2
Motivation
Our goal is to produce a state-of-the-art library which solves the triangular Sylvester-type matrix equations listed in Table 1. The library should be easy to use, provide excellent performance, and it should be expandable and tunable for new routines and different platforms. In this contribution, we present a new library, RECSY, which contains sequential and parallel implementations of all triangular matrix equations listed in Table 1. Also, the routines feature several different transpose and sign options, so all in all the RECSY routines are able to solve 42 different variants. Before we go into details of the library, we outline the rest of the paper. In Section 2, we describe the routines for solving one-sided triangular equations. We do this by reviewing our recursive approach, and how it gives better performance. Furthermore, we describe the fast and highly optimized kernels used
812
I. Jonsson and B. K˚ agstr¨ om
function [X] = recsyct(A, B, C, uplo, blks) if 1 ≤ M, N ≤ 4 then X = trsyct(A, B, C, uplo); else if 1 ≤ N ≤ M/2 % Case 1: Split A (by rows and colums), C (by rows only) X2 = recsyct(A22 , B, C2 , 1, blks); C1 = gemm(−A12 , X2 , C1 ); X1 = recsyct(A11 , B, C1 , 1, blks); X = [X1 ; X2 ]; elseif 1 ≤ M ≤ N/2 % Case 2: Split B (by rows and colums), C (by columns only) X1 = recsyct(A, B11 , C1 , 1, blks); C2 = gemm(X1 , B12 , C2 ); X2 = recsyct(A, B22 , C2 , 1, blks); X = [X1 , X2 ]; else % M, N ≥ blks, Case 3: Split A, B and C (all by rows and colums) X21 = recsyct(A22 , B11 , C21 , 1, blks); C22 = gemm(X21 , B12 , C22 ); C11 = gemm(−A12 , X21 , C11 ); X22 = recsyct(A22 , B22 , C22 , 1, blks); X11 = recsyct(A11 , B11 , C11 , 1, blks); C12 = gemm(−A12 , X22 , C12 ); C12 = gemm(X11 , B12 , C12 ); X12 = recsyct(A11 , B22 , C12 , 1, blks); X = [X11 , X12 ; X21 , X22 ]; end end
Algorithm 1: SYCT
Algorithm 1: Recursive blocked algorithm for solving the triangular continuous-time Sylvester equation (SYCT).
to solve small problems. We also show how we easily provide parallel routines for SMP/OpenMP-aware systems. The routines for solving two-sided triangular equations are listed in Section 3. The differences between the one-sided and the two-sided equations are explained, and we show how the library takes care of these issues. In Section 4, we give a short guide on how to use the library. Some performance results are given in Section 5.
2
One-Sided Triangular Matrix Equations
In [6], we present pseudo-code algorithms for three one-sided triangular matrix equations. These are the continuous-time standard Sylvester (SYCT), the standard Lyapunov (LYCT), and the generalized coupled Sylvester (GCSY) equations, together with the mathematical derivation of the algorithms. For reference, we show a compressed version of one of the algorithms, SYCT, in Algorithm 1. The equation is solved by recursion, where the problem is split into two or four new, smaller, problems of approximately the same size. The aim is to split the matrix exactly as close to the middle as possible. If the splitting point appears at a 2 × 2 diagonal block, the matrices are split just below the splitting point. 2.1
Performance Due to Recursive Level 3 Blocking and Updates
The recursive approach provides automatic cache blocking through great temporal locality. Furthermore, it reveals level 3 BLAS updates, shown as gemm calls
RECSY — A High Performance Library
813
in Algorithm 1. The same observations hold for the LYCT and GCSY equations. For LYCT, the updates are done by symmetric rank-k and symmetric rank-2k operations instead, due to its symmetry characteristics. However, most of the work in LYCT is done in SYCT, as LYCT uses SYCT to solve off-diagonal blocks in the solution X. It has been observed that for some platforms, the vendor’s DGEMM does not perform very well for small matrices. This is due to overhead from parameter checking and temporary buffer setup. Therefore, the library supplies its own matrix-matrix multiply which uses 4 × 4 outer loop unrolling, but without blocking for memory cache. It shows good performance on different types of processors. This routine is used instead of the regular DGEMM when the problem fits into level 1 cache. Note that recursion itself provides good temporal locality. 2.2
Performance Due to Fast Kernels
At the bottom of the recursion tree for the Sylvester equations (SYCT, GCSY), where the problem size is very small (≤ 4×4), the recursion is terminated and the problem is instead solved by a very fast kernel. Now, the small matrix equations are represented as a linear system of equations Zx = y, where Z is a Kronecker product representation of the Sylvester-type operator, and x and the right hand side y are vectors representing the solution and the the right hands side(s). For example, ZSYCT = In ⊗ A − B T ⊗ Im (see also [8,9]). This kernel features unrolling for the updates, and uses partial pivoting for solving small matrix equations. These equations lead to Kronecker product matrix representations Z of size 1 × 1 – 4 × 4 for SYCT and 2 × 2 – 8 × 8 for GCSY. If a good pivot candidate cannot be found (problem is nearly singular), or if the solution procedure is close to produce an overflow, the fast kernel is aborted. On abortion, the recursive procedure backtracks and we construct a new Kronecker product representation, now leading to a matrix Z of size 16 × 16 for SYCT and 32 × 32 for GCSY. The system is then solved using complete pivoting with perturbed pivot elements if necessary and right hand scaling to avoid overflow. For LYCT, the recursion continues until subproblems of size 1 × 1 or 2 × 2. Such a problem, with Kronecker product matrix Z of size 1 × 1 or 3 × 3, is solved using the fast kernel solver (partial pivoting), or with complete pivoting for very ill–conditioned cases. The reason for not optimizing LYCT as much as SYCT and GCSY is that most of the solving takes part in SYCT. The number of small problems solved in LYCT by the Lyapunov small problem solver is O(n), whereas the number of problems solved in LYCT by the SYCT solver is O(n2 ). 2.3
Performance Due to Parallelism
First of all, it is possible to get a fair amount of speedup on systems using only SMP versions of level 3 BLAS routines. This is due to fact that the recursive algorithms call the level 3 BLAS routines with large, squarish blocks. The routines can thus do each update in parallel with good efficiency.
814
I. Jonsson and B. K˚ agstr¨ om
As can be observed in Case 3 of Algorithm 1, there are two recursive calls that can be executed simultaneously. This is used in our implementation of SYCT and GCSY to create OpenMP versions of our algorithms for SMP systems, where not only the updates, but also the solves are done in parallel. The OpenMP versions take an additional argument, PROC, the number of processors available. Whenever PROC > 1, and the problem is a Case 3 problem, the second and third recursive calls (and corresponding updates) are done in parallel, giving a diamond-shaped execution graph. For more details, see [6]. For LYCT, there is no explicit parallelism in the Lyapunov solver. However, as most of the work takes place in the Sylvester solver SYCT, the LYCT routine gains performance from the OpenMP parallelism as well. In order for this scheme to work well, two conditions must hold. First, there should not be any degradation in the total performance when the SMP BLAS 3 routines are called simultaneously (multiple parallelism). This has been a problem on a small number of machines, where OpenMP and SMP BLAS did not work together. Second, if there are more than two processors in the system, the OpenMP compiler must support nested parallelism in order to fully utilize the system, which most modern OpenMP compilers do. The overhead of SMP BLAS routines can be pretty large, especially when solving too small problems. Fortunately, the library’s own matrix-matrix multiply routine alleviates this problem. As it is not parallelized, it is perfectly suited for small problems, where the parallel divide-and-conquer splitting already has taken place at higher levels of the recursion tree. 2.4
Fortran Interfaces
The routines are implemented in Fortran 90, which greatly simplifies the recursive calls. Also, this makes the routines’ signatures easier, as some arguments can be declared optional. Below, the six routines for solving one-sided triangular matrix equations are listed. • RECSYCT (UPLOSIGN, SCALE, M, N, A, LDA, B, LDB, C, LDC, INFO, MACHINE) • RECSYCT P (PROCS, UPLOSIGN, SCALE, M, N, A, LDA, B, LDB, C, LDC, INFO, MACHINE)
Solves the triangular continuous-time Sylvester equation. A and B are upper quasitriangular of size M × M and N × N , C is rectangular M × N . UPLOSIGN 0 1 2 3 4 5 6 7
SYCT equation AX + XB = scale C, C ← X AX − XB = scale C, C ← X AX + XB T = scale C, C ← X AX − XB T = scale C, C ← X AT X + XB = scale C, C ← X AT X − XB = scale C, C ← X AT X + XB T = scale C, C ← X AT X − XB T = scale C, C ← X
• RECLYCT (UPLO, SCALE, M, A, LDA, C, LDC, INFO, MACHINE)
RECSY — A High Performance Library
815
• RECLYCT P (PROCS, UPLO, SCALE, M, A, LDA, C, LDC, INFO, MACHINE)
Solves the symmetric triangular continuous-time Lyapunov equation. A is upper quasitriangular M × M , C is symmetric M × M , only upper part of C is accessed. UPLOSIGN LYCT equation 0 AX + XAT = scale C, C ← X 1 AT X + XA = scale C, C ← X
• RECGCSY LDE, F, • RECGCSY LDD, E,
(UPLOSIGN, SCALE, M, N, A, LDA, B, LDB, C, LDC, D, LDD, E, LDF, INFO, MACHINE) P (PROCS, UPLOSIGN, SCALE, M, N, A, LDA, B, LDB, C, LDC, D, LDE, F, LDF, INFO, MACHINE)
Solves the generalized coupled triangular Sylvester equation. A is upper quasitriangular M × M , D is upper triangular M × M , B is upper quasitriangular N × N , E is upper triangular N × N , C and F are rectangular M × N. UPLOSIGN 0 1 2 3 4 5 6 7
3
GCSY equation (AX + Y B = scale C, DX + Y E = scale F ), (C, F ) ← (X, Y ) (AX − Y B = scale C, DX − Y E = scale F ), (C, F ) ← (X, Y ) (AX + Y B T = scale C, DX + Y E T = scale F ), (C, F ) ← (X, Y ) (AX − Y B T = scale C, DX − Y E T = scale F ), (C, F ) ← (X, Y ) (AT X + Y B = scale C, D T X + Y E = scale F ), (C, F ) ← (X, Y ) (AT X − Y B = scale C, D T X − Y E = scale F ), (C, F ) ← (X, Y ) (AT X + Y B T = scale C, D T X + Y E T = scale F ), (C, F ) ← (X, Y ) (AT X − Y B T = scale C, D T X − Y E T = scale F ), (C, F ) ← (X, Y )
Two-Sided Triangular Matrix Equations
The recursive templates for solving two-sided triangular matrix equations are similar to the templates for solving one-sided triangular matrix equations. However, the two-sidedness of the matrix updates leads to several different implementation choices. In the RECSY library, two internal routines are used: RECSY MULT LEFT and RECSY MULT RIGHT. These routines compute C = βC + αop(A)X and C = βC + αXop(A), respectively, where op(A) is A or AT , and A is rectangular (DGEMM), upper triangular, upper quasitriangular, upper trapezoidal, or upper quasitrapezoidal (i.e., trapezoidal with 1×1 and 2×2 diagonal blocks). C and X are rectangular matrices. The actual computation is carried out by calls to DGEMM, DTRMM, the internal small matrix-matrix multiply, and level 1 BLAS routines for subdiagonal elements. Note that these routines require extra workspace when DTRMM is used, as DTRMM only computes the product (C = op(A)X) and not the multiply-and-add (C = C + op(A)X). Combining RECSY MULT LEFT and RECSY MULT RIGHT give the routine RECSY AXB. It computes the update C = βC + αop(A)Xop(B), where A and B can independently be any of the five matrix structures listed above. There are two ways of computing the update, either by doing the left multiplication first or by doing the right multiplication first. RECSY AXB chooses the ordering which uses the least amount of floating point operations. RECSY AXB also requires workspace to store the intermediate product.
816
I. Jonsson and B. K˚ agstr¨ om
However, for symmetric two-sided updates C = βC +αop(A)Xop(AT ), where C and X are symmetric, the SLICOT [12] routine MB01RD is used. MB01RD uses a series of level 3 updates to calculate C without requiring workspace and with fewer floating point operations than RECSY AXB. 3.1
Fortran Interfaces
As shown above, the two-sided matrix equation solvers require extra workspace. As Fortran 90 allows dynamic memory allocation, this facilitates the subroutine calls. The user now has the option of either supplying workspace to the routine, or, preferably, leave this task to the subroutine itself. Below, the ten routines for solving two-sided triangular matrix equations are shown. • RECSYDT (UPLOSIGN, SCALE, M, N, A, LDA, B, LDB, C, LDC, INFO, MACHINE, WORKSPACE, WKSIZE) • RECSYDT P (PROCS, UPLOSIGN, SCALE, M, N, A, LDA, B, LDB, C, LDC, INFO, MACHINE, WORKSPACE, WKSIZE)
Solves the triangular discrete-time Sylvester equation. A is upper quasitriangular M × M , B is upper quasitriangular N × N , C is rectangular M × N . UPLOSIGN 0 1 2 3 4 5 6 7
SYDT equation AXB + X = scale C, C ← X AXB − X = scale C, C ← X AXB T + X = scale C, C ← X AXB T − X = scale C, C ← X AT XB + X = scale C, C ← X AT XB − X = scale C, C ← X AT XB T + X = scale C, C ← X AT XB T − X = scale C, C ← X
• RECLYDT (UPLO, SCALE, M, A, LDA, C, LDC, INFO, MACHINE, WORKSPACE, WKSIZE) • RECLYDT P (PROCS, UPLO, SCALE, M, A, LDA, C, LDC, INFO, MACHINE, WORKSPACE, WKSIZE)
Solves the symmetric triangular discrete-time Lyapunov equation. A is upper quasitriangular M × M , C is symmetric M × M , only upper part of C is accessed. UPLOSIGN LYDT equation 0 AXAT − X = scale C, C ← X 1 AT XA − X = scale C, C ← X
• RECGSYL (UPLOSIGN, SCALE, M, N, A, LDA, B, LDB, C, LDC, D, LDD, E, LDE, INFO, MACHINE,WORKSPACE, WKSIZE) • RECGSYL P (UPLOSIGN, SCALE, M, N, A, LDA, B, LDB, C, LDC, D, LDD, E, LDE, INFO, MACHINE, WORKSPACE, WKSIZE)
Solves the triangular generalized Sylvester equation. A is upper quasitriangular M × M , C is upper triangular M × M , B is is upper quasitriangular N × N , D is upper triangular N × N , C and F are rectangular M × N .
RECSY — A High Performance Library UPLOSIGN 0 1 2 3 4 5 6 7 10 12
817
GSYL equation AXB + CXD = scale E, E ← X AXB − CXD = scale E, E ← X AXB T + CXD T = scale E, E ← X AXB T − CXD T = scale E, E ← X AT XB + C T XD = scale E, E ← X AT XB − C T XD = scale E, E ← X AT XB T + C T XD T = scale E, E ← X AT XB T − C T XD T = scale E, E ← X AXB T + CXD T = scale E, E ← X, B is upper triangular N × N , D is upper quasitriangular N × N (used by RECGLYCT) AT XB + C T XD = scale E, E ← X, B is upper triangular N × N , D is upper quasitriangular N × N (used by RECGLYCT)
• RECGLYDT (UPLO, SCALE, M, A, LDA, E, LDE, C, LDC, INFO, MACHINE, WORKSPACE, WKSIZE) • RECGLYDT P (PROCS, UPLO, SCALE, M, A, LDA, E, LDE, C, LDC, INFO, MACHINE, WORKSPACE, WKSIZE) • RECGLYCT (UPLO, SCALE, M, A, LDA, E, LDE, C, LDC, INFO, MACHINE, WORKSPACE, WKSIZE) • RECGLYCT P (PROCS, UPLO, SCALE, M, A, LDA, E, LDE, C, LDC, INFO, MACHINE, WORKSPACE, WKSIZE)
Solves the triangular generalized discrete-time and continuous-time Lyapunov equation, respectively. A is upper quasitriangular M × M , E is upper triangular M × M , C is symmetric M × M , only upper part of C is accessed. UPLOSIGN GLYDT equation GLYCT equation 0 AXAT − EXE T = scale C, C ← X AXE T + EXAT = scale C, C ← X 1 AT XA − E T XE = scale C, C ← X AT XE − E T XA = scale C, C ← X
4
Library Usage
The library, together with building instructions, can be downloaded from http://www.cs.umu.se/˜isak/recsy. The library is easily built with any Fortran 90 compiler, both UNIX and Windows compilers are supported. By using the routines listed in Sections 2 and 3, the user gets access to all of the different options and tuning parameters of the library. The types of the parameters to the routines are listed in Table 2. The parameter MACHINE, which can be omitted – as the authors recommend, sets different tuning parameters. For details, see the library homepage. 4.1
SLICOT/LAPACK Wrapper Routines
Instead of using the native routines listed, the library also provides wrapper routines which overload SLICOT [12] and LAPACK [1] routines, see Table 3. By linking with the library, calls to SLICOT and LAPACK Sylvester-type matrix equations solvers will be replaced to a call to the optimized RECSY equivalent. We remark that neither LAPACK nor SLICOT provide a GSYL solver. Note that not only the SLICOT and LAPACK routines listed above are affected. Any routine which uses any of these routines will also be using the RECSY library. In particular, this is true for the SLICOT routines which solve
818
I. Jonsson and B. K˚ agstr¨ om
Table 2. The parameters to the RECSY library routines, their type and meaning. Parameter UPLOSIGN, UPLO A, B, C, D, E, F SCALE
Type INTEGER DOUBLE PRECISION(*) DOUBLE PRECISION
LDA, LDB, LDC, LDD, LDE, LDF INFO
INTEGER
MACHINE WORKSPACE WKSIZE
Meaning Defines different options (see above). Coefficient/right hand side matrices (see above). Scaling factor (output argument), to avoid overflow in the computed solution. Leading dimension of matrices A to F.
Output parameter. INFO= −100: Out of memory, INFO= 0: No error, INFO> 0: The equation is (nearly) singular to working precision; perturbed values were used to solve the equation. DOUBLE PRECISION(0:9) Optional. Specify only if you want different parameters than the default parameters, which are set by RECSY MACHINE. DOUBLE PRECISION(*) Optional. Specify only if you provide your own workspace. INTEGER Optional. Specify only if you provide your own workspace. INTEGER
Table 3. SLICOT and LAPACK routines and their RECSY equivalents. Routine SLICOT SB03MX SLICOT SB03MY SLICOT SB04PY SLICOT SG03AX SLICOT SG03AY LAPACK DTRSYL LAPACK DTGSYL [10]
Native routine RECLYDT RECLYCT RECSYDT RECGLYDT RECGLYCT RECSYCT RECGCSY
Table 4. a) Performance results for the triangular Lyapunov equation—IBM Power3, 200 MHz (left) and SGI Onyx2 MIPS R10000, 195 MHz (right). b) Performance results for the triangular discrete-time Sylvester equation—IBM Power3, 4 × 375 MHz (left) and Intel Pentium III, 2 × 550 MHz (right).
(M =)N 100 250 500 1000 1500 2000
a) AX + XAT = C IBM Power3 MIPS R10000 Mflops/s Speedup Mflops/s Speedup A B B/A A B B/A 77.0 166.5 2.16 82.0 123.5 1.51 85.3 344.5 4.04 88.7 224.5 2.53 10.6 465.0 43.85 42.2 277.8 6.58 7.7 554.7 72.20 14.5 254.0 17.57 7.0 580.5 83.19 9.7 251.0 25.81
A – SLICOT SB03MY B – RECLYCT
b) AXB − X = C IBM Power3 Intel Pentium III Time (sec) Speedup Time (sec) Speedup C D D/C E/D C D D/C E/D 1.73e-2 7.41e-3 2.33 1.16 2.20e-2 2.20e-2 1.00 1.22 6.20e-1 6.93e-2 8.95 0.98 6.83e-1 2.31e-1 2.96 1.33 2.32e+1 4.60e-1 50.50 1.48 7.82e+0 1.56e+0 5.00 1.35 2.44e+2 3.26e+0 74.65 1.94 7.04e+1 1.10e+1 6.40 1.48 9.37e+2 1.08e+1 86.66 2.04 2.78e+2 3.58e+1 7.76 1.53 2.84e+3 2.41e+1 117.71 2.14 9.07e+2 8.03e+1 11.29 1.57 C – SLICOT SB04PY D – RECSYDT E – RECSYDT P
unreduced problems, e.g., SB04MD (SYCT solver) or SM03MD (LYDT/LYCT solver). SB04MD and SB03MD call DTRSYL and SB03MX/SB03MY, respectively, for solving the reduced equation. With the RECSY library, performance for SB04MD and SB03MD are also improved [7,9].
5
Performance Results
We list a few results obtained with the RECSY library. In Table 4 a), results obtained for the continuous-time Lyapunov equation (LYCT) are shown. Here, the speedup is remarkable. The RECSY library is up to 83 times faster than the original SLICOT library. This is both due to faster kernels and multi-level blocking from recursion. The same results can be observed for LAPACK routines.
RECSY — A High Performance Library
819
For example, the RECSY routine RECSYCT is more than 20 times faster then the LAPACK routine DTRSYL for M = N > 500 on the IBM PowerPC 604e. Timings for two-sided matrix equation examples and given in Table 4 b). For the largest example, the SLICOT library requires more than 45 minutes to solve the problem. The RECSY library solves the same problem in less than a half minute. The extra speedup from the OpenMP version of the library is also given. For further results, we refer to [8,9] and the library homepage. Acknowledgements. Financial support for this project has been provided by the Swedish Research Council under grants TFR 98-604, VR 621-2001-3284 and the Swedish Foundation for Strategic Research under grant A3 02:128.
References 1. E. Anderson, Z. Bai, J. Demmel, J. Dongarra, J. DuCroz, A. Greenbaum, S. Hammarling, A. McKenny, S. Ostrouchov, and D. Sorensen LAPACK Users Guide, Third Edition. SIAM Publications, 1999. 2. R.H. Bartels and G.W. Stewart. Algorithm 432: Solution of the Equation AX + XB = C, Comm. ACM, 15(9):820–826, 1972. 3. K. Dackland and B. K˚ agstr¨ om. Blocked Algorithms and Software for Reduction of a Regular Matrix Pair to Generalized Schur Form. ACM Trans. Math. Software, 24(4):425–454, December 1999. 4. J.D. Gardiner, A.J. Laub, J.J. Amato, and C.B. Moler. Solution of the Sylvester Matrix Equation AXB T + CXDT = E, ACM Trans. Math. Software, 18(2):223– 231, June 1992. 5. J.D. Gardiner, M.R. Wette, A.J. Laub, J.J. Amato, and C.B. Moler. A Fortran 77 Software Package for Solving the Sylvester Matrix Equation AXB T +CXDT = E, ACM Trans. Math. Software, 18(2):232–238, June 1992. 6. I. Jonsson and B. K˚ agstr¨ om. Recursive Blocked Algorithms for Solving Triangular Matrix Equations—Part I: One-Sided and Coupled Sylvester-Type Equations, SLICOT Working Note 2001-4. 7. I. Jonsson and B. K˚ agstr¨ om. Recursive Blocked Algorithms for Solving Triangular Matrix Equations—Part II: Two-Sided and Generalized Sylvester and Lyapunov Equations, SLICOT Working Note 2001-5. 8. I. Jonsson, and B. K˚ agstr¨ om. Recursive blocked algorithms for solving triangular systems — Part I: One-sided and coupled Sylvester-type matrix equations. ACM Trans. Math. Softw., 28(4):392–415, December 2002. 9. I. Jonsson, and B. K˚ agstr¨ om. Recursive blocked algorithms for solving triangular systems — Part II: Two-sided and generalized Sylvester and Lyapunov matrix equations ACM Trans. Math. Softw., 28(4):416–435, December 2002. 10. B. K˚ agstr¨ om and P. Poromaa. LAPACK–Style Algorithms and Software for Solving the Generalized Sylvester Equation and Estimating the Separation between Regular Matrix Pairs. ACM Trans. Math. Software, 22(1):78–103, March 1996. 11. B. K˚ agstr¨ om and L. Westin. Generalized Schur methods with condition estimators for solving the generalized Sylvester equation. IEEE Trans. Autom. Contr., 34(7):745–751, July 1989. 12. SLICOT library and the Numerics in Control Network (NICONET) website: www.win.tue.nl/niconet/index.html
Two Level Parallelism in a Stream-Function Model for Global Ocean Circulation Martin van Gijzen CERFACS, 42, avenue Gaspard Coriolis, 31057 Toulouse CEDEX 1, France
[email protected] http://www.cerfacs.fr/algor
Abstract. In this paper, we show how a stream-function model for global ocean circulation can be parallelised, and in particular how the continent boundary conditions can be treated in a parallel setting. Our iterative solution technique for the linear system that results after discretisation, combines loop-level parallelism with a domain-decomposition approach. This two-level parallelism can be exploited in a natural way on a cluster of SMP’s.
1
Introduction
It is customary (e.g. [1]) to split the 3D-ocean flow into a depth-averaged barotropic part, and the 3D-baroclinic deviations from it. The 2D-barotropic part is often formulated in terms of a stream function, which simplifies the governing partial differential equations. The disadvantage of this formulation is that the boundary conditions on continents and islands are more complicated. In [10] a Finite-Element model is described that incorporates these boundary conditions in a systematic and natural way. The stream-function model described in [10] can be vectorised by exploiting the regularity of the grid, see [9]. A similar loop-level parallelisation approach can be used on so called Symmetric Multi Processors (SMP’s) where all processors share the same memory space. For computers with a disjoint memory space, however, one has to use a coarser-grain parallelism, which is usually done by a domain-decomposition approach, as in [8]. In recent years, new computer architectures have appeared that combine disjoint memory address space between groups of processors and a global memory address space within each group of processors. This kind of computer is usually called a “Cluster of SMPs”. The organisation of the memory of a cluster of SMP’s perfectly matches the requirements of parallel algorithms that can exploit two levels of parallelism. The outer/coarser level is implemented between the SMPs and the inner/finer within each SMP. The corresponding parallel programming paradigms are message passing at the coarser level and loop-level parallelism at the finer. This two-level parallelism has received considerable attention lately, see [4] and its references. H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 820–829, 2003. c Springer-Verlag Berlin Heidelberg 2003
Two Level Parallelism in a Stream-Function Model
821
In this paper we show how a two-level parallelisation method can be implemented in a (simplified) stream function model by combining loop-level parallelisation with a domain decomposition. Special attention will be given to the treatment of the continent boundary conditions, which are literally global by nature and therefore complicate parallelisation. The parallel performance of the resulting method is examined by means of numerical experiments on a cluster of bi-processor PC’s.
2 2.1
A Simple Ocean Circulation Model Analytical Formulation
Steady barotropic flow in a homogeneous ocean with constant depth, zero lateral viscosity and in nearly equilibrium can be described by the following partial differential equation: − r∇2 ψ − β
∂ψ = ∇ × F in Ω, ∂x
(1)
in which ψ is the stream function and Ω the part of the world that is covered by sea. The external force field F is equal to the wind stress τ divided by the average depth of the ocean H times the water density ρ. The other parameters in this equation are the bottom friction r and the Coriolis parameter β. The no-normal-flow boundary condition implies that the stream function is constant on each continent, ψ = Ck
on Γk , k = 1, · · · , nk ,
(2)
with nk the number of continents. Additional conditions to determine these unknown constants are obtained by requiring continuity of the water-level around each continent boundary Γk . This yields for each continent an integral condition: ∂ψ r F · s ds. (3) ds = − Γk ∂n Γk The equations are commonly expressed in spherical coordinates, which maps the physical domain onto a rectangular domain with periodic boundary conditions ψ(−180o , θ) = ψ(+180o , θ).
(4)
Here θ is the latitude. The coordinate transformation introduces a singularity on the North pole (the South pole is land). This singularity is solved by imposing the additional boundary condition ψ = 0 if θ = 90o . 2.2
Discretisation
The main problem of discretising (1-4) is posed by the boundary conditions for the continents. As explained in [10], this problem can be solved in a natural way
822
M. van Gijzen
within the context of the Finite Element Method. The procedure is as follows: Step 1: Define a structured grid on the rectangular domain, disregarding islands and continents. Step 2: Discretise (1) everywhere, again disregarding islands and continents. This yields a right-hand-side vector f and a structured matrix K with only a few diagonals with nonzero elements. Discretisation with the Finite Element Method allows us to satisfy the integral conditions in a natural way. Step 3: Add together all the rows in f and the rows and columns in K that correspond to the same island or continent. We denote the resulting right-hand¯ respectively. Note that K ¯ is not side vector and system matrix by ¯ f and. K, structured. The above procedure yields the following system of linear equations: ¯x = ¯ K¯ f.
(5)
In this equation, x contains the values of the stream function in the sea gridpoints and the nk values of the stream function Ck (see equation (2)) on the continents. The parallelisation of the discretisation of (1-4) is straightforward. All the element matrices and vectors can be computed in parallel. Moreover, the time for the discretisation is almost negligible (less than 10 %) compared to the time it takes to solve (5). The parallel solution of the linear system of equations is the subject of the next Section.
3 3.1
The Parallel Solution of the Linear System of Equations Operations with the System Matrix
The diagonal structure of the matrix K is ruined by the inclusion of the conditions for the continents (Step 3 in the discretisation). Operations with the resulting unstructured matrix require indirect addressing, which may be inefficient on sequential computers and may also hamper parallelisation. ¯ we define To preserve as much as possible the structure in operations with K the matrix P that maps the structured-grid variables to the set that comprises the sea-point variables and one variable per continent only. This matrix P is defined as follows: - The column of P that corresponds to a sea-point j is just the corresponding j-th basis-vector: Pj = ej ∀ j in Ω. (6) - And a column that corresponds to a continent k is the sum of all the basis vectors corresponding to the gridpoints j on this continent (including the interior): ej . (7) Pk = j∈Γk
System (5) can now be written as x = PT f . PT KP¯
(8)
Two Level Parallelism in a Stream-Function Model
823
The above form is of no advantage if a direct solution method is used since ¯ in triangular factors. However, for itthese methods make a decomposition of K ¯ is erative solution methods there is an advantage since the only operation with K u can be performed the matrix-vector multiplication. The operation v ¯ = PT KP¯ in three steps: 1 Give all the land nodes their continent value, sea nodal values remain unchanged: u = P¯ u 2 Multiply: v = Ku (Structured matrix-vector product). 3 Add up the values on each continent: v ¯ = PT v. The main part of this operation is the structured matrix-vector multiplication, which can be performed diagonalwise, without indirect addressing. 3.2
Iterative Solution Method
To fully exploit the diagonal structure of K we solve (8) with an iterative solution method. Our iterative method of choice is GMRESR [7]. GMRESR is a nested Krylov subspace method which uses GCR [3] for the outer iterations and GMRES [6] for the inner iterations. Both GCR and GMRES explicitly store a set of basis vectors for the Krylov subspace (GCR even two). Nesting the methods has the advantage that the number of outer iterations can be reduced so that a smaller number of basis vectors has to be computed and stored. This normally outweighs the cost (both in storage and in computations) of the inner iterative process. In practice the number of outer GCR iterations can still be too big to store all basis vectors. The usual remedy is to truncate the process after k iterations, so that only the last k iterations are taken into account when the solution is updated. The main building blocks of GMRESR are: matrix-vector products, preconditioning operations, vector updates and inner products. For the preconditioning operation we simply use diagonal scaling, i.e. multiplication with the inverse of ¯ However, note that the GMRES-inner iterations can be the main diagonal of K. seen as a preconditioning method for the outer-iterative method GCR. Multiplication with a diagonal matrix is an inexpensive and easy to parallelise operation. In the remainder of this Section we will therefore concentrate on the parallelisation of the other three operations: the matrix-vector product, the vector update, and the inner product. 3.3
Parallelisation
Fine-Grain Parallelism A fine grain parallel implementation of the inner product and vector update can be made in a straightforward and portable way by adding appropriate OpenMPdirectives [2] around the respective loops. The first step in the matrix-vector multiplication, giving all the gridpoints on the continents their continent value, can be performed in parallel for the
824
M. van Gijzen
different continents. However, this operation is too inexpensive to be parallelised. The overhead of the parallelisation is bigger than the gain. For this reason the operation is performed sequentially. The second step in the matrix vector-multiplication, the diagonalwise multiplication with K, is composed of a double loop, the outer over the nonzero diagonals, the inner over the number of gridpoints. The inner loop, the multiplication with a diagonal of K, can be parallelised by adding the appropriate directives. The third step, adding all the contributions per continent together, can be parallelised by performing the additions for the different continents in parallel. However, as in the first step, this operation is too inexpensive for parallelisation, and is therefore performed sequentially. Coarse-Grain Parallelism Coarse grain parallelism is introduced by a domain-decomposition technique [8], in combination with message passing to exchange information between subdomains. The original domain is split into equally-sized subdomains. Each subdomain is mapped onto a processor. All finite elements are uniquely assigned to a subdomain, but the subdomains share the (sea) gridpoints at the interfaces. Continents give global connections between subdomains. Hence continent values can be shared between subdomains that are not direct neighbours. We have used MPI routines [11] to implement the communication between subdomains. The vector update is trivially parallelised, it is performed per subdomain. To compute an inner product, all contributions per subdomain can be computed locally. The results must be sent to the other subdomains (global communication) and added together. We have implemented this with the MPI ALLREDUCE routine. Special attention must be paid to the values corresponding to interface nodes and to continents to assure that they are taken into account only once. In the first step of the matrix-vector multiplication all the land points in a subdomain are given their continent value. This value is always locally available, therefore no communication is necessary. The second step is the diagonalwise matrix-vector multiplication. This operation can be performed by locally multiplying with the part of the system matrix that corresponds to the subdomain. At the end of this step, communication has to be performed with neighbouring subdomains to add up the interface values. We have implemented this with the MPI SENDRECV routine. The third step is the addition of all the continent values. A specific continent may be in many, even in all domains. To exchange the local contribution of the subdomains to a continent value we therefore perform a global communication operation, so that all local continent values are send to all subdomains after which they are added together. We have implemented this with the MPI ALLREDUCE routine. Combining Finer- Grain and Coarse. Grain Parallelism The domain-decomposition approach can be combined in a straightforward way with fine-grain parallelisation by exploiting the loop parallelism (using OpenMP) per subdomain. This is a natural idea on clusters of SMP’s, where each subdomain can be mapped onto an SMP. MPI can be used for the communication between SMP’s and OpenMP on an SMP [4].
Two Level Parallelism in a Stream-Function Model
4 4.1
825
Numerical Experiments The Test Problems
The ocean model has been discretised with linear triangular elements on two different grids, the first of 1o resolution (361 × 181 gridpoints) and the second of 0.5o resolution (721 × 361 gridpoints). The values for the physical parameters have been taken from [10]. The resulting two linear systems have been solved with GMRESR, with 30 GMRES inner-iterations per outer iteration, and with truncation after 20 outer iterations. 4.2
The Cluster of SMP’s
The numerical experiments have been performed on a cluster of 8 bi-processor Pentium PC’s running under LINUX. The main features of this cluster of SMP’s are tabulated in Table 1.The Portland Group Fortran 90 compiler, which supports the OpenMP directives [2], has been used to compile the program. The message-passing library is based on MPI-CH [5]. Table 1. Main features of the cluster of Pentium PC’s SMP
Processor
Cache
Network
2 proc. Pentium III 16 KB L1 Myrinet 1 GB RAM 933 Mhz 256 KB L2 250 MB/s bandwidth 8 SMP’s 933 MFlops each way in full-duplex
4.3
Numerical Results: 1o -Resolution Problem
The total number of gridpoints for the 1o -resolution problem (including all land points) is about 65,000. The number of GMRESR iterations to solve this problem is 51, which corresponds to 1581 matrix-vector multiplications (MATVEC), a little more than 26,000 inner products (INPROD), and about 27,000 vector updates (UPDATE). Table 2. Elapsed time [s], combined OpenMP/MPI, 1o -resolution problem # Processors SOLVER MATVEC INPROD UPDATE (1*1)*1 (1*1)*2 (2*1)*2 (2*2)*2 (4*2)*2
= = = = =
1 2 4 8 16
181.8 90.5 30.0 15.6 9.2
64.9 45.3 9.6 4.7 2.5
48.2 22.4 12.3 7.1 4.9
50.8 12.8 3.8 1.9 0.9
826
M. van Gijzen
The time measurements (elapsed time) for the combination of OpenMP and MPI are given in Table 2. The first column of this Table gives the number of subdomains in longitudinal and latitudinal direction (the two numbers between paranthesis), the number of threads, and the total number of processors used. The second column, denoted SOLVER, gives the total time spent in GMRESR. The third, fourth and fifth columns give the timings (including communication time) for the three operations MATVEC, INPROD and UPDATE. At a first glance, the speed-up for pure OpenMP (first two lines of Table 2) seems to be quite satisfactory. The apparently linear speed-up, however, is largely due to the super-linear speed-up for the UPDATE operation. This super-linear speed-up can be explained by a better use of the L2 cache in the procedure to orthogonalise a new basis vector for the Krylov subspace in GMRES. The orthogonalisation of a vector w with respect to another vector v is performed in two steps, an inner product followed by a vector update. If the vectors v and w both fit in the L2 cache, they have to be loaded only for the inner-product operation and they can remain in cache for the update. The amount of memory needed to store one vector is 361 × 181 × 4 (number of gridpoints times storage for a single precision number), which is 255,2 KB, just about the size of the L2 cache. If the operations are performed in parallel by two processors, the effective vector length per processor is halved, which means that two vectors should fit in the cache. The fact that we also see a super-linear speed-up for the update-operation from 2 to 4 processors may be an indication that this is not exactly the case. A similar cache effect can be observed for the matrix-vector product. The structured matrix-vector multiplication v = Ku is composed of a sequence of multiplications with a diagonal matrix and vector updates. During these operations, the vectors u and v can remain in the L2 cache if there is enough space for three vectors, namely u and v and also for one diagonal of K. This is the case if the vectors can be distributed over four L2-caches. This explains the super-linear speed-up for the MATVEC, going from 2 to 4 processors. The adverse effects of communication are rather small. Some effect is visible in the results on 16 processors, in particular for the inner product. Table 3. Elapsed time [s], MPI only, 1o -resolution degree problem # Processors SOLVER MATVEC INPROD UPDATE (1*1)*1 (2*1)*1 (2*2)*1 (4*2)*1 (4*4)*1
= = = = =
1 2 4 8 16
181.8 109.7 38.9 14.9 9.4
64.9 27.1 13.2 4.1 2.1
48.2 35.3 12.7 7.4 5.7
50.8 35.2 9.0 1.8 0.9
Table 3 gives the time measurements if only MPI is used. The most striking feature in this Table is the poor parallel performance on 2 processors, i.e. two MPI processes share one SMP. Disregarding the cache effect in the update,
Two Level Parallelism in a Stream-Function Model
827
this was also the case for the OpenMP results given in Table 2. This problem has also been observed and analysed in [4] where it was explained by memory contention. In Table 3 we see several instances of super-linear speed-up. These cases, however, are harder to relate directly to a specific cache effect than in the combined OpenMP/MPI case. The speed-up of the time for the solution phase (with respect to the single processor time) displayed in Figure 1, shows that, for this problem, the combination of OpenMP and MPI is at least competitive with pure MPI.
Speed−up: 1−degree resolution problem MPI OpenMP/MPI 16
Speed−up
8
4
2
1
1
2
4 Number of processors
8
16
Fig. 1. Speed-up, 10 -resolution problem
4.4
Numerical Results: 0.5-Degree Resolution Problem
In this case, the total number gridpoints is about 260,000. The number of GMRESR iterations to solve this problem is 120, which corresponds to 3720 matrixvector multiplications, and a little more than 62,000 inner products and 64,000 vector updates. The time measurements for the combination of OpenMP and MPI are presented in Table 4. The 0.5o -resolution problem is four times as large as the 1o -resolution problem. For this reason we see cache effects for the update and for the matrix-vector multiplication that are similar to the 1o -resolution problem, but now on four times the number of processors. Table 5 gives the time measurements if only MPI is used. Noticeable is again, as for the 1o -resolution problem, the poor parallel scaling on a single SMP, with a speed-up of 1,6 going from one to two processors. The parallel scaling of OpenMP (see first two lines of Table 4) is even worse, with a speed-up of only 1,4.
828
M. van Gijzen Table 4. Elapsed time [s], combined OpenMP/MPI, 0.5o -resolution problem # Processors SOLVER MATVEC INPROD UPDATE (1*1)*1 (1*1)*2 (2*1)*2 (2*2)*2 (4*2)*2
= = = = =
1 2 4 8 16
1818 1310 593 216 80
643 477 167 81 26
421 303 178 67 34
560 379 180 44 9
Table 5. Elapsed time [s], MPI only, 0.5o -resolution problem # Processors SOLVER MATVEC INPROD UPDATE (1*1)*1 (2*1)*1 (2*2)*1 (4*2)*1 (4*4)*1
= = = = =
1 2 4 8 16
1818 1130 574 286 109
643 321 155 70 32
421 288 172 110 50
560 376 180 79 18
Speed−up: 0.5−degree resolution problem MPI OpenMP/MPI
16
Speed−up
8
4
2
1
1
2
4 Number of processors
8
16
Fig. 2. Speed-up, 0.50 -resolution problem.
The speed-up of the time for the solution phase (with respect to the single processor time) is shown in Figure 2. This Figure shows a significantly better performance of the combination of OpenMP and MPI on 8 and 16 processors. Note that we saw the same improved performance for the combination of OpenMP and MPI for the 10 -resolution problem on 2 and 4 processors. The timing results given in Tables 4 and 5 indicate that, as for the 10 -resolution problem, the improved performance of the combined method can be explained by a better usage of the L2-cache in the update operation and in the matrix-vector multiplication.
Two Level Parallelism in a Stream-Function Model
5
829
Concluding Remarks
We have discussed the parallelisation of a stream-function model for global ocean circulation by a combination of loop-level parallelisation using OpenMP and a domain decomposition method in which communication between subdomains is implemented with MPI. Experiments on a cluster of bi-processor PC’s show good parallel scaling properties.In our examples, the combined approach is competitive with, and often more efficient than a pure MPI-approach.
Acknowledgments. The author thanks Marielba Rojas, Bruno Carpentieri, Luc Giraud and Shane Mulligan for their valuable comments on an earlier version of this paper.
References 1. Bryan, K., and Cox, M.D.: The circulation of the world ocean: a numerical study. Part I, A homogeneous model. J. Phys. Oceanogr. 2 (1972) 319–335 2. Chandra, R., Menon, R., Dagum, L., Kohr, D., Maydan, D.J and McDonald, J.: Parallel Programming in OpenMP Morgan Kaufmann Publishers (2000) 3. Eisenstat, S.C., Elman, H.C., and Schultz, M.H.: Variational iterative methods for nonsymmetric systems of linear equations. SIAM J. Numer. Anal. 20 (1983) 345–357 4. Giraud, L.: Combining Shared and Distributed Memory Programming Models on Clusters of Symmetric Multiprocessors: Some Basic Promising Experiments. Int. J. High Perf. Comput. Appl. 16:4 (2002) 425–430 5. Gropp, W.D., Lusk, E., Doss, N., and Skjellum, A.: A high-performance, portable implementation of MPI message passing interface standard. Parallel Computing 22(6) (1996) 789–828 6. Saad Y., and Schultz, M.H.: GMRES: A generalized minimum residual algorithm for solving nonsymmetric linear systems, SIAM J. Sci. Statist. Comput. 7 (1986) 856–869 7. Van der Vorst, H.A., and Vuik C.: GMRESR: A family of nested GMRES methods, Num. Lin. Alg. Appl. 1 (1993) 1–7 8. Van Gijzen, M.B.: Parallel ocean flow computations on a regular and on an irregular grid. In: H.Liddell et al (eds.) Proceedings HPCN 96, Lecture Notes in Computer Science, Vol. 1067. Springer-Verlag, Berlin Heidelberg New York (1996) 207–212 9. Van Gijzen, M.B., Sleijpen, G.L.G., and Van der Steen, A.J.: The data-parallel iterative solution of the finite element discretization of stream-function models for global ocean circulation. In: A. Sydow, et al (eds.): Proceedings 15th IMACS World Congress on Scientific Computation, Modelling and Applied Mathematics, August 24–29, 1997, Berlin, Part III, Computational Physics, Biology, and Chemistry. Wissenschaft & Technik Verlag, Berlin, Germany (1997) 479–484 10. Van Gijzen, M.B., Vreugdenhil, C.B., and Oksuzoglu, H.: A finite element discretization for stream-function problems on multiply connected domains, J. Comp. Phys. 140 (1998) 30–46 11. Message Passing Interface Forum: MPI: A message-passing interface standard. Int. J. Supercomputer Applications and High Performance Computing, 8(3/4), 1994. (Special issue on MPI)
Scalable Parallel RK Solvers for ODEs Derived by the Method of Lines Matthias Korch and Thomas Rauber University of Bayreuth, Faculty of Mathematics and Physics {matthias.korch, rauber}@uni-bayreuth.de
Abstract. This paper describes how the specific access structure of the Brusselator equation, a typical example for ordinary differential equations (ODEs) derived by the method of lines, can be exploited to obtain scalable distributed-memory implementations of explicit Runge-Kutta (RK) solvers. These implementations need less communication and therefore achieve better speed-ups than general explicit RK implementations. Particularly, we consider implementations based on a pipelining computation scheme leading to an improved locality behavior.
1
Introduction
Several approaches towards the parallel solution of ODEs exist. These include extrapolation methods [8], relaxation techniques [2], multiple shooting [6], and iterated RungeKutta methods, which are predictor-corrector methods based on implicit Runge-Kutta methods [9,14]. Two-step RK methods based on the computation of s stage approximations are proposed in [10]. The approach exploits parallelism across the method and yields good speed-ups on a shared address space machine. A good overview of approaches for the parallel execution of ODE solution methods can be found in [2,3,4]. Most of these approaches are based on the development of new numerical algorithms with a larger potential for a parallel execution, but with different numerical properties than the classical embedded RK methods. In this paper, we consider the solution of initial value problems (IVPs) y (t) = f (t, y(t)),
y(t0 ) = y0 ,
y : IR → IRn ,
f : IR × IRn → IRn ,
(1)
by explicit RK methods. In particular, we concentrate on ODEs derived from partial differential equations (PDEs) with initial conditions by discretizing the spatial domain using the method of lines. We choose the 2D-Brusselator equation [5] that describes the reaction of two chemical substances as a representative example for such ODEs. Two unknown functions u and v represent the concentration of the substances. A standard five-point-star discretization of the spatial derivatives on a uniform N × N grid with mesh size 1/(N − 1) leads to an ODE system of dimension 2N 2 for the discretized solution {Uij }i,j=1,... ,N and {Vij }i,j=1,... ,N dUij 2 Vij − 4.4Uij + α(N −1)2 (Ui+1,j + Ui−1,j + Ui,j+1 + Ui,j−1 − 4Ui,j ) , = 1 + Uij dt dVij 2 = 3.4Uij − Uij Vij + α(N −1)2 (Vi+1,j + Vi−1,j + Vi,j+1 + Vi,j−1 − 4Vi,j ) , dt
H. Kosch, L. B¨osz¨orm´enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 830–839, 2003. c Springer-Verlag Berlin Heidelberg 2003
(2)
Scalable Parallel RK Solvers for ODEs Derived by the Method of Lines
831
which is a non-stiff ODE system for α = 2·10−3 and appropriate values of N . Non-stiff ODE systems can be solved efficiently by explicit RK methods with stepsize control using embedded solutions. An embedded RK method with s stages which uses the argument vectors w1 , . . . , ws to compute the two new approximations ηκ+1 and ηˆκ+1 from the two previous approximations ηκ and ηˆκ is represented by the computation scheme wl = ηκ + hκ ηκ+1 = ηκ + hκ
s l=1
l−1
ali f (xκ + ci hκ , wi ),
l = 1, . . . , s,
i=1
bl f (xκ + cl hκ , wl ),
ηˆκ+1 = ηκ + hκ
s
ˆbl f (xκ + cl hκ , wl ).
l=1
(3) The coefficients aij , ci , bi , and ˆbi are determined by the RK method used. If the right hand side function f of the ODE system is assumed to have an arbitrary dependence structure, in general, the process of computing the argument vectors is sequential, and only minor program modifications are possible to increase locality [12]. The discretized Brusselator equation—similar to other ODEs derived from PDEs—has a loosely coupled dependence structure that is induced by the five-point-star discretization. Thus, the evaluation of component Uij only accesses the components Vij and Uij , Ui+1,j , Ui,j+1 , Ui−1,j , Ui,j−1 , if available. Similarly, the evaluation of component Vij uses the components Uij and Vij , Vi+1,j , Vi,j+1 , Vi−1,j , Vi,j−1 , if available. When such special properties of f are known, they can be exploited to obtain faster solvers for this particular class of ODEs. In [7], a sequential pipelining computation scheme has been proposed that leads to an improved locality behavior resulting in a significant reduction of the execution time for different processors. In this paper, we develop parallel implementations for a distributed address space that exploit special properties of the right hand side function f . In contrast to a straightforward implementation that uses global communication operations, these implementations are based on single transfer operations only. The new implementations try to hide the communication costs as far as possible by overlapping communication and computation. As a result, these parallel implementations show good scalability even for large numbers of processors. We demonstrate this for an UltraSPARC II SMP, a Cray T3E-1200 and a Beowulf cluster.
2
Exploiting Specific Access Structure for Locality Improvement
Though it is possible to improve the locality of RK methods by modifications of the loop structure and other rearrangements of the code [12], we can maximize locality only by exploiting special properties of the problem to be solved. In order to store the components {Uij }i,j=1,... ,N and {Vij }i,j=1,... ,N of the Brusselator equation in a linear vector of dimension n = 2N 2 , different linearizations can be used. In our implementations, we use the mixed row-oriented storage scheme that reduces the maximum distance of the components accessed in the function evaluation of a single component: U11 , V11 , U12 , V12 , . . . , Uij , Vij , . . . UN N , VN N .
(4)
832
M. Korch and T. Rauber
Here, the evaluation of function component fl accesses the argument components l − 2N, l − 2, l, l + 1, l + 2, l + 2N (if available) for l = 1, 3, . . . , 2N 2 − 1 and l − 2N, l − 2, l − 1, l, l + 2, l + 2N (if available) for l = 2, 4, . . . , N 2 . For this access structure, the most distant components of the argument vector to be accessed for the computation of one component of f have a distance equal to 2N . We consider the division of the mixed row-oriented storage scheme (4) into N blocks of size 2N . Analyzing the data dependences occurring during the function evaluations, we can derive a pipelining computation scheme that computes all argument vectors during a single diagonal sweep [7]. Figure 1(a) illustrates the computation order of the pipelining computation scheme, and Fig. 1(b) shows the working space and the dependences of one pipelining step executed to compute block J of ηκ+1 and ηˆκ+1 . In Fig. 1(b), apart from the blocks completed during the pipelining step, all blocks are highlighted that are accessed during this step. Blocks required for function evaluations are marked by crosses and blocks updated using results of function evaluations are marked by squares. The working space of one pipelining step consists of Θ(s2 ) blocks of size 2N . This corresponds to a total of Θ(s2 N ) components. In contrast, at each stage a general implementation potentially accesses one complete argument vector during the function evaluations and all succeeding argument vectors in order to update the partial sums they hold. This corresponds to a working space of Θ(sn), which is equal to Θ(s · 2N 2 ) in the case of the Brusselator function. The reduction of the working space by the pipelining computation scheme leads to increased locality and thus to better execution times on different sequential machines [7].
ηκ = w1 w2 w3 w4
ηκ = w1 w2 w3 w4
7 11 17 23 29 35
1
2
4
3
5
8 12 18 24 30 36
6
9 13 19 25 31 37
10 14 20 26 32 38
ηκ+1
15 21 27 33 39
ηκ+1
16 22 28 34 40
ηκ+1 ηκ+1
1 2 3 4 5
N
(a) Computation order.
1 2 3
J
N
(b) Working space and dependences.
Fig. 1. Illustration of one pipelining step.
3 3.1
Exploiting Specific Access Structure for Parallelization General Parallel Implementation
The general parallel implementation (Fig. 2) based on computation scheme (3) computes the argument vectors w1 , . . . , ws sequentially one after another. To exploit parallelism, the computation of the components of the argument vectors is distributed equally among the processors. But since the access structure is not known in advance, it must be assumed that each component of f accesses all components of the argument vector. Therefore, the argument vectors must be made available by multibroadcast operations.
Scalable Parallel RK Solvers for ODEs Derived by the Method of Lines
833
multibroadcast(ηκ ); for (j = first component; j ≤ last component; j++) { compute F = hfj (t + c1 h, ηκ ); set wi [j] = ηκ [j] + ai1 F for i = 2, . . . , s ηκ+1 [j] = ηκ [j] + b0 F ; ηˆκ+1 [j] = ηκ [j] + ˆ b0 F ; } for (l = 2; l ≤ s; l++) { multibroadcast(wl ); for (j = first component; j ≤ last component; j++) { compute F = hfj (t + cl h, wl ); update wi [j] += ail F for i = l + 1, . . . , s bl F ; }} ηκ+1 [j] += bl F ; ηˆκ+1 [j] += ˆ perform error control and stepsize selection
Fig. 2. Pseudocode of one time step of a general parallel RK implementation for arbitrary right hand side functions f .
Usually, the execution time of multibroadcast operations increases linearly with the number of participating processors [15]. Therefore, this implementation is not scalable to a large number of processors. Previous experiments have shown that for ODEs resulting from the spatial discretization of PDEs only a limited speed-up can be expected (e.g., DOPRI8(7) on Intel Paragon obtained 5.1 on 8 processors, 4.6 on 32 processors). 3.2
Parallel Blockwise Implementation
The special structure of Brusselator-like functions allows the derivation of more efficient implementations by applying the division of argument vectors into blocks of size 2N . Thus, N/p adjacent blocks of each argument vector are assigned to each processor for computation. One possible parallel blockwise implementation is illustrated in Fig. 3. As the general implementation (Fig. 2), this implementation processes the stages successively. But in contrast to the general implementation, no global communication is necessary. Since only the blocks J − 1, J, and J + 1 of the previous argument vector are required to compute block J of one argument vector, it is sufficient to send the first and the last block of a processor’s part of each argument vector to its predecessor and its successor, respectively, to satisfy the dependence structure of the function. This can be done using single transfer operations like MPI Isend() and MPI Irecv(). These operations usually have execution times consisting of a constant startup time and a transfer time increasing linearly with the data size to be transferred. Consequently, the blockwise implementation has communication costs invariant to the number of processors. Using the order {first block + 1, . . . , last block − 1, last block, first block} to compute the blocks of a stage, the communication time can be hidden completely if the neighboring processors send off their first block and their last block of the preceding argument vector by a non-blocking send operation directly after their computation is finished and if the time to compute the N/p − 2 inner blocks is longer than the time needed to send two blocks through the network. The overall communication overhead in this ideal case consist of 2s times the startup time of the send operation and 2s times the startup time of the receive operation. If it is not possible to overlap communication and computation completely, the communication time increases by a fraction of 2s times the transfer time of 2N floating point values.
834
M. Korch and T. Rauber
ηκ = w1 w2 w3 w4
8 16
1
2
3
4
5
6
7
8
9 10 11 12 13 14 15 16
1
2
3
4
5
6
7
8
9 10 11 12 13 14 15 16
1
2
3
4
5
6
7
8
9 10 11 12 13 14 15 16
1
2
3
4
5
6
7
9 10 11 12 13 14 15
24 17 18 19 20 21 22 23 24 17 18 19 20 21 22 23 24 17 18 19 20 21 22 23 24 17 18 19 20 21 22 23 32 25 26 27 28 29 30 31 32 25 26 27 28 29 30 31 32 25 26 27 28 29 30 31 32 25 26 27 28 29 30 31
ηκ+1
47 33 35 37 39 41 43 45 47 33 35 37 39 41 43 45 47 33 35 37 39 41 43 45 47 33 35 37 39 41 43 45
η κ+1
48 34 36 38 40 42 44 46 48 34 36 38 40 42 44 46 48 34 36 38 40 42 44 46 48 34 36 38 40 42 44 46
1
P1
8 9
P2
16 17
P3
24 25
P4
32
Fig. 3. Illustration of the parallel blockwise implementation.
3.3
Parallel Pipelining
Though the blockwise implementation has reduced communication costs, it does not exploit locality optimally. The pipelining computation scheme [7] allows to derive parallel implementations with similar communication costs but higher locality. At each time step, the processors start the computation of the blocks (first block + i − 1), . . . , (last block − i + 1) of the argument vectors wi , i = 1, . . . , s − 1, in pipelining order. To ensure that each processor computes at least one block of ws , we assume that at least 2s blocks are assigned to each processor. In the second phase, the processors finalize their pipelines at both ends of their data area. To provide efficient communication, neighboring processors use a different order to finalize the two ends of their pipelines. Thus, when processor j finalizes the side of its pipeline with higher index, processor j + 1 finalizes the side of its pipeline with lower index simultaneously, and vice versa. Figure 4 illustrates the parallel pipelining scheme. The numbers determine the computation order of the blocks. Blocks accessed during the finalization phase and blocks transmitted between processors are highlighted. Communication is first required during the finalization of the pipelines, because function evaluations of the first and the last blocks access blocks stored in neighboring processors. When processor j computes its last block of argument vector wi , i = 2, . . . , s, it needs the first block of argument vector wi−1 of processor j +1, and vice versa. Again, we use non-blocking single send and receive operations in this transfer. If the processors work simultaneously, the computation of blocks stored in the neighboring processor that are required to perform function evaluations is finished before they are needed. In fact, one diagonal across the argument vectors is computed between the time when processor j + 1 finishes its first block of wi−1 and the time when processor j needs this block to compute its last block of wi , and vice versa. This time can be used to transfer the data required between the processors, thus overlapping communication and computation. But as the finalization phase proceeds, the length of the diagonals decreases from s − 1 to 1. Therefore, it is usually not possible to hide the transfer times at the end of the finalization phase completely. As a result, the communication costs of the parallel pipelining implementation are similar to the blockwise implementation, but the fraction of the transfer times that cannot be overlapped by computations is usually larger.
Scalable Parallel RK Solvers for ODEs Derived by the Method of Lines
ηκ = w1 w2 w3 w4
835
7
5
3
2
1
2
3
5
7
5
3
2
1
2
3
5
21 18 14 11
8
6
4 35 35
4
6
8 11 14 18 21 21 18 14 11
8
6
4 35 35
4
6
8 11 14 18 21
17 13 10
26 22 19 15 12
1
9 36 40 40 36
7 10 13 17 17 13 10
9 12 15 19 22 26 26 22 19 15 12
1
9 36 40 40 36
7 10 13 17
9 12 15 19 22 26
30 27 23 20 16 37 41 44 44 41 37 16 20 23 27 30 30 27 23 20 16 37 41 44 44 41 37 16 20 23 27 30
ηκ+1
33 31 28 24 38 42 45 47 47 45 42 38 24 28 31 33 33 31 28 24 38 42 45 47 47 45 42 38 24 28 31 33
η κ+1
34 32 29 25 39 43 46 48 48 46 43 39 25 29 32 34 34 32 29 25 39 43 46 48 48 46 43 39 25 29 32 34
1
P1
8 9
P2
16 17
P3
24 25
P4
32
Fig. 4. Illustration of the parallel pipelining implementation (I).
3.4
Parallel Pipelining with Alternative Finalization
The start-up times, which largely determine the communication costs of the specialized parallel implementations previously described, can be quite expensive on some machines. Therefore, it is reasonable to investigate different implementations of parallel pipelining which execute fewer communication operations but transfer more data. Such an implementation can be obtained by letting each processor finalize the higher end of its own pipeline and the lower end of the pipeline of its cyclic successor. Thus, to compute the stages, only one pair of communication operations is necessary to transfer data from a processor to its cyclic predecessor. This communication involves 2s + s ˆκ+1 , and s i=2 i − 1 blocks of argument vectors, s blocks of ηκ+1 , s blocks of η additional blocks (information for stepsize control). All in all, s(s + 9)/2 blocks have to be transferred. The transfer can be started when the pipeline has been initialized, but the data are not needed before the finalization starts. Hence, the diagonal sweep across the argument vectors can be executed in parallel to the data transfer. During this sweep, (N/p − 2s) · s blocks of argument vectors are computed. During the finalization of the pipelines, the processors compute parts of ηκ+1 that are required by their successors to start the next time step. Thus, another communication operation is needed to send s blocks of ηκ+1 to this processor. This is only necessary if the step is accepted by the error control, because otherwise the step is repeated using ηκ and, therefore, ηκ+1 is not needed (implementation (II)). But by exchanging the appropriate parts of ηκ+1 in every step, it is possible to hide part of the transfer time by computation. This presumes that the finalization starts at the lower end of the pipeline of the succeeding processor and proceeds toward the higher end of the pipeline of the processor considered (implementation (III)). Figure 5 illustrates the computation order of both implementations and the data transmitted.
4
Runtime Experiments and Analysis
We have executed several runtime experiments on three different parallel systems. The first machine is a shared-memory multiprocessor equipped with four Sun UltraSPARC II processors at 400 MHz. The second machine is a Cray T3E-1200 equipped with DEC Alpha 21164 processors at 600 MHz. Also, we have performed measurements on the Chemnitz Linux Cluster (CLiC)—a Beowulf cluster consisting of 528 Pentium III machines at
836
M. Korch and T. Rauber
ηκ = w1 w2 w3 w4
7
5
3
2
7
5
3
2
7
5
3
2
7
5
3
2
21 18 14 11
8
6
4 26 21 18 14 11
8
6
4 26 21 18 14 11
8
6
4 26 21 18 14 11
8
6
4 26
17 13 10
1 17 13 10
9 35 31 27 22 19 15 12
27 22 19 15 12
1 17 13 10
9 35 31 27 22 19 15 12
1 17 13 10
9 35 31 27 22 19 15 12
1
9 35 31
32 28 23 20 16 42 39 36 32 28 23 20 16 42 39 36 32 28 23 20 16 42 39 36 32 28 23 20 16 42 39 36
ηκ+1
37 33 29 24 47 45 43 40 37 33 29 24 47 45 43 40 37 33 29 24 47 45 43 40 37 33 29 24 47 45 43 40
η κ+1
38 34 30 25 48 46 44 41 38 34 30 25 48 46 44 41 38 34 30 25 48 46 44 41 38 34 30 25 48 46 44 41
1
P1
8 9
P2
16 17
P3
24 25
P4
32
Fig. 5. Illustration of the parallel pipelining implementations (II) and (III).
800 MHz connected by a Fast Ethernet network. All programs have been implemented in C and use double precision. To perform communication, the MPI library [13] has been used, which provides message passing operations for distributed-memory machines but can also be used as a means of communication between processes on a shared-memory system. As RK methods we use the DOPRI5(4) and DOPRI8(7) methods [11]. The speed-up values presented compare the parallel execution times of the different parallel implementations with the execution time of the fastest sequential implementation on the respective machine. All experiments we performed on the UltraSPARC II machine use the gridsize N = 384. The execution times of the general and the pipelining implementations measured for the two RK methods and the integration interval H = 4.0 are displayed in Tab. 1. With these parameters, the specialized implementations obtain better speed-ups than the general implementation. For example, using DOPRI5(4) and H = 4.0, speedups between 3.21 and 3.46 have been measured for the specialized implementations while the general implementation only obtains a speed-up of 1.27. Because of improved locality, the pipelining implementations are faster than the blockwise implementation, particularly for small numbers of processors. But the difference in the speed-ups decreases when the number of processors is increased. The pipelining implementations (II) and (III) both obtain smaller speed-ups than the pipelining implementation (I). The difference increases with the number of processors. Using four processors, the pipelining implementations (II) and (III) are slower than the blockwise implementation for the DOPRI5(4) method.
Table 1. Execution time (in seconds) of the parallel implementations on an UltraSPARC II SMP. Parameters Processors general blockwise pipelining (I) pipelining (II) pipelining (III)
DOPRI5(4), H=4.0, N=384 1 2 3 4 536.07 323.77 307.23 339.90 521.37 264.16 174.50 130.00 445.40 231.57 154.53 124.24 447.15 230.92 162.48 131.73 450.98 232.00 161.99 133.87
DOPRI8(7), H=4.0, N=384 1 2 3 4 863.77 511.33 447.13 476.85 832.07 428.89 295.53 218.79 676.48 346.46 234.15 184.53 678.37 350.64 241.82 198.46 674.99 351.78 242.97 197.48
Scalable Parallel RK Solvers for ODEs Derived by the Method of Lines
837
Table 2 shows the execution times measured for the gridsizes N = 384 and N = 896 and the integration intervals H = 0.5 and H = 4.0 using DOPRI5(4) and DOPRI8(7) on the Cray T3E. In all these experiments, the specialized implementations obtain much better speed-ups than the general implementation. Particularly, the experiment using DOPRI5(4) with H = 0.5 and N = 896 where many processors can be used, shows very good scalability for these implementations. Because of the precondition that at least 2s blocks must be assigned to every processor, the pipelining implementations are limited to 64 processors in this experiment. Their maximum speed-ups are in the range of 54.15 to 55.39. Using 64 processors, the blockwise implementation obtains a speed-up of 55.28. But since the limitation to at most N/(2s) processors does not apply to it, higher speed-ups can be reached with larger numbers of processors. For example, a speed-up of 109.93 has been measured using 128 processors. The scalability of the general implementation is limited to about 32 processors in this experiment. Its maximum speed-up measured is 12.15. Using the PCL library [1], we have measured that it takes about 370 cycles to perform the evaluation of one component of the right hand side function f on the Cray T3E. Since the cycle length of the processors of this machine is 1.67 ns and the maximum network bandwidth is 500 MB/s, at least 24.7 blocks of argument vectors consisting of 2N double values must be computed to hide the transfer time of one such block completely. Thus, considering the experiment using N = 896, the parallel blockwise implementation can overlap (N/p − 2)/24.7 ≈ 49 % of the communication in the case of 64 processors and 20 % when 128 processors are used. To hide all transfer times completely, less than 33 processors should be used. For the parallel pipelining implementations, the fraction of the communication time that can be overlapped by computations is smaller. The first data transfer concerning blocks of ηκ can usually be performed in the background since it can be started at the beginning of the time step. In the case of DOPRI5(4) (s = 7), N = 896 and p = 64, 56 blocks are computed in the meantime. But the first diagonal of the finalization phase only consists of s − 1 blocks. Thus, using s = 7, only 24 % of the block transfer time is covered. At each stage of the finalization phase, the length of the diagonal computed is reduced by one block. At the end of the finalization phase, only 4 % of the transfer time is overlapped. In the pipelining implementations (II) and (III), s(s + 9)/2 blocks are transferred in parallel to the computation of sN/p − 2s2 blocks. That means that 98 blocks are computed during the transfer of 56 blocks in our example with p = 32. This corresponds to an overlap of 7 %. If the current step is accepted, the parallel pipelining implementations (II) and (III) additionally need to transfer s blocks of ηκ+1 to the succeeding processor before the next time step can be started. In implementation (II), this transfer is not overlapped by computations. But implementation (III) tries to hide part of that transfer time by sending those s blocks in every step. This allows the computation of s/2(s/2 + 1) blocks in parallel to this transfer. Using s = 7, this leads to an overlap of 9 %. The results of the experiments with the parallel implementations on the CLiC are shown in Tab. 3. Due to the poor performance of global communication operations caused by the slow interconnection network, the execution time of the general implementation cannot be improved by parallel execution. Its execution time increases when two processors are used. The pipelining implementation (I) and the blockwise imple-
838
M. Korch and T. Rauber
Table 2. Execution time (in seconds) of the parallel implementations on a Cray T3E-1200. Parameters Processors general blockwise pipelining (I) pipelining (II) pipelining (III)
DOP5(4), H=0.5, N=896 1 64 128 4359.27 375.12 4598.50 74.19 37.31 4782.31 74.05 n/a 4874.49 75.62 n/a 5401.25 75.74 n/a
DOP5(4), H=4.0, N=384 1 24 64 128 1135.75 91.80 94.27 1269.78 54.74 20.85 10.61 1629.33 54.67 n/a n/a 1559.61 55.32 n/a n/a 1578.41 55.76 n/a n/a
DOP8(7), H=0.5, N=896 1 32 128 11629.54 651.80 12375.57 405.15 101.30 12702.03 400.09 n/a 12790.38 399.60 n/a 12889.71 402.05 n/a
Table 3. Execution time (in seconds) of the parallel implementations on the CLiC. Parameters Processors general blockwise pipelining (I) pipelining (II) pipelining (III)
DOP5(5), H=0.5, N=896 1 32 64 128 1141.49 986.45 35.15 18.69 10.74 1005.00 35.71 18.99 n/a 1343.48 55.42 39.80 n/a 991.33 54.75 39.20 n/a
DOP5(4), H=4.0, N=384 1 2 24 128 316.05 666.68 274.50 141.26 13.66 4.26 252.66 128.27 12.36 n/a 345.06 140.67 25.93 n/a 247.73 137.61 25.14 n/a
DOP8(7), H=0.5, N=896 1 32 128 1936.29 1724.24 59.35 17.62 1747.24 60.29 n/a 2164.65 93.86 n/a 1721.89 92.70 n/a
mentation obtain similar speed-ups as on the Cray T3E. The pipelining implementations (II) and (III) are slower. In the experiment with DOPRI5(4), H = 0.5 and N = 896, their speed-ups are 52.75 and 53.58, respectively, on 64 processors. The other pipelining implementations only reach 25.17 and 25.55. With 128 processors, the blockwise implementation even obtains a speed-up of 93.30. The part of the communication time that can be overlapped by computations is smaller than on the Cray T3E, because the processors are faster (cycle length 1.25 ns) and the network is slower (100 Mbit/s). Assuming that the evaluation of one component of f also takes about 370 cycles, 1349 blocks must be computed to hide the transfer time of one single block. Therefore, even in our experiment with the blockwise implementation using N = 896 and 64 processors where 49 % of the transfer times could be covered on the Cray T3E only an overlap of 0.9 % is possible on the CLiC.
5
Conclusions
We have derived parallel implementations of embedded RK methods for the Brusselator equation, a typical example for ODEs resulting from the spatial discretization of PDEs. These implementations require less communication than general implementations supporting arbitrary right hand side functions f . While the new parallel blockwise implementation already reduces the communication remarkably, the parallel pipelining implementations also exploit locality by reducing the working space of the algorithm. Runtime experiments confirm that the scalability of the new implementations is better than the scalability of the general implementation. Using 64 processors, speed-ups of about 55 have been measured on the Cray T3E and about 53 on the Beowulf cluster. The results on the Beowulf cluster are particularly interesting as the slow interconnection network (Fast Ethernet) makes it difficult to obtain good speed-ups. Whether the
Scalable Parallel RK Solvers for ODEs Derived by the Method of Lines
839
locality optimizations of the pipelining implementations are successful, depends on the architecture of the machine. If large numbers of processors are used, communication issues usually outweigh the acceleration achieved by locality improvements. Overlapping communication and computations showed improvements on the Cray T3E. But on the Beowulf cluster the slow interconnection network prevented significant improvements. Acknowledgment. We thank the NIC J¨ulich for providing access to the Cray T3E and the TU Chemnitz for providing access to the CLiC.
References 1. R. Berrendorf and B. Mohr. PCL - The Performance Counter Library: A Common Interface to Access Hardware Performance Counters on Microprocessors (Version 2.2). Research Centre J¨ulich, January 2003. 2. K. Burrage. Parallel and Sequential Methods for Ordinary Differential Equations. Oxford Science Publications, 1995. 3. C. W. Gear. Massive Parallelism across Space in ODEs. Applied Numerical Mathematics, 11:27–43, 1993. 4. C. W. Gear and Xu Hai. Parallelism in Time for ODEs. Applied Numerical Mathematics, 11:45–68, 1993. 5. E. Hairer, S. P. Nørsett, and G. Wanner. Solving Ordinary Differential Equations I: Nonstiff Problems. Springer–Verlag, Berlin, 1993. 6. B. M. S. Khalaf and D. Hutchinson. Parallel Algorithms for Initial Value Problems: Parallel Shooting. Parallel Computing, 18:661–673, 1992. 7. M. Korch, T. Rauber, and G. R¨unger. Pipelining for locality improvement in RK methods. In Proc. of 8th Int. Euro-Par Conf. (Euro-Par 2002), pages 724–733. Springer (LNCS 2400), 2002. 8. L. Lustman, B. Neta, and W. Gragg. Solution of ordinary differential initial value problems on an Intel Hypercube. Computer and Math. with Applications, 23(10):65–72, 1992. 9. S. P. Nørsett and H. H. Simonsen. Aspects of Parallel Runge–Kutta methods. In Numerical Methods for Ordinary Differential Equations, volume 1386 of Lecture Notes in Mathematics, pages 103–117, 1989. 10. H. Podhaisky and R. Weiner. A class of explicit two-step Runge-Kutta methods with enlarged stability regions for parallel computers. Lecture Notes in Computer Science, 1557:68–77, 1999. 11. P. J. Prince and J. R. Dormand. High order embedded Runge-Kutta formulae. J. Comp. Appl. Math., 7(1):67–75, 1981. 12. T. Rauber and G. R¨unger. Optimizing locality for ODE solvers. In Proceedings of the 15th ACM International Conference on Supercomputing, pages 123–132. ACM Press, 2001. 13. M. Snir, S. W. Otto, S. Huss-Lederman, D. W. Walker, and J. Dongarra. MPI the complete reference. MIT Press, Cambridge, Mass., second edition, 1998. 14. P. J. van der Houwen and B. P. Sommeijer. Parallel iteration of high–order Runge–Kutta methods with stepsize control. Journal of Computational and Applied Mathematics, 29:111– 127, 1990. 15. Z. Xu and K. Hwang. Early Prediction of MPP Performance: SP2, T3D and Paragon Experiences. Parallel Computing, 22:917–942, 1996.
Hierarchical Hybrid Grids as Basis for Parallel Numerical Solution of PDE Frank H¨ ulsemann1 , Benjamin Bergen1 , and Ulrich R¨ ude1 System Simulation Group of the Computer Science Department, Friedrich-Alexander University Erlangen-Nuremberg, Germany,
[email protected] Abstract. This paper describes the hierarchical hybrid grid framework, which is a storage scheme for the efficient representation and application of discrete operators and variables on globally unstructured but patchwise regular grids. By exploiting patch-wise regularity it overcomes the performance penalty commonly associated with unstructured grids. Furthermore, it is well suited for distributed memory parallel architectures.
1
The Project Goal
The aim of the hierarchical hybrid grid approach is to perform multilevel algorithms efficiently on unstructured grids. However, it is well known that operations on unstructured grids achieve a lower run-time performance as measured in GFLOP/s as operations on structured grids. The main advantage of the structured operations is that regular patterns in the problem are known at compile time and can therefore be exploited for optimisation purposes. Hence, our strategy to reduce the performance penalty associated with unstructured grids is to introduce regularity in the computations. The importance of regularity as a key to high performance is also investigated by the FEAST [1] and ExPDE [5] projects, amongst others. Current versions of the hierarchical hybrid grid library have been used in projects such as natural attenuation simulations in environmental sciences, see [2].
2
The Hierarchical Hybrid Grid Concept
To achieve our aim, we combine and extend techniques from geometric multigrid (generation of nested grids) and computational fluid dynamics (multiblock grids) to construct grids that exhibit hierarchical, patch-wise regular structures in a globally unstructured, hybrid grid. The construction of a hierarchical hybrid grid is illustrated in figure 1. Thus, by construction, our approach combines the performance of operations on structured regions with the main advantage of
This project is funded by the KONWIHR grant gridlib of the Bavarian High Performance Computing Initiative.
H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 840–843, 2003. c Springer-Verlag Berlin Heidelberg 2003
Hierarchical Hybrid Grids as Basis for Parallel Numerical Solution
841
Fig. 1. From left to right: The hybrid input grid and the first three levels of refinement. After two levels of refinement (second from right) each patch has a structured interior which can be exploited to increase performance.
unstructured grids, namely the ability to represent complex problem domains. For further background information see [4]. It has to be pointed out that the patch-wise regularity of the refined grid levels alone does not produce an automatic performance improvement. The algorithms and the storage schemes have to adapted to reflect the structure. The standard way to represent a differential operator discretized by linear finite elements on a hierarchical hybrid grid is to store the stencil weights for each point in some sparse matrix format. Inside a volume cell, a cube say, the shape of the stencil is constant for all interior points. If, furthermore, the problem is linear with constant problem parameters inside the volume cell, then the entries of the stencil are also constant for all points in the cube. Hence, in this ideal case, the application of the operator to a variable can be reduced to the application of one stencil to all points inside the cell. Similar considerations apply to other cell types (prisms, tetrahedra) and faces (triangles, quadrilaterals) in 3D grids. Vertices and edges have to be treated in an unstructured fashion, given that the unstructured nature of the base grid manifests itself at these objects. The efficient way of representing the operator is complemented by a storage scheme for the unknowns that allows index arithmetic to access neighbouring values, thus doing away with the need to store neighbourhood information explicitly.
3
Performance Results
In this section, we present performance results on different current architectures to show that the approach performs well on PCs and on supercomputers alike. Given that the aim is to make better use of the individual processors in a parallel application, we concentrate on MFLOP rates and not on the total run time of the applications, thus neglecting network influences on the parallel performance. The PCs are equipped with Intel PentiumIV processors, running at 2.4 GHz and a theoretical peak performance of 4.8 GFLOP/s. The processors of the Hitachi SR8000 supercomputer run at 375 MHz with a theoretical peak performance of 1.5 GFLOP/s.
842
F. H¨ ulsemann, B. Bergen, and U. R¨ ude
Table 1. Performance of Gauß-Seidel iterations on a hexahedral element using a 27 point stencil. The results for the different implementations are given in MFLOP/s. Refinement levels #unknowns PentiumIV SR8000
CRS HHG CRS HHG
4 5 6 7 3375 29791 250047 2048383 328 352 347 352 1278 1036 1106 980 42 43 44 45 146 333 530 443
The grids in the computations consist of hexahedra and quadrilaterals. In order to illustrate the maximum performance that can be achieved by exploiting regular structures, we concentrate on the case where the operator indeed reduces to a single stencil per object. The MFLOP rate is computed by dividing the number of floating point operations on that partition by the process time. The process time in turn is computed as the quotient of the number of all cycles used by the section of the process being assessed over the number of cycles per second. On the Intel platform, the PAPI library [3] was used to count the number of floating point operations and the number of cycles, while on the Hitachi the profiler of the compiler suite was employed. We start with a comparison of the performance of Gauß-Seidel iterations on a cube, once implemented with the common compressed row storage format, the other time using stencil operations. As table 1 shows, the regular implementation is significantly faster on both tested platforms. More complex algorithms also achieve good performance, as can be seen from table 2 which gives the results for the conjugate gradient algorithm and a V(2,2) multigrid cycle. The example of the conjugate gradient method demonstrates that even algorithms with few matrix-vector operations per iteration gain in speed. To support our claim that the increase in the overall performance is due to the matrix-vector product, we include the performance of the daxpy-operation, which is the main other component in the algorithms and which has not been optimised yet.
4
Conclusions
We have presented a framework for the generation and exploitation of patch-wise regular structures from an unstructured input grid, thus preserving geometric flexibility while allowing efficient computations. Our experiments show that a substantial gain can be expected on all platforms over standard unstructured implementations. While architectures like the Intel PentiumIV permit a speed up factor of up to 3, the results on the Hitachi SR8000 show an improvement of a factor close to 10. The implementation does not yet realize the full scope of the concept. The extension to different cell types, to problems with variable coefficients and the inclusion of grid adaptivity is ongoing work.
Hierarchical Hybrid Grids as Basis for Parallel Numerical Solution
843
Table 2. Performance of conjugate gradient (CG) and geometric multigrid (GMG) algorithms, again using a 27 point stencil. These results were obtained on a single node in a MPI-parallel computation. The partition consisted of one cube with six remote faces. By daxpy we denote the operation of adding one vector, scaled by a scalar, to another vector. M-V product denotes the matrix-vector multiplication. All results are again given in MFLOP/s. Refinement levels #unknowns PentiumIV
SR8000
daxpy M-V product CG V(2,2) GMG daxpy M-V product CG V(2,2) GMG
4 5 6 7 3375 29791 250047 2048383 324 141 152 159 1345 1372 1216 1183 280 441 549 571 190 441 632 749 58 102 158 209 186 430 672 847 22 113 247 391 24 97 243 392
Acknowledgements. The authors acknowledge the financial support through KONWIHR project gridlib of the Bavarian High Performance Computing Initiative, which also provided the access to the Hitachi SR8000.
References 1. Becker, C., Kilian, S., Turek,S.: Some concepts of the software package FEAST, In: Palma, J., Dongarra, J., Hernandez, V. (Eds.): Proceedings Vector and Parallel Processing – VECPAR98. Lecture Notes in Computer Science 1573, 271–284 Springer-Verlag, 1999 2. Bergen, B.: Hierarchical Hybrid Grids, Technical Report 03-5, System Simulation Group, University Erlangen, 2003 3. Browne, S., Dongarra, J., Garner, N., Ho, G., Mucci, P.: A Portable Programming Interface for Performance Evaluation on Modern Processors, The International Journal of High Performance Computing Applications 14:3 (2000), 189–204 4. H¨ ulsemann, F., Kipfer, P., R¨ ude, U., Greiner, G.: gridlib: Flexible and efficient grid management for Simulation and Visualization, In: Sloot, P., Tan, C.J.K., Dongarra, J., Hoekstra, A. (Eds.): Computational Science – ICCS 2002. Lecture Notes in Computer Science 2331, 652–661 Springer-Verlag, 2002 5. Pflaum, C.: Semi-unstructured grids, Computing, 67 (2001), 141–166
Overlapping Computation/Communication in the Parallel One-Sided Jacobi Method El Mostafa Daoudi, Abdelhak Lakhouaja, and Halima Outada University of Mohammed First, Faculty of Sciences Department of Mathematics and Computer Science LaRI Laboratory, 60 000 Oujda, Morocco e-mail : {mdaoudi,lakhouaja,outada}@sciences.univ-oujda.ac.ma
Abstract. In this work, we propose some techniques for overlapping the communication by the computation in the parallelization of the one-sided Jacobi method for computing the eigenvalues and the eigenvectors of a real and symmetric matrix. The proposed techniques are experimented on a cluster of PCs and on the parallel system TN310.
1
Introduction
In this paper, we study, on distributed memory architecture, the parallelization of the one-sided Jacobi method for computing the eigenvalues and the eigenvectors of a real symmetric matrix A of size n. The one-sided algorithm is better suited to parallel computers [2,6], since it needs less communications than the classical Jacobi method (”two-sided Jacobi method ” [3]). It only needs one-to-one communications type (column translation) in contrast to the two-sided Jacobi method which needs, in addition, global communications (rotations broadcast) [8,4]. On the other hand, the computation time complexity of the one-sided and the two sided (without exploiting the symmetry) algorithms are equivalent for computing the eigenvalues and the eigenvectors. Our objective in this study is to propose some techniques for overlapping the communications by the computations during the translation phase by carrying out simultaneously the update and the translation of one column [5]. We consider a distributed memory architecture composed of p processors, denoted by Pm , 0 ≤ m ≤ p − 1. The communication cost of L data between two neighbor processors is modeled by comm(L) and the computation time of one floating point operation is designed by ω.
2
Sequential Algorithm
The basic idea of the Jacobi method [3] consists in constructing the sequences of matrices {A(k+1) = J (k) A(k) J (k) } and {U (k+1) = U (k) J (k) } which converge respectively to a diagonal matrix that contains the eigenvalues and
Supported by the “Comit´e mixte Franco-Marocain”, AI no : MA 01/19, and by the European INCO-DC Program, “DAPPI” Project
H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 844–849, 2003. c Springer-Verlag Berlin Heidelberg 2003
Overlapping Computation/Communication
845
to a matrix that contains the eigenvectors. Where A(1) = A, U (1) = In and J (k) = J(i, j), for 1 ≤ j < i ≤ n, is a Jacobi rotation, in the (i, j) plane, chosen in (k) (k) (k) (k) (k) order to annihilate ai,j = aj,i . It is completely determined by ai,i , aj,j and ai,j . The one-sided Jacobi method: The application of the rotation J(i, j) to the matrix, requires to update the columns i and j of the matrix and by symmetry the rows i and j. This update needs to exchange the rotations, which leads to an all-to-all communication type (global synchronization). In order to avoid the global synchronizations, one constructs the sequence {A¯(k+1) = A¯(k) J (k) } instead of the sequence {A(k) }, where A¯(1) = A. At each step k of the algorithm, one computes one rotation (determination of J (k) ) and one updates two columns of A¯(k) and two columns of U (k) . Since the determination of J (k) = J(i, j), needs (k) (k) (k) the knowledge of the coefficients ai,i , aj,j and ai,j , and since, we only know (k) (k) (k) (k) A¯(k) = (A¯1 , · · · , A¯n ) and U (k) = (U1 , · · · , Un ), these coefficients can be (k) (k) (k) computed by using the following relation [6]: am,l = (Um ) A¯l for all 1 ≤ m, l ≤ n which requires 6nω.
3
Study of the Parallelization of the One-Sided Jacobi
We consider a column distribution of A and U . Each processor Pm , for 0 ≤ m ≤ p − 1, holds np columns of A, denoted by A¯m1 (j) , A¯m2 (j) and np columns of U n . Initially, we have m1 (i) = m np + i, denoted by Um1 (j) and Um2 (j) , for 1 ≤ j ≤ 2p n n n for 1 ≤ i ≤ 2p and m2 (i) = m p + i, for 2p + 1 ≤ i ≤ np , which corresponds to np consecutive columns of A and np consecutive columns of U . The elements are annihilated according to the block odd-even ordering presented in [6]. During each step, all possible rotations are performed as shown on figure 1 for n = 12 and p = 3. For example, at the first step, P1 holds the columns 5, 6, 7 and 8, then it performs all possible rotations (5, 6), (5, 7), (5, 8), (6, 7), (6, 8) and (7, 8). At the second step, it holds the columns 1, 2, 7 and 8, then it performs all possible rotations (1, 7), (1, 8), (2, 7) and (2, 8). The rotations (1, 2) and (7, 8) are done during the first step. m 1(j)
1 2
5 6
9 10
m 2(j)
3 4
7 8
11 12
First step (odd step)
3 4
1 2
5 6
7 8
11 12
9 10
Second step (evenstep)
Fig. 1. The first and second steps of the block ordering for n = 12 and p = 3. The arrows show the sense of the column translation.
There are two phases for the parallelization. At each step, each processor: – Phase 1: columns:
computes all possible local rotations and updates its local
846
E.M. Daoudi, A. Lakhouaja, and H. Outada
• in the first step, it computes all possible rotations J(m1 (i), m2 (j)), n , which corJ(m1 (i), m1 (j)) and J(m2 (i), m2 (j)) for 1 ≤ i < j ≤ 2p n
( n −1)
responds to p p2 rotations. n • during the other steps, for 1 ≤ i, j ≤ 2p , it computes the rotations n 2 J(m1 (i), m2 (j)), which corresponds to ( 2p ) rotations. – Phase 2: communicates only with its two neighbors in order to send and n n receive 2p columns of A¯ and 2p columns of U , in order to complete a sweep.
4
Overlapping the Communications
The basic idea to overlap the communication with the computation is to delay the computation of some rotations in order to be computed simultaneously during the translation phase. We assume that at one given step, the processor Pm , for 0 ≤ m ≤ p − 1, holds the columns A¯m1 (j) , A¯m2 (j) , Um1 (j) and Um2 (j) , for n 1 ≤ j ≤ 2p : • it computes all possible rotations (illustrated by black squares in the figures n (rotations illustrated 2) except the rotations J(m1 (j), m2 (j)), for 1 ≤ j ≤ 2p by white squares in the figures 2). • it updates the corresponding columns.
m 1(1)
m 1(2)
m 1(3)
m 2(1)
m 2(2)
m 2(3)
m 1(1)
m 1(2)
m 1(3)
m 2(1)
m 2(2)
m 2(3)
1 0 1 0 1 0 0 1
1 0 0 1 1 0 1 0
1 0 0 1 1 0 0 1
1 0 1 0
1 0 1 0
1 0 00 0 11 1 00 11 1 0 0 1
11 00 00 11
1 0 0 1 1 0 0 1
(a) first step
1 0 0 1
1 0 1 0 1 0 0 1
1 0 0 1
(b) other steps
Fig. 2. Computation of the rotations during the overlapping phase.
The translation will be performed simultaneously with the computation of n . the remaining rotations J(m1 (j), m2 (j)), for 1 ≤ j ≤ 2p 4.1
First Strategy
The theoretical study of this strategy is developed in [5]. Each processor computes J(m1 (j), m2 (j)) and updates A¯m2 (j) and Um2 (j) , then, simultaneously:
Overlapping Computation/Communication
847
– updates A¯m1 (j) and Um1 (j) – sends A¯m2 (j) and Um2 (j) according to the odd-even ordering. The update of one column needs 3nω and the computation of one rotation needs 6nω, while the communication of two columns needs comm(2n) Lemma: If 6nω ≥ comm(2n), all the communications can be overlapped by the computations. 4.2
Second Strategy
Before translating columns, we divide each column of A¯ and U into q blocks each of size nq . The translation of each column is done in pipeline, as follow: Each processor: • computes J(m1 (1), m2 (1)) and updates the first block of the columns A¯m1 (1) , Um1 (1) , A¯m2 (1) and Um2 (1) . n • For 1 ≤ j ≤ 2p − 1: * for 2 ≤ i ≤ q, simultaneously: – updates the ith blocks of A¯m1 (j) , Um1 (j) , A¯m2 (j) and Um2 (j) . – sends the (i − 1)th blocks of A¯m2 (j) and Um2 (j) . * simultaneously: – computes J(m1 (j + 1), m2 (j + 1)) and updates the first block of the columns A¯m1 (j+1) , Um1 (j+1) , A¯m2 (j+1) and Um2 (j+1) . – sends the last blocks of A¯m2 (j) and Um2 (j) . n • for j = 2p * for 2 ≤ i ≤ q, simultaneously: – updates the ith blocks of A¯m1 (j) , Um1 (j) , A¯m2 (j) and Um2 (j) . – sends the (i − 1)th blocks of A¯m2 (j) and Um2 (j) . * sends the last blocks of A¯m2 (j) and Um2 (j) . The update of one block of a column needs 3 nq ω, while the communication of two blocks needs comm(2 nq ). If we assume that the communication cost of L data between two neighbor processors is modeled by comm(L) = β + Lτ , where β is the start-up time and τ is the time to transmit one data, then the parallel time which necessitates this part of the parallel algorithm is modeled by: T (q) = 6nω + 12nω q + n 12nω − 1)[(q − 1) max(β + 2nτ ( 2p q , q ) + max(β + 12nω 2nτ (q − 1) max(β + 2nτ q , q ) + (β + q )
2nτ q , 6nω
+
12nω q )]+
Lemma: The optimal size which minimize the parallel time is given by: ) 12nω ], n) if β + 2nτ 1. q ∗ = min([ 2n(6ω−τ q ≤ q β 6pω 12nω ∗ 2. q = min([2 ≤β+ β ], n) if 6nω + q 2nτ 12nω 3. if 12nω ≤ β + ≤ 6nω + : q q q
2nτ q
−2nτ ], n) if nτ ≤ 6nω + 2pτ a) q ∗ = min([ 12nω+4pτ β 6ω−τ ∗ b) q = min([2n β−6nω ], n) if β − 6nω ≥ 0 and 6ω − τ ≥ 0
The proof of this lemma is straightforward by derivating the function T (q).
848
5
E.M. Daoudi, A. Lakhouaja, and H. Outada
Experimental Results
The proposed techniques have been implemented on our local parallel machine TN310 based on Transputers and on the cluster of PCs installed at IMAGGrenoble-France [1]. Our objective is not to compare the obtained performances on the two different parallel systems, but to compare the performances obtained of the proposed techniques on each system. For the numerical tests, we have used the Frank matrix defined by: A = (ai,j ), where aij = n − max(i, j) + 1, for i, j = 1, 2, ..., n, which the eigenvalues are given by: λk = 2(1−cos(12k−1 π)) for 2n+1
k = 1, 2, ..., n. In table 1 (resp. table 2), we compare the execution times for one sweep on the cluster (resp. TN310) without overlapping (n ov) and the first strategie of the overlapping (str1). The obtained results on the cluster show that the use of the techniques of overlapping have weakly improved the execution time. On the TN310, the results show that good improvements are obtained, but the communications are not completely overlapped. This corroborate with the theoretical study. Note that the estimated machine parameters, for the TN310, under PVM environment, are β = 0.002s, τ = 22µs and ω = 0.25µs. Table 1. Execution times in seconds for one iteration on the cluster n p 2 4 8 16 32
128 n ov str1 0.06 0.05 0.04 0.04 0.03 0.03 0.04 0.04 0.03 0.03
256 n ov str1 0.64 0.66 0.25 0.26 0.16 0.16 0.13 0.12 0.12 0.11
512 n ov str1 6.62 5.81 3.08 2.97 1.56 1.37 0.68 0.69 0.49 0.46
768 n ov str1 20.32 19.81 10.39 10.1 5.65 5.25 2.9 2.72 1.39 1.4
1024 n ov str1 48.57 47.07 24.11 23.62 13.11 12.31 7.14 6.66 3.61 3.51
2048 n ov str1 391.51 377.25 191.81 190.51 102.45 97.52 53.7 51.56 28.83 27.3
Table 2. Execution times in seconds for one iteration on the TN310 n p 2 4 8 16
128 n ov str1 14.42 10.35 6.65 5.47 3.94 3.55 3.9 2.8
256 n ov str1 109.05 72.32 45.42 36.66 23.53 20.38 13.58 12.52
512 768 1024 n ov str1 n ov str1 n ov str1 941.17 672.46 471.77 374.81 1562.94 1267.31 172.34 140.71 791.51 472.02 1857.85 1576.82 89.76 77.03 286.9 249.29 669.22 576.8
The table 3, shows that the lower execution time, on 16 processors, for the second strategie, is obtained for q = 1. This means that it is not necessary to subdivide the columns to be translated. This result corroborates with the theoretical study. Note that the algorithm for the case q = 1 is not identical to the algorithm proposed for the first strategie.
Overlapping Computation/Communication
849
Table 3. Execution times on the TN310, with pipeline for 16 processors, using different values of q
q 1 2 4 8
6
128 2.7 3.23 4.33 6.43
256 11.49 12.94 14.91 19.43
n 512 74.57 76.94 81.96 89.81
768 236.48 240.21 252.3 264.92
1024 554.17 554.82 564.41 572.7
Conclusion
In this paper, we have proposed two algorithms, which enable the overlapping of the communications by the computations in the parallelization of the one-sided Jacobi method. Theses techniques are experimented under MPI on a cluster of PCs and under PVM on the parallel system TN310, since MPI is not implemented on our system. The experimental execution times obtained on the TN310 corroborate with the theoretical results. The results obtained on the cluster are not those expected. We think that the overlap is not available with this platform. The proposed algorithms can be easily extended to the block one-sided Jacobi version.
Acknowledgment. The authors thanks the anonymous reviewers for there valuable comments.
References 1. http://icluster.imag.fr, “The icluster project”. 2. J. Cuenca, D. Gim´enez, “Implementation of parallel one-sided block Jacobi methods for the symmetric eigenvalue problem”, ParCo’99, Imperial College Press (2000), 291-298. 3. G.H. Golub, C.F. Van Loan, “Matrix computation”, Johns Hopkins University Press, 1989, 2nd edition. 4. E.M. Daoudi, A. Lakhouaja, “Exploiting the symmetry in the parallelization of the Jacobi method”, Parallel Computing 23 (1997), 137–151. 5. E.M. Daoudi, A. Lakhouaja, H. Outada, “Study of the parallel block one-sided Jacobi method”, HPCN’2001, LNCS 2110 Springer Velag, 2001, 454–463. 6. P.J. Eberlein, H. Park, “Efficient implementation of Jacobi algorithms and Jacobi sets on distributed memory architectures”, Journal of Parallel and Distributed Computing 8 (1990), 358–366. 7. F.T. Luck, H. Park, “On parallel Jacobi orderings”, SIAM J. Sci Stat. Comp. 10 (1989), 18–26. 8. M. Pourzandi, B. Tourancheau, “A parallel performance study of Jacobi-like eigenvalue solution”, In Proceeding of First International Meeting on Vector and Parallel Processing, Porto, Portugal, September 1993.
Topic 12 Architectures and Algorithms for Multimedia Applications Ishfaq Ahmad, Pieter Jonker Bertil Schmidt, and Andreas Uhl Topic Chairs
In the recent years multimedia technology has emerged as a key technology, because of its ability to represent information in disparate forms as a bit-stream. This enables data from text to video and sound to be streamed, stored, processed, and delivered in digital form. A great part of the current research community effort has emphasized the delivery of the data as an important issue of multimedia technology. However, in the long run, also the creation, processing, and management of multimedia will most likely dominate the scientific interest. The aim to deal with information coming from video, text, and sound may result in a data explosion. The requirement to store, process, and manage large data sets naturally leads to the consideration of programmable parallel processing systems as strong candidates for supporting and enabling of multimedia technology. This makes, together with the inherent data parallelism in the data types, multimedia computing a natural application area for parallel processing. Concepts developed for parallel and distributed algorithms in general are quite useful for the implementation of distributed multimedia systems and applications. And hence, also the adaptation of these general methods to distributed multimedia systems is an interesting topic worth while studying. This year, 9 papers discussing multimedia technology topics were submitted. Each paper was reviewed by at three reviewers and, finally, we selected 3 regular and 2 short papers, on the topics of: The design aspects of video mapping and network traffic balancing improving Double P-Tee architectures; An investigation on transmission schedulers that reduce the traffic burstiness in a Server-less Video-on-Demand System on Internet-like network topologies; A study on distributing multimedia content over peer-to-peer network by using a scheduling scheme that results in minimum buffering delay. A dynamic multicasting policy based on a phased proxy caching to reduce network bandwidth from a server to its clients. An enhanced register file architecture that performs either matrix multiplications or can be used to enlarge register bandwidth. We would like to thank all authors who submitted a contribution, the EuroPar Organizing Committee and the referees whose efforts have made this conference and this Topic possible.
H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, p. 850, 2003. c Springer-Verlag Berlin Heidelberg 2003
Distributed Multimedia Streaming over Peer-to-Peer Networks Jin B. Kwon1 and Heon Y. Yeom2 1
2
Sunmoon University, Dept. of Computer Science, Asan, Chungnam, 336-708, South Korea
[email protected] Seoul National University, Dept. of Computer Science, Seoul, 151-742, South Korea
[email protected] Abstract. A peer-to-peer model is very useful in solving the server link bottleneck problem of a client-server model. In this work, we discuss the problems of distributing multimedia content over peer-to-peer network. We focus on two problems in peer-to-peer media content distribution systems. The first is the transmission scheduling of the media data for a multi-source streaming session. We present a sophisticated scheduling scheme, which results in minimum buffering delay. The second problem is on the fast distribution of media content in the peer-to-peer system that is self-growing. We propose a mechanism accelerating the speed at which the system’s streaming capacity increases.
1
Introduction
It is generally believed that the streaming media will constitute a significant fraction of the Internet traffic in the near future. Almost all of the existing works on multimedia streaming is based on client-server models. Since multimedia streaming requires high bandwidth, server network bandwidth runs out rapidly if unicast client-server model is used. Single source multicast is one of the solutions that use a single stream to feed all the clients. The deployment of IP multicast has been slowed by difficult issues related to scalability, and support for higher layer functionality like congestion control and reliability. A peer-topeer(P2P) model is ideal as a model to solve the server link bottleneck problem. In the P2P model, multimedia contents are distributed by using the bandwidth of the clients themselves. The clients in P2P systems contribute resources to the community and in turn use the resources provided by other clients. More specifically, supplying peers holding a certain media file may stream it to requesting peers. Thus, data traffic is not localized on a specific site since the peers cooperate for sharing contents. It is typical that there is no central server that holds the contents, and peers work on an equal footing. How the contents exist at first in the P2P system is another question. We assume that there are seed peers. Examples of P2P content distribution systems include Napster[2], Gnutella[1], and so on. H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 851–858, 2003. c Springer-Verlag Berlin Heidelberg 2003
852
J.B. Kwon and H.Y. Yeom
There has been much work on P2P systems in recent years[3,6,7,8]. Those works dealt mainly with data lookup and storage management in a general P2P system. The problems on distributing streaming media over P2P network have also been studied in [4,5,9]. However, we dare say that the work on P2P media streaming systems is still in the early stage, and there is still some room for improvement of performance and generalization of model. In this paper, we focus on two problems in P2P media content distribution systems. We first concentrate on the transmission scheduling of the media data for a multi-supplier P2P streaming session. More specifically, given a requesting peer and a set of supplying peers with heterogeneous out-bound bandwidth, the problem is how to schedule the segments of the media data using multiple channels respectively established with each supplying peer. A buffering delay is determined by the streaming schedule of each channel. The transmission scheduling has already been studied in [9], where Xu et al. presented OTS as its solution, which is optimal when the length of segments, the scheduling units, is identical. We present another scheduling scheme called fixed-length slotted scheduling(FSS), which further reduces the buffering delay. Unlike OTS, FSS employs variable length segments whose size is determined based on the bandwidth with which each segment is transmitted. The second problem is the fast distribution of media contents. Initially, there are only a few seed peers holding a content, and non-seed peers request the content as requesting peers. The P2P system is self-growing since the requesting peers can become supplying peers after they receive all the data. However, in the beginning, since there is only a small number of peers holding the content, the system can accommodate only a limited request arrival rate. The number of supplying peers grows as the content spreads out, and the system would eventually be able to service all the requests from other peers. Therefore, for a given arrival rate, it is very important to convert the requesting peers to the supplying peers in a short time so that all the incoming requests can be serviced. We have come up with a mechanism accelerating the speed at which the P2P system capacity increases, called FAST. It accelerates the distribution speed by allowing requesting peers that satisfy a condition to supply contents to other peers. At first, we define a P2P media streaming model and state our assumptions. Our model is more practical and more general than the models of the previous works. A set of peers that can supply a media data is defined as the candidate set of the media content. A requesting peer selects its supplying peers from the set, opens a channel with each selected supplying peer, and requests the data segments from them according to a scheduling mechanism. The requesting peer receives the data from multiple channels and store them in its local storage, and then the peer may become a candidate of the media content. Note that a supplying peer can supply the media data to multiple requesting peers. Since the content searching problem in P2P network is another research issue, we assume that a requesting peer can get the candidate peer set and the information about resource usage of each peer using an appropriate searching mechanism. Let γ denote the playback rate of the media data. Each requesting peer Pr has an
Distributed Multimedia Streaming over Peer-to-Peer Networks
853
in-bound bandwidth of Rin (r) and an out-bound bandwidth of Rout (r). Peers are heterogeneous in their in-bound and out-bound bandwidth. We assume that 0 < Rin (r) ≤ γ and Rout (r) > 0.
2
Transmission Schedule
In this section, we study the problem of media data transmission scheduling. The problem is stated as follows: For a requesting peer Pr and n channels, to determine the data segments to be transmitted over each channel and the transmission order of the segments. The goal is to minimize buffering delay while ensuring a continuous playback at Pr , with minimum buffering delay. The buffering delay is defined as the time interval between the start of streaming and the start of playback at Pr . How to select the supplying peers of Pr is dealt with in Section 3. If the data is transmitted over multiple channels of various bandwidth lower than γ, the buffering delay is inevitable. That is because the transmission order of data is not the same as the playback one. Therefore, a well-devised data transmission schedule is essential in reducing the buffering delay. A function p(t) is defined as the amount of data being played for t seconds since the beginning of the playback. And a function d(t) is defined as the amount of consecutive data from the beginning of the media file received for t seconds since the beginning of the streaming, at Pr . Since the data is assumed to be encoded in CBR, we express the amount of data in seconds. The data amount of k means the amount to be played for k seconds. To ensure the continuous playback, the following condition must be satisfied: ∀t ≥ 0, d(t) ≥ p(t).
(1)
Fig. 1(a) illustrates the increasing shape of p(t) and d(t) for the example of OTS[9]. In the figure, there are four channels with bandwidth of γ2 , γ4 , γ8 and γ γ 8 , respectively, and the requesting peer schedules the transmission of eight 8 second segments over the channels, according to OTS. As shown in the figure, the buffering delay δ is required to satisfy the condition of Eq.(1). δ is 3L 8 in this example. Hence, p(t) can be expressed as follows: p(t) = min{max{t − δ, 0}, L}, t ≥ 0,
(2)
where δ is the maximum of the difference between the dashed line(y = t) and d(t). That is, δ = maxt≥0 {t − d(t)}. The closer d(t) is to the straight dashed line, the smaller the buffering delay. A well-designed scheduling can make d(t) close to the straight line. We propose the fixed-length slotted scheduling(FSS) as such a scheduling scheme. To increase d(t) linearly, the requesting peer Pr should receive the data in a sequential order. Based on this idea, FSS assigns the data to fixed-length slots of each channel, and the one-slot data chunks are defined as the data segments of FSS. Since the bandwidth of each channel is variable, the segment length also varies according to the channel bandwidth to which the data segment is assigned. And, the variable-length segments are assigned to the channels in a
J.B. Kwon and H.Y. Yeom
data amount
data amount
854
L
L
p(t)
d(t) p(t)
d(t)
0
t
delay
ch.1
ch.2 ch.3 ch.4
1
2
8
4
3
7 6 5
0 delay
t
ω
ch.1
1
5
9
13
ch.2
2
6
10
14
ch.3 ch.4
3 4
7 8
11 12
15 16
(a) OTS[9]
(b) FSS Fig. 1. Transmission Schedules
round-robin fashion. When the slot length is ω and i-th channel bandwidth is Bi , the segment length of channel i is ωBi . Fig. 1(b) illustrates the concept of FSS, where the number of channels and the bandwidth of each channel are the same as those of Fig. 1(a). The slot length ω is L4 . In this example, the buffering delay of FSS is L8 , which is only a third of that of OTS. Let us find the buffering delay, δ, of FSS. When the number of channels is n n, the aggregated in-bound bandwidth B ∗ = i=1 B i . Then, d(t) is: ∗ B − B1 t B1 ω, L}. (3) d(t) = min{ t + γ γ ω By expanding Eq.(1) with the d(t) of Eq.(3) and the p(t) of Eq.(2) and solving for δ, we can obtain the minimum δ. The detailed derivation is omitted here. The minimum buffering delay δ is: γ B ∗ − B1 − 1 L + (4) δ = ω, if B ∗ ≤ γ. B∗ γ From the above equation, δ depends on B1 and ω. This relationship is such that δ becomes smaller as B1 gets greater and ω gets smaller. Thus, FSS chooses the channel with the maximum bandwidth as the first channel to minimize the buffering delay. When the aggregated bandwidth is equal to the playback rate, the shorter the slot length is, the shorter the buffering delay gets. However, the slot length is required to be long enough to cover the fluctuation of bandwidth and the overhead of transmission, delivery processing, and so on. Also, if it is too smaller than the maximum packet size of the underlying physical network, the network utilization would be low and the bandwidth would be wasted. Therefore, the slot length is a system parameter that should be carefully determined.
Distributed Multimedia Streaming over Peer-to-Peer Networks
3
855
Fast Distribution
In [9], the candidate set consists of only the peers holding the whole media file. However, some of the peers while downloading the media data may be able to supply the data to requesting peers. Since the system capacity is proportional to the size of the candidate set, it would be beneficial to make the candidate set as large as possible. If it has a large candidate set, it would be easier for a requesting peer, Pr , to acquire sufficient bandwidth, and Pr would become a candidate peer within a short time. Thus, our goal is to find the peers that have enough data to act as supplying peers, in order to make the candidate set as large as possible. A peer holding the whole media file is defined as a mature peer, and a peer being downloading the media data is defined as an immature peer. Fig. 2 shows Pr and its supplying peers. The shaded ones indicate mature peers, and the white ones indicate immature peers. Let di (t) be d(t) of a peer Pi , which we defined above, and we define another function as follows. – xi (t, r): when Pi is assumed to be selected as a supplying peer of a requesting peer Pr , the position within the media file of the data to be requested to transmit at time t. According to FSS, a supplying peer does not transmit the data continuously in a consecutive order, but transmits some data segments and skips some data segments in a predetermined pattern. Considering the instance of Fig. 1(b). The second channel transmits four segments of γ4 ω with the distance of 3γ 4 ω. Therefore, as shown Fig. 2(b), xi (t, r) is a slanted staircase-shaped function, and the rate of increase of the solid line is Biγ(r) , where Bi (r) is the i-th channel bandwidth of Pr . Also, the width of a step in the function corresponds to the slot length. di (t) is L in case that Pi is a mature peer, and is determined by Eq.(3) in case that Pi is an immature peer. Fig. 2(b) shows di (t) when Pi is an immature peer. That xi (t, r) crosses di (t) as shown in the figure means that Pr would request the data which Pi would not have already received. Thus, Pi cannot be a supplying peer of Pr in this case. However, for an immature peer Pi to be a supplying peer of Pr , the following conditions must be satisfied: ∀t ≥ t0 , di (t) ≥ xi (t, r).
(5)
The immature peers satisfying this condition are called semi-mature peers of Pr . Consequently, the candidate set of Pr can consist of mature peers and semimature peers of Pr . Although di (t) has already been determined at the present time t0 , xi (t, r) has not. That is because xi (t, r) cannot be determined until Pr finishes selecting its supplying peers. For this reason, we use an upper bound function of xi (t, r), ¯r (t) = Rin (r) · (t − t0 )/ω ω. In x ¯r (t), instead. The upper bound function x Fig. 2(b), the staircase-shaped dashed line indicates x ¯r (t). Therefore, a sufficient ¯r (t). It can be used for a criterion to condition of Eq.(5) is: ∀t ≥ t0 , di (t) ≥ x determine whether an immature peer Pi can be a semi-mature peer. However, since it is a sufficient condition, not satisfying it does not mean that Pi is not
J.B. Kwon and H.Y. Yeom
data
856
L
i Bi (r)
r d(t)
B* (r)
x(t,r)
ωγ mature peer immature peer supplying peers of peer r
(a) Semi-mature Peer
0
t0
tf
t
(b) Data Growth of Peer i Fig. 2. Candidate Peers
a semi-mature peer. But, if it is satisfied, we only ensure it is a semi-mature peer. The procedure to select supplying peers is as follows: Pr determines its candidate set Cr such that each peer in Cr is a mature peer or a semi-mature peer and it has some available out-bound bandwidth 1 . To determine Cr , Pr should know whether or not each immature peer is a semi-mature peer, by ¯r (t). Since the higher the first channel bandwidth is, testing ∀t ≥ t0 , di (t) ≥ x the shorter the buffering delay in FSS(Eq.(4)), the peer with the maximal vout (i) among Cr is chosen as the first supplying peer of Pr . This procedure is repeated until the aggregated bandwidth B ∗ (r) is equal to Rin (r). The formal algorithm is omitted here due to the space limitation. In case that there are so many requesting peers that the P2P system is beyond its capacity, it may not acquire enough bandwidth. However, in the latter case, Pr can have a choice among three policies. The first is to start downloading with the acquired channels(FAST1), the second is to withdraw the request and retry after σ(FAST2), and the third is to start downloading with the acquired channels and retry to acquire the remainder after T minutes(FAST3). In FAST3, unlike the others, the number of in-bound channels of Pr may be changed during a session. This means the change of dr (t), which accordingly affects the transmission schedule and the condition for a semi-mature peer. The details of this dynamic session is omitted here due to the space limitation.
4
Performance Study
First, we compare the buffering delay of FSS with that of OTS. However, since OTS has the restrictions on channel bandwidth and FSS has the concept of slots, it is not simple to compare them directly. For a fair comparison, we set a criterion on the slot length and evaluate the two schemes under the condition satisfying the restrictions of OTS. The restrictions of OTS is the following: 1) A single 1
Pr is assumed to be able to know all the information needed to determine di (t) for each immature peer Pi
Distributed Multimedia Streaming over Peer-to-Peer Networks 30000
FAST3 FAST2 FAST1 BASE
25000 number of mature peers
857
20000 15000 10000 5000 0 0
5
10
15
20
25
30
35
40
time (hour)
Fig. 3. Distribution Speed
channel bandwidth has one of γ2 , γ4 , γ8 , . . ., 2γN . 2) The aggregate bandwidth of a session B ∗ (r) = γ. Let m be the number of segments. Given a set of n supplying peers and a requesting peer Pr , OTS achieves δOT S = (n − 1)L/m[9], if we assume that the content can be played while being downloaded. Under the same condition, the buffering delay of FSS is: δF SS = (1−B1 (r)/γ)·ω, according to Eq.(4). Since δOT S depends on m and δF SS on ω, the relation between m and ω should be drawn for the comparison. In OTS, the time taken for a segment to be transmitted over the channel with the highest bandwidth is corresponding L = ω, and then, by applying it to δOT S : to the slot length. That is, B1γ(r) · m δOT S = (n − 1) B1γ(r) · ω. Here, let c = B1 (r)/γ. Then, since B1 (r) is the highest bandwidth, n must be greater than or equal to 1/c to satisfy that B ∗ (r) = γ. Hence, n ≥ 1/c. For example, if B1 (r) = γ/4, n ≥ 4. Therefore, δOT S = (n − 1)c · ω ≥
(1/c − 1) c · ω = δF SS .
Finally, the buffering delay of FSS is smaller than or equal to that of OTS. In the example used in Fig. 1(a) and 1(b), δOT S = 3ω/2 and δF SS = ω/2. FSS has only a third the buffering delay of OTS in that case. Second, we study the performance of FAST using a simulation. We simulate a P2P system with total of 50,100 peers. Initially, there are only 100 ’seed’ peers, while the other 50,000 peers request the media data according to a given request arrival rate. The request arrival follows a Poisson distribution with mean 1/θ. Each seed peer possesses a copy of a popular video file. The running time of the video is 60 minutes. The in-bound bandwidth of all the peers is γ, equal to the playback rate. The out-bound bandwidth of the seed peers is γ2 , and that γ with the distribution of 10%, 10%, 40%, and of the others is γ2 , γ4 , γ8 , or 16 40%, respectively. The time interval T for retrying to acquire bandwidth has an uniform distribution with a mean of 10 minutes. And, the slot length ω is set also to be 10 minutes. A BASE scheme for comparison to FAST considers only mature peers as supplying peers, and it allows a requesting peer to start downloading when it acquires a sufficient bandwidth like FAST2. As the content is distributed to more peers, the P2P system’s capacity increases. Thus, when requests arrive as Poisson process with a rate of 1/θ, not all of the requests may be serviced at the beginning stage when there are only
858
J.B. Kwon and H.Y. Yeom
a small number of mature peers and semi-mature peers. This congestion due to the high arrival rate lasts for a while, until the number of mature peers grows sufficiently. We define θ-capacity time as the time to reach a capacity being able to service all the requests arriving at a rate of 1/θ . The smaller θ-capacity time means the faster distribution of the media content. Fig. 3 shows the number of mature peers of BASE, FAST1, FAST2, and FAST3 when θ = 5 seconds. If the system reaches θ-capacity, the number of mature peers increases linearly at a rate of 1/θ. However, as shown in the figure, the lines increase in a concave fashion in the early stage, and then increase in a straight line. The concave curves indicate that the system is congested. The 5-capacity time of FAST3, about 21 hours, is shorter than those of FAST2, FAST1, and BASE, which are about 24, 28, and 30 hours, respectively. And, FAST3 has almost more than twice the number of mature peers than that of BASE.
5
Conclusion
We have discussed the problems of distributing multimedia content over P2P network. The first problem is the transmission scheduling of the media data for a multi-source streaming session. We have presented a sophisticated scheduling scheme, which results in minimum buffering delay. The second one is on the fast diffusion of media content in the P2P system that is self-growing. We have also proposed a mechanism accelerating the speed at which the system’s streaming capacity increases.
References 1. Gnutella. http://gnutella.wego.com. 2. Napster. http://www.napster.com. 3. I. Clake, O. Sandberg, B. Wiley, and T. Hong. Freenet: A Distributed Anonymous Information Storage and Retrieval System. In Proc. of Workshop on Design Issues in Anonymous and Unobservability, July 2000. 4. T. Nguyen and A. Zakhor. Distributed Video Streaming Over Internet. In Proc. of Multimedia Computing and Systems, San Jose, California, January 2002. 5. Venkata N. Padmanabhan, Helen J. Wang, Philip A. Chou, and Kunwadee Sripanidkulchai. Distributing Streaming Media Content Using Cooperative Networking. In Int. Workshop on Network and Operating Systems Support for Digital Audio and Video, Miami Beachi, FL, May 2002. 6. S. Ratnasamy, P. Francis, M. Handley, R. Karp, and S. Shenker. A Scalable ContentAddressable Network. In ACM SIGCOMM, August 2001. 7. A Rowstron and P. Druschel. Pastry:Scalable Distributed Object Location and Routing for Large-Scale Peer-to-Peer Systems. In Proc. of IFIP/ACM Middleware, November 2001. 8. I. Stoica, R. Morris, D. Karger, F. Kaashoek, and H. Balakrishnan. Chord: A Scalable Peer-to-Peer Lookup Service for Internet Applications. In Proc. of ACM SIGCOMM, August 2001. 9. Dongyan Xu, Mohamed Hefeeda, Susanne Hambrusch, and Bharat Bhargava. On Peer-to-Peer Media Streaming. In Proc. of Int. Conf. on Distributed Computing Systems 2002, Austria, July 2002.
Exploiting Traffic Balancing and Multicast Efficiency in Distributed Video-on-Demand Architectures * Fernando Cores, Ana Ripoll, Bahjat Qazzaz, Remo Suppi, Xiaoyuan Yang, Porfidio Hernández, and Emilio Luque Computer Science Department – University Autonoma of Barcelona – Spain {Fernando.Cores,Ana.Ripoll,Remo.Suppi,Porfidio.Hernandez, Emilio.Luque}@uab.es {Bahjat,Xiaoyuan}@aows10.uab.es
Abstract. Distributed Video-on-Demand (DVoD) systems are proposed as a solution to the limited streaming capacity and null scalability of centralized systems. In a previous work, we proposed a fully distributed large-scale VoD architecture, called Double P-Tree, which has shown itself to be a good approach to the design of flexible and scalable DVoD systems. In this paper, we present relevant design aspects related to video mapping and traffic balancing in order to improve Double P-Tree architecture performance. Our simulation results demonstrate that these techniques yield a more efficient system and considerably increase its streaming capacity. The results also show the crucial importance of topology connectivity in improving multicasting performance in DVoD systems. Finally, a comparison among several DVoD architectures was performed using simulation, and the results show that the Double P-Tree architecture incorporating mapping and load balancing policies outperforms similar DVoD architectures.
1 Introduction Video on Demand (VoD) has been gaining popularity over recent years with the proliferation of high-speed networks. Distributed continuous media applications, are expected to provide service to a large number of clients often geographically dispersed over a metropolitan, country-wide or even global area. Employing only one large centralized continuous media server to support these distributed clients results in a high cost and non-scalable system with inefficient resource allocations. To address this problem, researchers have proposed distribution of the service in order to manage client dispersal. Systems based on this approach are termed Distributed VoD systems and they have demonstrated the ability to provide minimum communication-storage cost for distributed continuous media streaming applications [1]. A DVoD system requires the arrangement of those servers that offer the video retrieval and playback services in a distributed system, in order to support a large num-
*
This work was supported by the MCyT-Spain under contract TIC 2001-2592 and partially supported by the Generalitat de Catalunya- Grup de Recerca Consolidat 2001SGR-00218.
H. Kosch, L. Böszörményi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 859–869, 2003. © Springer-Verlag Berlin Heidelberg 2003
860
F. Cores et al.
ber of concurrent streams. In the literature, these approaches range from: 1) the use of Independent servers, 2) one level proxies, to 3) hierarchical distributed systems. The initial approach is based on replicating VoD servers close to clients’ networks so that these users do not need to access the main server [2]. One-level proxies try to reduce the size of local servers in such a way that they only store those videos with a higher access frequency; these servers are managed as main-server caches and are called proxies [9]. Hierarchical DVoD systems are based on a network with a hierarchical topology, with individual servers on the nodes and network links on the edge of the hierarchy. The nodes at the leaves of the hierarchy, termed head-ends, are points of access to the system where clients are usually connected [3][9][12][14]. In [6] we proposed an architecture for a fully distributed VoD system (called Double P-Tree) which, in addition to supporting a large number of concurrent streams, allows for the distribution of network traffic in order to minimize the network’s bandwidth requirements. This is achieved by distributing both the servers as well as the clients throughout the topology, avoiding the concentration of communication traffic on the last level of the hierarchy (head-end). It is demonstrated through an analytical study that this distributed architecture is fault-tolerant and guarantees unlimited and low-cost growth for a large-scale VoD system. In this paper, we focus on the design aspects of the Double P-Tree architecture with the view to optimizing its performance and to supporting a greater streaming capacity. We concentrate particularly on two aspects: incorporating a video-mapping mechanism to minimize service distance, and the proposal of traffic-balancing policies that allow a reduction in network bandwidth requirements for the system. These proposed policies have been evaluated through several simulation experiments and the results have shown significant improvement in the Double P-Tree architecture’s performance. In addition, on the one hand we study the influence of the architecture’s connectivity in improving the efficiency of multicast policies in distributed systems, and on the other hand we analyze the proxy storage capacity. The remainder of this paper is organized as follows: in section 2, we first give an overview of the Double P-Tree architecture and we describe some topics related to its implementation. In section 3, we propose some techniques related to video placement and traffic balancing. Performance evaluation is shown in section 4 and, finally, in the last section, we indicate the main conclusions to be drawn from our results.
2 Distributed VoD Architecture Fig.1a depicts the architecture of the proposed DVoD system. This architecture is designed as a network with a tree topology, with individual small servers (proxies) as the nodes, and network links as the edges of the hierarchy. Nodes are assumed to be able to store a limited number of videos and stream a finite number of videos. Meanwhile, networks links are expected to guaranteed the specific QoS requirements of video communications. A brief description of the system architecture is given below.
Exploiting Traffic Balancing and Multicast Efficiency
861
Non Segmented Switch P1
Level 1
P2
Bandwidth requirements: n
∑ Traffic ( P )
Level 2 Level 3
Pn-1
Pn
p
p =1
Segmented Switch
Level 4
P1
P2
Bandwidth requirements: Proxy
[
Max np =1 Traffic ( Pp )
Level switch
]
Brother networks Pn-1
Pn
Clients
(a) Topology
(b) Network infrastructure
Fig. 1. Double P-Tree Architecture
2.1 Network Topology For the network topology we have selected a fully distributed topology based on proxies. The structure of this topology consists of a series of levels, in accordance with the number of local networks and the order of the tree. Each hierarchy level is made up of a series of local networks with its local-proxy and clients that forms the following tree level. To improve topology connectivity, several local networks (named brothers networks) from the same level are interconnected, increasing the number of adjacent local networks without changing the topology size or last level width. This new architecture is named Double P-Tree because the brother networks are joined in a second tree within the original tree topology [6]. In order to reduce network bandwidth requirements the network infrastructure is designed using segmented switches in local networks. Fig 1b, shows network bandwidth requirements for non-segmented and segmented switches. This selection is based on the fact that in segmented switches, every port (Pi) has an Independentbandwidth, and therefore, it is only necessary to have enough bandwidth in order to support the maximum traffic from all ports. Segmented switches allow the reduction of switchbandwidth requirements if traffic is distributed among different ports. Double P-Tree architecture can make better use of segmented switches due to its network traffic being distributed among different sources. A possible drawback of this utilization is that topology-port traffic (ports used to implement the topology) and server-port (port used to connect proxy-server) could be unbalanced when the proxy load is centralized in one server-port. In order to solve this unbalance (increasing the bandwidth requirements), the architecture connects the proxies to local-networks using several ports. 2.2 Proxy Server The simple inclusion of a hierarchical system with proxies does not, in itself, obtain improvements in the system’s scalability or efficiency: since, as all the proxies are
862
F. Cores et al.
caching the same videos, if a proxy cannot serve a request from its client, then it is also very probable that none of the other proxies will be able to serve this request, and the solution will then require accessing the main server. We therefore need to use a new proxy organization and functionality to increase the hit probability as the request climbs the various levels on the tree. This proposal lies in dividing the storage space available within proxies into two parts: one of these will continue functioning as a cache, storing the most requested videos; the other proxy space will be used for making a distributed mirror of system videos [6]. In order to provide True VoD we have concentrated on multicast transmission techniques [7][10]. These techniques can greatly reduce server I/O and network bandwidth. But with them, it is difficult to implement VCR functions since a dedicated channel is not allocated to each user. Whenever a user tries to play the VCR functions he will disjoin the multicast channel, and some new resources that have not been planned before must then be reserved and assigned to him. These resources will be used to meet the VCR functions and/or to provide a unicast channel so that the user can continue with the normal playback. Several ideas has been proposed to solve this problem basically reserving some channels for these specific actions [5][11] . Our implementation for the VCR functions does not reserve specific channels and is based on the observation that there are periods of times during which network bandwidth is under-utilized. During these periods, the proxy server sends more video in advance (pre-fetching) to an appropriate client’s buffer. Whenever a user invokes a VCR action, the resources that have been assigned before the peak time are recovered in favor of this VCR request. Another important element that affects the proxy performance is the proxy file system. Proxy servers that implement conventional file systems have been designed to reduce load on servers as well as client access time [4][13]. Nevertheless for continuous media with soft real-time constraints, typical file systems based on best-effort paradigm are inadequate for achieving this new requirement. Proxy servers can provide performance guarantees to applications only in conjunction with an operating system that can allocate resources in a predictable manner. In our case, the most representative workload is the updates and removes in the caching and mirroring subsystem; consequently, the disk broker must employ placement policies that minimize fragmentation of disk space resulting from frequent writes and deletes. In order to obtain the best performance from the disk driver, track-buffer techniques are used. These techniques eliminate rotational delays in reads and obtain maximum performance on write operations.
3 Architecture Design Issues In order to implement an efficient DVoD architecture several challenging research problems have to be solved to allow an efficient management of network and the services. Some of these problems are related with the subjects of reducing service distance and balancing communication traffic. In this section we propose some policies to accomplish these goals.
Exploiting Traffic Balancing and Multicast Efficiency
863
Table 1. Performance of Video-Mapping Heuristic for Double P-Tree Mean service distance
Unicast
Effective bandwidth
sequential
11.001 Mb/s
1,80
heuristic
12.849 Mb/s
1,747
Multicast
Mirror Distribution
sequential
14.913 Mb/s
1,80
heuristic
15.951 Mb/s
1,747
3.1 Videos Mapping on Distributed Mirror The main factor that penalizes DVoD architectures performance, measured as the number of concurrent clients supported by the system (effective bandwidth), is the over-bandwidth required due to requests that cannot be served locally. In this case, a remote service requires: local bandwidth in the remote server, bandwidth in the serverport in the switch, bandwidth in the remote switch-topology port, bandwidth in the local switch-topology port and finally bandwidth in the client switch-port. A good approximation to evaluate this over-bandwidth is mean service distance (the distance needed to reach all movies from every node in the system). In particular, Double P-Tree mean service distance is affected by diverse factors, such as topology connectivity, proxy storage distribution between caching and mirroring, and mirror-videos mapping in proxies. The first and the second issues were analyzed in a previous paper [6], and the last one is studied below. Given that it is too complex achieve a optimal video distribution, we have developed a heuristic to choose which videos need to be mapped in every proxy-mirror. This heuristic consists of calculating, for each proxy within the architecture, the minimum distance where we can find each of the movies of the system-repository (taking into account videos already mapped in the previous proxies). Then, in order to minimize the mean service-distance, we always choose those videos that are stored in the proxy-mirrors furthest from this proxy. In the case of there being various videos at the same distance, the most popular are then selected. Table 1 shows the mean service distance and effective bandwidth obtained by the heuristic and a sequential distribution of videos based simply on assigning a group of videos to each one of the proxies in a sequential manner. These results use the simulation parameters given in section 4, taking unicast and multicast policies into account. As we can see, the heuristic reduces mean-service distance from 1.80 to 1.74, which allows for an improvement in system performance (“Effective Bandwidth” column) of 7% and 14% for unicast and multicast, respectively. 3.2 Traffic Balancing Policies In architectures using segmented switches, the bandwidth requirements for a network depends on the maximum traffic supported by any of its ports. Therefore, it is very
864
F. Cores et al. Table 2. Performance of Traffic Balancing Policies for Double P-Tree
Multicast
Unicast
Traffic Balancing Policy
Effective bandwidth
Mean Distance
Imbalance
Unbalance
12.849 Mb/s
1,747
56,44%
Mirror Balanced
12.240 Mb/s
1,769 (3)
55,61% (1)
Traffic Balancing
15.653 Mb/s
1,747
30,45% (5)
Unbalance
15.951 Mb/s
1,732
Mirror Balanced
15.630 Mb/s
1,769
Traffic Balancing
19.700 Mb/s
1,747
78,27% (4)
71,70% (2) 48,07% (6)
important to balance port traffic in order to reduce bandwidth requirements and to increase system performance. In Double P-Tree architecture, most loaded ports are of the server and topology ports. Server ports can easily be balanced because the proxy server can choose at any time through which port a request is attended. On the other hand, balancing topology ports is more difficult because their load depends on video placement on distributed mirrors. For example, the links connecting proxy servers that map the most popular videos would be more overloaded than others, producing a traffic imbalance. To achieve traffic-load balancing, we have studied two different approximations: • Mirror Balanced. An initial approximation to avoid imbalance is by building the distributed mirror in more balanced way. This objective can be achieved tuning the previous mapping heuristic because we do not always select the most popular videos; rather, we choose a mixture of highly and less popular videos. • Dynamic traffic balancing. However, balancing through mirror distribution has a very limited maneuverability and cannot easily adapt to changes in traffic patterns or video access frequency. Therefore we have proposed another more dynamic policy for traffic load balancing. As the imbalance problem only appears with remote requests (which are the only ones that use topology ports), when a request cannot be attended, the local proxy also receives information about traffic from all alternative sources (i.e. proxies that have a copy of requested video and enough resources to attend the request). Using this information, and if there are two or more alternative paths (meaning some video replication), our balancing policy always chooses the least-loaded path in order to balance traffic. Using the simulation parameters given in section 4, in Table 2, we show the results obtained with these balance policies. As we can see, compared with imbalance, the 1,2 mirror balanced policy is successful in reducing imbalance , but this reduction is not 3,4 enough to compensate for the rise in service distance . In contrast, the results obtained with the dynamic traffic load balancing policy are much better, decreasing 5,6 imbalance significantly (without affecting service distance) and increasing performance by around 24% (15.951Mbs and 19.700Mbs as against 12.849Mbs and 15.653Mbs , for unicast and multicast respectively).
Exploiting Traffic Balancing and Multicast Efficiency
4
865
Performance Evaluation
In this section, we show the simulation results for Double P-Tree architecture and contrast these with other distributed architectures. We conducted several experiments to 1) evaluate the effect of multicast, and 2) study the effect of proxy storage capacity on system behavior. Table 3. Simulation Parameters
Parameter
Value
Parameter •
Multicast technique Client buffer size Request rate (λ)
•
Number of videos
100
• • •
Video length Local networks Network bandwidth Server bandwidth 1server port (1Sp) Server bandwidth 3server port (3Sp)
90 minutes 63
• •
100 Mb/s
•
Poisson distribution
•
Zipf distribution
• • •
Proxy capacity
100 Mb/s 300 Mb/s 20 videos
Value Patching 5 minutes 10 req/min by net
pi = px =
λik −λ e k!
i
(1)
1 Sv
x ⋅∑ 1
(2)
z
i =1
iz
4.1 Simulation Environment To guide this objective, we have designed and implemented an object-oriented VoD simulator. The main parameters of the simulation environment are summarized in Table 3. In all studies, we use architectures with 63 local networks (6 levels in Double P-Tree topology) using 100 Mbps segmented switches. The request inter-arrival time 1 is generated by the simulation of a Poisson distribution with a mean of 1/λ, where λ is the request arrival rate in every local network in the VoD system. The selection of 2 the video is modeled with a Zipf distribution , with a skew factor of 0.7 (z), which models the popularity of rental movies [1]. 4.2 Effect of Multicast on Distributed VoD Architectures In this section, we evaluate DVoD system performance, using effective bandwidth (number of users attended * 1.5Mbs) as the main metric. This study allows us to evaluate the maximum streaming capacity for different architectures: Independent servers, one level proxies and Double P-Tree (using heuristic mapping and dynamic traffic balancing policy), for both unicast and multicast techniques. In order to obtain the results, plotted in Fig 2, we have assigned an aggregate network bandwidth of 6.300Mbs and an aggregate sever bandwidth of 6.300Mbs (with 1 server port) or 18.900Mbs (with 3 Server ports). To obtain the maximum system streaming capacity, we saturated the system using a high request ratio (10 req/min by network) and simulated the system behavior until the aggregate bandwidth is ex-
866
F. Cores et al.
Effective bandwidth Gb/s
40
Unicast
Patching
35 30 25 20 15 10 5 0 Ind. servers Ind. servers 1-level Double P- Double P- Double P1Sp 3Sp Proxies 3Sp Tree 3B1Sp Tree 3B3Sp Tree 7B1Sp
Fig. 2. Performance in Distributed VoD Architectures
hausted. When there is no bandwidth available, the system achieves its maximum streaming capacity and we evaluate the system performance (effective bandwidth using unicast and patching [12]) . Using unicast, we can see that the Independent servers with 3Sp is the architecture that obtains the best results, achieving the theoretical maximum stream capacity of 19Gbs (63 networks * 100Mbs * 3Sp). Meanwhile Double P-Tree with 7 brothers (7B) and 3Sp obtains an effective bandwidth of 18.2Gbs (4% less), due to the additional bandwidth required to attend distributed requests and lower storage requirements (20 as against 100 videos in every proxy). However, this underperformance is less than expected according to the criteria of mean service distance between Independent servers (1) and Double P-Tree architectures (1,747 according to Table 2). This result demonstrates the strength of our architecture in distributing and balancing traffic among topology ports in order to reduce network requirements. Also, as we can see, our architecture is better than 1-level proxies architecture (improved by 75%). Double P-Tree architecture exploits its characteristics to realize its potential advantage when client streams are shared using a multicast technique (Patching, in our case). In this more realistic scenario, Double P-Tree is the best solution, improving Independent servers by 27% (38.4Gbs as against 30Gbs) and one-level proxies by 200% (38.4Gbs as against 12.7Gbs). The principal argument for this improvement is that the Double P-Tree has a better connectivity than the Independent server and one-level architectures. This better connectivity means that, in Double P-Tree, mirror video streams can potentially be shared among requests coming from all adjacent networks, multiplying the sharing probability by topology connectivity. Meanwhile, in low interconnected architectures, potential stream sharing is limited only to local requests.
Exploiting Traffic Balancing and Multicast Efficiency
867
40 35 30
Ind. Servers (Patching)
25
Ind. Servers (Unicast)
20 15 10
90%
84%
78%
72%
66%
60%
54%
48%
Double P-Tree 7B3Sp (Patching)
42%
30%
28%
25%
22%
19%
16%
13%
4%
1%
7%
Double P-Tree 7B3Sp (Unicast)
0
36%
5
10%
Effective bandwidth (Gb/s)
45
Fig. 3. Double P-Tree Effective Bandwidth & Proxy Storage Capacity
4.3 Effect of Proxy Storage Capacity In this section, we evaluate the effect of proxy storage capacity on Double P-Tree performance. In Fig. 3, we can first see that a small proxy storage leads to low performance due to large service distance being required to attend remote requests, and to its over-bandwidth requirements. With only storage for 15% of system videos, Double P-Tree performance (with patching) is equivalent to Independent severs performance, but uses 6 times less storage. Also, we notice that the highest performance is obtained with proxy storage of around 25%. In this case, Double P-Tree performs Independent servers in more than 38% (41.5Gbs against 30Gbs). From this point, we can observe that in the measure that storage capacity grows, performance decreases until reaching Independent server performance, in which case the proxy has enough capacity to store a full video catalog copy. Why do more resources give less performance? The reasons for this interesting effect can be explained by the fact that, when Double P-Tree proxies have a lot of storage (more than 30%), their architectural behavior is very similar to that of Independent server systems. In this case, proxy mirrors have enough storage to reach all videos at distance-2 mirrors, therefore all remaining storage is assigned for caching. Increasing proxy-cache size increases the number of requests attended locally, creating two consequences. First, server ports have more load, leading to network traffic unbalancing and a faster network saturation. Second, there are videos with a medium access pattern that were previously managed under mirroring scheme. Proxy-mirrors, where these were mapped, centralized all requests coming from adjacent nodes, improving stream sharing. If we now place these videos in the cache (replicating them in all proxies), we are decreasing access frequency for every video copy, reducing both the stream-sharing probability and system performance. This result clearly demonstrates the goodness of distributed mirroring as against several full mirror replications.
868
F. Cores et al.
5 Conclusions This paper deals with two decisive aspects for DVoD architectures performance: video placement policies in distributed mirrors and network-traffic balancing policies. The proposed policies attain a reduction in mean service distance and minimize network requirements for the Double P-Tree architecture. Simulation results show that proposed policies substantially increase the number of concurrent clients who can be served by the system. The video mapping heuristic achieves an improvement of 7%, while the dynamic traffic-balancing policy yields an additional increase of 26%. These results clearly demonstrate the importance of network-traffic balancing policies as a fundamental instrument in diminishing the network bandwidth requirements in DVoD systems. On the other hand, we have also shown the importance of topology connectivity in DVoD systems (in particular, in the Double P-Tree) in order to improve multicasting performance. The Double P-Tree using multicasting and similar resources clearly outperforms classical DVoD architectures, namely, Independent servers (by 38%) and one-level proxies (by 200%). Finally, we have demonstrated that full mirror replication in every local network (as in Independent servers) not only requires more storage but also achieves a poorer performance in comparison to distributed mirroring (Double P-Tree).
References 1.
S.A. Barnett and G. J. Anido, "A cost comparison of distributed and centralized approaches to video-on-demand," IEEE Journal on Selected Areas in Communications, vol. 14, pp. 1173–1183, August 1996. 2. S.-H. G. Chan and F. Tobagi, “Caching schemes for distributed video services”, in Proc. of IEEE Int’l Conference on Communications (ICC'99), Canada, June 1999, pp. 994–1000. 3. S.-H. G. Chan and F. Tobagi, “Distributed Servers Architecture for Networked Video Services”, in IEEE/ACM Transactions on Networking, Vol. 9, No. 2 April 2001. 4. A. Chankhunthod, P.B. Danzing, C. Neerdaels, M. F. Schwartz, and K.J. Worrell. “A Hierarchical Internetwork Object Cache". In Proceeding of the 1996 USENIX Technical Conference, San Diego, CA, January 1996. 5. J-M Choi, S-W Lee, K-d Chung, “A Multicast Scheme for VCR Operations in a Large VOD system”, ICPADS 2001:555–561. 6. F. Cores, A. Ripoll, E. Luque, “Double P-Tree: A Distributed Architecture for Large-Scale Video-on-Demand”, Euro-Par 2002, LNCS 2400, pp. 816–825, Aug. 2002. 7. A. Dan, D. Sitaram, and P. Shahabuddin, "Dynamic batching policies for an on-demand video server," Multimedia Systems 4, pp. 112–121, June 1996. 8. H. Fabmi, M. Latif, S. Sedigh-Ali, A. Ghafoor, P. Liu, L.H. Hsu, “Proxy servers for scalable interactive video support” , IEEE Computer , Vol. 34 Iss. 9, pp. 54–60 Sept. 2001. 9. C. Griwodz. “Wide-Area True Video-on-Demand by a Decentralized Cache-based Distribution Infrastructure.” PhD dissertation, Darmstadt Univ. of Technology, Germany, Apr. 2000. 10. K. A. Hua, Ying Cai and S. Sheu, Patching: A multicast tecnique for true video-on-demand services, ACM Multmedia’98, pages 191–200.
Exploiting Traffic Balancing and Multicast Efficiency
869
11. J. Y. B. Lee, “On a Unified Architecture for Video-on-Demand Services”, IEEE Transactions on Multimedia, Vol. 4, No. 1, March 2002. 12. Cyrus Shahabi, Farnoush Banaei-Kashani, “Decentralized Resource Management for a Distributed Continuous Media Server” , IEEE Transactions on Parallel and Distributed Systems, Vol. 13, No. 7, July 2002 13. “Squid Internetwork Objet Cache Users Guide”. Available on line at http://squid.nlanr.net, 1997. 14. Xiaobo Zhou, R. Luling, Li Xie, “Solving a Media Mapping Problem in a Hierarchical Server Network with Parallel Simulated Annealing”, Procs. 2000 International Conference on Parallel Processing, pp. 115–124, 2000.
On Transmission Scheduling in a Server-Less Video-on-Demand System1 C.Y. Chan and Jack Y.B. Lee Department of Information Engineering The Chinese University of Hong Kong {cychan2, yblee}@ie.cuhk.edu.hk
Abstract. Recently, a server-less video-on-demand architecture has been proposed which can completely eliminate costly dedicated video servers and yet is highly scalable and reliable. Due to the potentially large number of user hosts streaming video data to a receiver for playback, the aggregate network traffic can become very bursty, leading to significant packet loss at the access routers (e.g. 95.7%). This study tackles this problem by investigating two new transmission schedulers to reduce the traffic burstiness. Detailed simulations based on Internet-like network topologies show that the proposed Staggered Scheduling algorithm can reduce packet loss to negligible levels if nodes can be clock synchronized. Otherwise, a Randomized Scheduling algorithm is proposed to achieve consistent performance that is independent of network delay variations, and does not require any form of node synchronization.
1
Introduction
Peer-to-peer and grid computing have shown great promises in building highperformance and yet low cost distributed computational systems. By distributing the workload to a large number of low-cost, off-the-shelve computing hosts such as PCs and workstations, one can eliminate the need for a costly centralized server and at the same time, improve the system’s scalability. Most of the current works on grid computing are focused on computational problems [1], and on the design of the middleware [2]. In this work, we focus on another application of the grid architecture – video-on-demand (VoD) systems, and in particular, investigate the problem of transmission scheduling in such a distributed VoD system. Existing VoD systems are commonly built around the client-server architecture, where one or more dedicated video servers are used for storage and streaming of video data to video clients for playback. Recently, Lee and Leung [3] proposed a new server-less VoD architecture that does not require dedicated video server at all. In this 1
This work was supported in part by the Hong Kong Special Administrative Region Research Grant Council under a Direct Grant, Grant CUHK4211/03E, and the Area-of-Excellence in Information Technology.
H. Kosch, L. Böszörményi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 870–879, 2003. © Springer-Verlag Berlin Heidelberg 2003
On Transmission Scheduling in a Server-Less Video-on-Demand System
871
server-less architecture, video data are distributed to user hosts and these user hosts cooperatively server one another’s streaming requests. Their early results have shown that such a decentralized architecture can be scaled up to hundreds of users. Moreover, by introducing data and capacity redundancies into the system, one can achieve system level reliability comparable to or even exceeding those of high-end dedicated video servers [4]. Nevertheless, there are still significant challenges in deploying such server-less VoD systems across the current Internet. In particular, Lee and Leung’s study [3] did not consider the issue of network traffic engineering. With potentially hundreds or even thousands of nodes streaming data to one another, the aggregate network traffic can become very bursty and this could lead to substantial congestion at the access network and the user nodes receiving the video data. Our study reveals that packet loss due to congestion can exceed 95% if we employ the common first-come-first-serve algorithm to schedule data transmissions. In this study, we tackle this transmission scheduling problem by investigating two transmission scheduling algorithms, namely Staggered Scheduling and Randomized Scheduling. Our simulation results using Internet-like network topologies show that the Staggered Scheduling algorithm can significantly reduce packet loss due to congestion (e.g. from 95.7% down to 0.181%), provided that user nodes in the system are clock-synchronized using existing time-synchronization protocols such as the Network Time Protocol [5]. By contrast, the Randomized Scheduling algorithm does not need any form of synchronization between the user nodes albeit does not perform as well. Nevertheless, the performance of the Staggered Scheduling algorithm will approach that of the Randomized Scheduling algorithm when the network delay variation or clock jitter among nodes are increased. In this paper, we present these two scheduling algorithms, evaluate their performance using simulation, and investigate their sensitivity to various system and network parameters.
2
Background
In this section, we first give a brief overview of the server-less VoD architecture [3] and then define and formulate the transmission scheduling problem. Readers interested in the server-less architecture are referred to the previous studies [3-4] for more details. 2.1 Server-Less VoD Architecture A server-less VoD system comprises a pool of fully connected user hosts, or called nodes in this paper. Inside each node is a system software that can stream a portion of each video title to as well as playback video received from other nodes in the system. Unlike conventional video server, this system software serves a much lower aggregate bandwidth and thus can readily be implemented in today’s set-top boxes (STBs) and
872
C.Y. Chan and J.Y.B. Lee
PCs. For large systems, the nodes can be further divided into clusters where each cluster forms an autonomous system that is independent from other clusters. STB
(N – 1) nodes
STB
Access router Playback Internet STB
STB
Fig. 1. A N-node server-less video-on-demand system
For data placement, a video title is first divided into fixed-size blocks and then equally distributed to all nodes in the cluster. This node-level striping scheme avoids data replication while at the same time share the storage and streaming requirement equally among all nodes in the cluster. To initiate a video streaming session, a receiver node will first locate the set of sender nodes carrying blocks of the desired video title, the placement of the data blocks and other parameters (format, bitrate, etc.) through the directory service. These sender nodes will then be notified to start streaming the video blocks to the receiver node for playback. Let N be the number of nodes in the cluster and assume all video titles are constantbit-rate (CBR) encoded at the same bitrate Rv. A sender node in a cluster may have to retrieve video data for up to N video streams, of which N – 1 of them are transmitted while the remaining one played back locally. Note that as a video stream is served by N nodes concurrently, each node only needs to serve a bitrate of Rv/N for each video stream. With a round-based transmission scheduler, a sender node simply transmits one block of video data to each receiver node in each round. The ordering of transmissions for blocks destined to different nodes becomes the transmission scheduling problem. 2.2 Network Congestion In an ideal system model, video data are transmitted in a continuous stream at a constant bit-rate to a receiver node. However, in practice data are always transmitted in discrete packets and thus the data stream is inherently bursty. In traditional clientserver VoD system this problem is usually insignificant because only a single video server will be transmitting video data to a client machine and thus the data packets will be transmitted at constant time intervals. By contrast, video data are distributed across all nodes in a server-less VoD system and as a result, all nodes in the system participate in transmitting video data packets to a node for playback. If these data transmissions are not properly coordinated, a large number of packets could arrive at the receiver node’s access network at the same time, leading to network congestion.
On Transmission Scheduling in a Server-Less Video-on-Demand System
873
For example, Fig. 2 depicts a straightforward transmission scheduler - On Request Scheduling (ORS), which determines the transmission schedule based on the initial request arrival time. Specifically, a node transmits video data in fixed-duration rounds, r1
r2 1
Node 0 r1
1
r2 1
Node 1
r1 Node 9
2
2
1
2
r2 1 2
1 2
ri : request from node i Fig. 2. Transmission schedules generated by the On Request Scheduling algorithm
with each round further sub-divided into N timeslots. The node can transmit one Qbyte data packet in each time slot. Let Tr be the length of round and Ts = Q/Rv be the length of a timeslot, then with a video bit-rate Rv we can compute Tr and Ts from Tr=NTs=NQ/Rv. When a node initiates a new video streaming session, it will send a request to all nodes in the system. A node upon receiving this request will reserve an available timeslot in a first-come-first-serve manner to begin transmitting video data for this video session. For example, consider the scenario in Fig. 2 where there are 10 timeslots per round. Request r1 reaches node 0, and is assigned to slot 5. On the other hand, when request r2 reaches node 0 the upcoming slot has already been assigned to another stream and in this case the request will be assigned to the first available slot (i.e. slot 0). Note that for simplicity we do not consider disk scheduling in this study and simply assumed that data are already available for transmission. It is easy to see that this ORS algorithm can minimize the startup delay experienced by end users as well as spread out data transmissions destined for different receivers to reduce burstiness of the aggregate network traffic leaving a node. While this algorithm may work well in traditional client-server VoD systems, its performance is unacceptably poor in a server-less VoD system. In our simulation of a 500-node system with Q=8KB and Rv=4Mbps, this ORS algorithm can result in over 95% packet losses due to congestion in the access network. The fundamental problem here is due to the very large number of nodes in the system and the fact that data transmissions are packetized. With the ORS algorithm, a new video session will likely be assigned to timeslots that are temporally close together. Thus once transmission begins, all nodes in the system will transmit video data packets to the receiver node in a short time interval, and then all cease transmission for Tr seconds before transmitting the next round of packets. While the average aggregate transmission rate is still equal to the video bit-rate, the aggregate
874
C.Y. Chan and J.Y.B. Lee
traffic is clearly very bursty and thus leads to buffer overflows and packet drops at the access network router connecting the receiver node. Node 0
1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0
Node 1
2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
Node 9
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9
Fig. 3. Transmission schedules generated by the Staggered Scheduling algorithm
3
Transmission Scheduling
To tackle the network congestion problem discussed earlier, we investigate in this section two transmission scheduling algorithms, namely the Staggered Scheduling (SS) and the Randomized Scheduling (RS) algorithms. 3.1 Staggered Scheduling (SS) As mentioned in Section 2.2, the ORS algorithm can reduce the burstiness of the network traffic leaving a sender node, but the combined traffic from multiple sender nodes can still be very bursty. The fundamental problem is that the ORS algorithm attempt to schedule a video session to nearby timeslots in all nodes and thus rendering the combined traffic very bursty. This observation motivates us to investigate a new scheduler – Staggered Scheduling, which schedules a video session to nodes in non-overlapping timeslots as shown in Fig. 3. Specifically, the timeslots are all pre-assigned to different receiver nodes. For node i serving node j data will always be transmitted in timeslot (j–i–1) mod N. For example, in Fig. 3 node 9 is served in timeslot 8 in node 0 while it is served in timeslot 7 in node 1. Thus the timeslot assignment of a video session forms a staggered schedule and hence the name for the algorithm. Assuming the nodes are clock-synchronized, then transmissions from different nodes to the same receiver node will be separated by at least Ts seconds, thus eliminating the traffic burstiness problem in ORS. Nevertheless, the need for clocksynchronization has two implications. First, as clocks in different nodes cannot be precisely synchronized in practice, the performance of the algorithm will depend on the clock synchronization accuracy. Second, depending on the application, the
On Transmission Scheduling in a Server-Less Video-on-Demand System
875
assumption that all nodes in the system are clock-synchronized may not even be feasible. Table 1. Initial assignment of the parameters of the data loss problem model
Parameters Cluster size Video block size Video bitrate Access network bandwidth Router buffer size (per node) Mean propagation delay Variance of propagation delay Mean router queueing delay Variance of clock jitter
Values 500 8KB 4Mbps 1.1Rv 32KB 0.005s -6 10 0.005s -6 10
3.2 Randomized Scheduling (RS) Staggered Scheduling attempts to perform precise control of the data transmissions to smooth out aggregate network traffic. Consequently, close synchronization of the nodes in the system is essential to the performance of algorithm. In this section, we investigate an alternative solution to the problem that does not require node synchronization. Specifically, we note that the fundamental reason why aggregate traffic in ORS is bursty is because data transmission times of all the sender nodes are highly correlated. Thus if we can decorrelate the data transmission times then the burstiness of the traffic will also be reduced. This motivates us to investigate a new Randomized Scheduling algorithm that schedules data transmissions for a video session in random timeslots. Moreover, the randomly assigned timeslot is not fixed but randomized in each subsequent round to eliminate any incidental correlations. It is easy to see that under Randomized Scheduling, one does not need to perform clock synchronization among nodes in the system. Each node simply generates its own random schedule on a round-by-round basis. We compare the performance of Staggered Scheduling and Random Scheduling in the next section.
4
Performance Evaluation
We evaluate and compare the three scheduling algorithms studied in this paper using simulation. The simulator simulates a network with 500 nodes. To generate a realistic network topology, we implement the extended BA (EBA) model proposed by Barabási et al. [6] as the topology generator, using parameters measured by Govindan et al. [7].
876
C.Y. Chan and J.Y.B. Lee
Packet Loss Rate
To model access routers in the network, we assume an access router to have separate buffers for each connected node. These buffers are used to queue up incoming data packets for transmission to the connected node in case of bursty traffic. When the buffer is full, then subsequent arriving packets for the node will be discarded and thus resulting in packet loss. 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0
100
200
300
400
500
600
Cluster Size (nodes) ORS
SS
RS
Fig. 4. Comparison of packet loss rate versus cluster size for ORS, SS, and RS
To model the network links, we separate the end-to-end delay into two parts, namely, propagation delay in the link and queueing delay at the router. While the propagation delay is primary determined by physical distance, queueing delay at a router depends on the utilization of the outgoing links. We model the propagation delay as a normally distributed random variable and the queueing delay as an exponentially-distributed random variable [8]. To model clock synchronization protocol, we assume that the clock jitter of a node, defined as the deviation from the mean time of all hosts, is normally-distributed with zero mean. We can then control the amount of clock jitter by choosing different variances for the distribution. Table 1 summarizes the default values of various system parameters. We investigate in the following sections the effect of four system parameters, namely cluster size, router buffer size, clock jitter, and queueing delay on the performance of the three scheduling algorithms in terms of packet loss rate. Each set of results is obtained from the average results of 10 randomly generated network topologies. 4.1 Packet Loss Rate versus Cluster Size Fig. 4 plots the packet loss rate versus cluster size ranging from 5 to 500 nodes. There are two observations. First, the loss rate decreases rapidly at smaller cluster size and
On Transmission Scheduling in a Server-Less Video-on-Demand System
877
Packet Loss Rate
becomes negligible for very small clusters. For example, for a 10-node cluster the loss rate is only 6.6%. This confirms that the traffic burstiness problem is unique to a server-less VoD system where the number of nodes is typically large. Second, comparing the three algorithms, On Request Scheduling (ORS), Staggered Scheduing (SS), and Randomized Scheduling (RS), ORS performs extremely poorly with loss rates as high as 95%, which is clearly not acceptable in practice. RS performs significantly better, with a loss rate approaching 9.3% when the cluster size is increased to 500. Finally, the SS algorithm performs best with 0.18% packet loss regardless of the cluster size, demonstrating its effectiveness in eliminating bursts in the aggregate traffic.
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0
20
40
60
80
100
Router Buffer Size (KB) ORS
SS
RS
Fig. 5. Comparison of packet loss rate versus router buffer size for ORS, SS, and RS
4.2 Packet Loss Rate versus Router Buffer Size Clearly, the packet loss rate depends on the buffer size at the access router. Fig. 5 plots the packet loss rate against router buffer sizes ranging from 8KB to 80KB. The packet size is Q=8KB so this corresponds to the capacity to store one to ten packets. As expected, the loss rates for all three algorithms decrease with increases in the router buffer size. In particular, the performance of RS can approach that of SS when the router buffer size is increased to 80KB. Nevertheless, ORS still performs very poorly even with 80KB buffer at the routers and thus one cannot completely solve the problem by simply increasing router buffer size.
878
C.Y. Chan and J.Y.B. Lee
4.3 Packet Loss Rate versus Queueing Delay
Packet Loss Rate
On the other hand, delay variations in the network can also affect performance of the schedulers. To study this effect, we vary the routers’ mean queueing delay from 0.0005 to 5 seconds and plot the corresponding packet loss rate in Fig. 6.
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0.0001
0.001
0.01
0.1
1
10
M ean of Queueing Delay (sec.) ORS
SS
RS
Fig. 6. Comparison of packet loss rate versus router queueing delay for ORS, SS, and RS
There are two interesting observations from this result. First, performance of the RS algorithm is not affected by changes in the mean queueing delay. This is because packet transmission times under RS are already randomized, and thus adding further random delay to the packet transmission times has no effect on the traffic burstiness. Second, surprisingly performances of all three algorithms converge when the mean queueing delay is increased to 5 seconds. This is because when the mean queueing delay approaches the length of a service round (i.e. Tr=8.192 seconds), the random queueing delay then effectively randomize the arrival times of the packets at the access router and hence performances of both the ORS and SS algorithms converge to the performance of the RS algorithm. This significance of this results is that transmission scheduling is effective only when random delay variations in the network are small compared to the service round length. Moreover, if the amount of delay variation is unknown, then the RS algorithm will achieve the most consistent performance, even without any synchronization among nodes in the system.
On Transmission Scheduling in a Server-Less Video-on-Demand System
5
879
Conclusions
We investigated the transmission scheduling problem in a server-less VoD system in this study. In particular, for networks with comparatively small delay variations and with clock-synchronization, the presented Staggered Scheduling algorithm can effectively eliminates the problem of traffic burstiness and achieve near-zero packet loss rate. By contrast, the Randomized Scheduling algorithm can achieve consistent performance despite variations in network delay. More importantly, Randomized Scheduling does not require any form of node synchronization and thus is most suitable for server-less VoD systems that do not have any centralized control and management. Since the problem is defined in the packet level, we would expect these results can be easily applied to both stream-based and object-based media, given that the requested media is packed and sent in the scheduled slots.
References 1. A. Oram, Peer-to-Peer: Harnessing the Power of Disruptive Technologies, O’Reilly Press, USA, 2001. 2. M. Baker, R. Buyya, and D. Laforenza, “The Grid: International Efforts in Global Computing,” International Conference on Advances in Infrastructure for Electronic Business, Science, and Education on the Internet, Rome, Italy, 31 July, 2000. 3. Jack Y. B. Lee and W. T. Leung, “Study of a Server-less Architecture for Video-on-Demand Applications,” Proc. IEEE International Conference on Multimedia and Expo., Lausanne, Switzerland, 26–29 Aug 2002. 4. Jack Y. B. Lee and W. T. Leung, “Design and Analysis of a Fault-Tolerant Mechanism for a Server-Less Video-On-Demand System,” Proc. 2002 International Conference on Parallel and Distributed Systems, Taiwan, 17–20 Dec, 2002. 5. D. L. Mills, “Improved algorithms for synchronizing computing network clocks,” IEEE/ACM Transaction on Networks 3, June 1995, pp. 245–254. 6. R. Albert, and A.-L. Barabási, “Topology of Evolving Networks: Local Events and Universality,” Physical review letters, vol.85, 2000, p. 5234. 7. R. Govindan, and H. Tangmunarunkit, “Heuristics for Internet Map Discovery,” IEEE Infocom 2000, Tel Aviv, Israel, Mar. 2000, pp.1371–1380. 8. D. Gross, and C. M. Harris, Fundamentals of Queueing Theory, 3rd ed. New York: Wiley.
A Proxy-Based Dynamic Multicasting Policy Using Stream’s Access Pattern Yong Woon Park1 and Si Woong Jang2 1
Department of Computer Science, Dong-eui Institute of Technology, san72, Yangjung Dong, Pusanjin Ku, Pusan, Rep. of Korea
[email protected] 2 Department of Computer Science, Dong-eui University, san24, Kaya Dong, Pusanjin Ku, Pusan, Rep. of Korea
[email protected] Abstract. This paper proposes a multicasting channel allocation policy in the proxy server that a multicasting channel is not allocated for all objects but for some objects whose access pattern statisfies some conditions. For an object, one unicasting channel is allocated from the origin server to the proxy server and subsequently the proxy server either unicasts or multicasts that object from the proxy server to the client(s) based on the number of avtive streams. The proxy server also caches each object partially or fully based on each object’s access frequency : the cached portion of each object increases as its access frequency increases whereas the cached portion of each object decreases as its acces frequency decreases.
1 Introduction In the network based VoD systems, in order to service customers in a strictly on demand basis, VoD systems usually assigns a transmission stream to each client. So, the network resource requirement to transfer video streams over the network is overwhelming that for several hundred concurrent video streams, several giga bits per second of I/O and network bandwidth are required. Batching[1] is a stream service policy that a group of requests for the same object in a given time interval or batching window are collected and then serviced with one multicasting channel; the batching window size is defined either in terms of time or number of users[3]. So, the multicasting channel allocation scheme can reduce the transfer bandwidth. Moreover, caching media objects on the edge of the Internet become increasingly important in reducing the high start-up overhead and isochronous requirement as well[5]. It is especially effective in a batching environment with deterministic batching window. According to the simulation by [4], there is a considerable improvement in the cache hit ratio and mean disk I/O stream bandwidth reduction by adopting a caching policy. Proxy caching[2] has been widely known to reduce the Internet communication overhead but has mainly concerned with Web files. Some proxy caching schemes for continuous media objects are introduced but they are focused on reducing the initial latency or smoothing the burstiness of the VBR stream[8]. Partial segment caching of H. Kosch, L. Böszörményi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 880–883, 2003. © Springer-Verlag Berlin Heidelberg 2003
A Proxy-Based Dynamic Multicasting Policy Using Stream s Access Pattern
881
media objects was proposed to be combined with a dynamic skyscraper broadcasting to reduce the media delivery cost from a remote server to regional servers[4]. The emphasis in [7] was on caching the initial segments of many media objects and relying on remote multicast delivery of the later segments, rather than fully caching fewer highly popular objects.
2 Multicasting Channel Allocation and Proxy Caching Nodes connected to a LAN usually communicate over a broadcast network, while nodes connected to a WAN communicate via a switched network. In a broadcast LAN, a transmission from any one node is received by all the nodes on the network; thus, multicasting is easily implemented on a broadcast LAN. On the other hand, implementing multicasting on a switched network is quite challenging [6]. Hence, we assumed that in a LAN, both unicasting and multicasting facilities including proxy caching are available whereas in the WAN only unicasting is possible to transfer streams to the proxy server. 2.1 Multicasting Policy Base on Streams’ Arrival Interval 1. A client sends a stream request to the proxy server. 2. The proxy server then immediately allocates one unicasting channel for that stream and transmits the requested object to the client. If the requested object is cached partially or not at all in the proxy server, then the proxy server sends a request on behalf of the client to the origin server and relays it to that client. At the same time, it caches the requested object partially or fully based on the caching status and access pattern of the requested object. 3. If there are more than two active unicasting streams for that object, the proxy server calculates the multicasting window size based on the streams’ average arrival interval so as to minimize the number of the expected maximum concurrent streams. 4. If the calculated multicasting window size is smaller than the average arrival interval, those streams are grouped together into that multicasting window and a multicasting channel is allocated for that multicasting window. Otherwise, those streams will keep their already allocated unicasting channels. 2.2 Phased Proxy Caching th
In our proposed caching policy, caching is done in three steps. Assume that the k th request for object o i and the j new multicasting window for o i is generated.
1.
First phase: In case Oi is not cached at all, as large initial portion of Oi as to hide the initial latency is prefetched 2. Second phase: In case Oi’s caching status is its first phase, part of Oi amounting to the jth multicasting window size is cached.
882
3.
Y.W. Park and S.W. Jang int int Third phase: In this phase, caching is done only when AVGarr (oi )1k > AVGarr (oi )1k −1 th int ( AVGarr (oi )1k is the accumulated arrival rate from the first to the k streams for
Oi); the remaining portion of Oi is cached step by step that as a result, some objects can be cached entirely to avoid server access.
3 Experimental Results In this section, we measure the performance of the proposed dynamic multicasting policy and we use five different notations to express each algorithm. The direct means the case when only unicasting channels are used to service requests. The cast shows the result of our tests when only our proposed multicasting policy is applied. The s+cast(cache size) means the case when our proposed multicasting policy works with the caching policy where the caching step is limited to the second phase and the t+cast(cache size) means the case when our proposed caching policy works with the third phase caching policy. As the graphs in Figure 1 shows, our proposed multicasting channel policy shows about 50% of increased performance as compared with the case when all streams are serviced with unicasting channels without the proposed multicasting policy. Moreover, in case the multicasting policy works with our phased caching policy, it shows more enhanced performance than without caching policy; about 35% of decrease with respect to both the accumulated byte hits and peak time channel numbers. The performance gap between s+cast and t+cast is so small that it can be conjectured that in case of continuous media objects with the
Fig. 1. Performance evaluation with different number of objects
A Proxy-Based Dynamic Multicasting Policy Using Stream s Access Pattern
883
multicasting policy, for an object, caching as much initial portion of that object as the length of the multicasting window is effective rather than caching the entire blocks of that object.
4 Conclusions and Future Research In this paper, we proposed the access frequency-based dynamic multicasting policy based on the phased proxy caching for continuous media stream service. In the proposed caching policy, a multicasting channel for an object is started only after there are more than two active streams for that object. For proxy caching, each object is cached step by step to hide the initial latency as well as to manage cache space efficiently. With our proposed policy, the channel resource usage is reduced by 50%. It is known that when multicasting policy runs, in stead of caching the entire or most of blocks of objects, caching as much portion of each object as the maximum multicasting window size is more effective. In addition, it is also known that increasing proxy capacity cannot reduce the initial latency when the arrival rate is low. Acknowledgement. This work was supported by ECC(Electronic Ceramics Center) at Dong-eui University as RRC•TIC program which is financially supported by KOSEF(Korea Science and Engineering Foundation) under MOST(Ministry of Science and Technology), and ITEP(Korea Institute of Industrial Technology Evaluation and Planning) under MOCIE(Ministry of Commerce, Industry and Energy), and Busan Metropolitan City.
References 1. Aggarwal C. C., Wolf J.L., Yu P.S., “On Optimal Batching Policies for Video-on-Demand Storage Servers”, Proceedings of the Third IEEE International Conference on Multimedia Computing and Systems, 1996, Page(s): 253–258 2. Aggarwal C., Wolf J.L., Yu P.S., “Caching on the World Wide Web”, Knowledge and Data Engineering, IEEE Transactions on Volume: 11 1, Jan.-Feb. 1999 , Page(s): 94–1 3. Debasish Ghose, Hyoung Joong Kim, “Scheduling Video Streams in Video-on-Demand systems: A Survey”, Multimedia Tools and Applications, 11, 167–195, 2000 Kluwer Academic Publishers 4. D. Eager, M. Ferris, and M. Vernon, “Optimized Regional Caching for On-Demand Data Delivery”, in Proc. Multimedia Computing and Networking, Jan 1999 5. K.L. Wu, P.S. Yu. and J.L. Wolf, “Segment-based Proxy Caching of Multimedia Streams,”, th Proceedings of 10 World Wide Web Conference, Elsevier Science, Amsterdam, 2001, pp. 36–44 6. Laxman H Sahasrabuddhe, Biswanath Mukherjee, “Multicast Routing Algorithms and Protocols: A tutorial”, IEEE Network Jan/Feb 2000 pp. 90–102 7. Rexford S., J. Towsley, “Proxy Prefix Caching for Multimedia Streams”, INFOCOM '99. Eighteenth Annual Joint Conference of the IEEE Computer and Communications Societies. Proceedings. IEEE Volume: 3 , 1999 , Page(s): 1310–1319 vol.3 8. Zhi-Li Zhang, Du, D.H.C., Dongli S, Yuewei Wang, "A Network-Conscious Approach to End-to-End Video Delivery over Wide Area Networks Using Proxy Servers”, INFOCOM '98. the 17th Annual Joint Conference of the IEEE Computer and Communications Societies. Proceedings. IEEE Volume: 2, 1998 , Page(s): 660–667 vol.2.
Topic 13 Theory and Algorithms for Parallel Computation Christos Kaklamanis, Danny Krizanc, Pierre Fraigniaud, and Michael Kaufmann Topic Chairs
As theory of modelling and algorithms form the basis for parallel computing, the theory workshop is still a key part of Europar despite of the diversity of the topics of the conference. This year, the submissions mainly concentrate on algorithms to improve communication issues. Only a few contributions to other topics made it. In more detail, Schmollinger presents a new algorithm for parallel radix sort emphasizing on the communication especially for unbalanced data and demonstrates considerable improvement compared to the traditional approach. In a new paper on parallel list ranking, Sibeyn considers diverse algorithms under the aspect of global communication, and presents a new strategy and analysis when local communication is prefered. The third paper by Laforest considers an interesting network design problem where a certain kind of graph spanner has to be constructed, and gives nonapproximability results as well as efficient algorithms for trees. Cordasi, Negro, Rosenberg and Scarano addresses the mapping problem of data structures to parallel memory modules in the paper “c-Perfect Hashing Schemes for Binary Trees, with Applications to Parallel Memories”. A non-optimal but simple and elegant algorithm for parallel multiplication of large integers has been developed by Bunimov and Schimmler. Their parameter and analysis techniques are adapted from VLSI theory. The paper “A Model of Pipelined Mutual Exclusion on Cache-Coherent Multiprocessors” by Takesue has a more practice-oriented topic. A new model of pipelined mutual exclusion is proposed and evaluated by diverse simulations. Summarizing, we found the collection of papers satisfying and diverse enough to enlight several theoretical aspects of parallel computation. Finally, we wish to thank all the authors and the referees for their efforts.
H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, p. 884, 2003. c Springer-Verlag Berlin Heidelberg 2003
Improving Communication Sensitive Parallel Radix Sort for Unbalanced Data Martin Schmollinger Wilhelm-Schickard-Institut f¨ur Informatik, Universit¨at T¨ubingen
[email protected] Abstract. Radix sort is an efficient method to sort integer keys on parallel computers. It is easy to parallelize and simple to implement. The main drawbacks of existing algorithms are load balancing problems and communication overhead. These problems are caused in data characteristics like data-skew and duplicates. There are several approaches how to parallelize the radix sort algorithm, which yield to reduce communication operations or to improve the load balance. If an algorithm has its focus on the optimization of the load balance then, its communication is inefficient. Otherwise, if the focus is on the communication minimization, then the algorithms are only efficient for well-distributed data. For the latter case, we will present an efficient improvement which helps to overcome the problems with unbalanced data characteristics. The suggested improvements are tested practically on a Linux-based SMP cluster.
1
Introduction and Related Work
Radix Sort is a method to sort a set of integer keys using their binary representation. The main idea is to sort the keys in several iterations. In each iteration the keys are sorted according to a certain number of bits (radix of size r), starting in the first iteration with the least significant bit. Assuming the keys consist of l bits, the algorithm needs l/r iterations to sort the keys. Detailed descriptions of sequential radix sort algorithms can be found in standard textbooks, see e.g. [13,10]. The main idea of the parallelization is that the processors are viewed as buckets. Roughly, each parallel iteration of radix sort consists of the following 3 steps. Each processor scans its local keys with a certain radix and stores them in the corresponding buckets. Then the sizes of the locally created buckets are exchanged by the processors in order to calculate a communication pattern for the buckets. The keys are exchanged according to the computed communication pattern. These steps are repeated until all bits of the keys were scanned. More detailed descriptions for shared-memory and distributedmemory machines can be found in [1,8]. The main problem with this approach is its irregularity in communication and computation which arise because of data characteristics like e.g. data skew or duplicates. Therefore, in [15] a load balanced parallel radix sort is presented. This algorithm splits the locally generated buckets with respect to balance the number of keys that have to be sent to each processor. Therefore, the resulting communication step is a real balanced all-to-all operation by the processors. H. Kosch, L. B¨osz¨orm´enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 885–893, 2003. c Springer-Verlag Berlin Heidelberg 2003
886
M. Schmollinger
Until now, all mentioned algorithms need l/r key- and counter-communication steps. Especially for distributed-memory machines, where the communication is done over an interconnection network, this might be very time consuming. The communication and cache conscious radix sort algorithm (C3-Radix) [7] tries to improve this situation by starting the radix sort at the most significant bit. The intention of C3-Radix is to partition the data into several buckets which can be distributed equally among the processors. Therefore, the choice of the radix is very important. If the distribution of the buckets is not possible in a balanced way, the radix is enlarged and more fine-grained buckets are created. This is repeated until a good load-balance is achieved. Then the keys are sent with an all-to-all operation among the processors. The remaining task is to sort the buckets locally. For this step, the authors use a very efficient cache conscious radix sort algorithm which uses the fact that the data is already sorted for a certain number of bits. The algorithm has to communicate the keys only once, but because of the characteristics of the data there might be several iterations where the bucket counters have to be exchanged. Depending on the number of iterations and the size of the radix this can increase the running time of the algorithm extremely. Another approach is based on Sample Sort which is often used in parallel sorting, see e.g. [6,3,2,14]. Each processor samples q keys from its N/P keys and exchanges them with the other processors. Sorting the set of samples in each processor makes it possible to create a set of s − 1 < q equidistant keys called splitters. These splitters can be used to create s buckets of approximately size N/s. Load balance can be achieved, if s is a multiple of P . The parallel counting split radix sort algorithm (PCS-Radix) [9] uses Sample Sort for the partitioning of the data instead of using the radix of the keys directly. Radix sort is only used to sort the buckets locally. The prize for the independence of the data characteristics is the detection of global samples in the distributed system, but in a badly distributed environment this investment is worth doing. Despite of that, in environments where the data is mostly well distributed, the C3-Radix algorithm should be the algorithm of choice, because each processor is able to start the local creation of the buckets directly, and in general no further iterations are necessary to build the local buckets. On the other hand, also in a well-distributed environment, sometimes there may arise badly distributed data sets. The C3-Radix algorithm does not have the capability of being efficient in these cases, but it should be guaranteed that it is not far away from being efficient. Hence, in the following we survey the possibilities for C3-Radix to be more stable and more predictable when working on unbalanced data distributions. In Section 2, we sketch the C3-Radix without going into detail for the steps we will not improve and without describing their optimizations made in [7]. In Section 3, we will explain the problems of C3-Radix in cases of unbalanced data more deeply, we will give an example, and we will describe improvements. In Section 4 we will give and interpret results of experimental test for the improved algorithm, and in Section 5 we conclude.
2
Outline of the C3-Radix Algorithm
The algorithm sorts N keys and each key consists of l bits. The used radix has length r. Initially, each processor stores N/P keys. The keys are globally sorted if the keys
Improving Communication Sensitive Parallel Radix Sort for Unbalanced Data
887
within each processor are sorted, and there is a known order of the processors for which all keys of one processor are greater oder equal to all keys in the preceding processor. Each processor builds buckets of keys by starting to observe the first r bits of each key. The initial length of the radix should be chosen in the way that 2r > P . Keys with the same radix belong to the same bucket. (1) Reverse Sorting. Each processor scans its N/P integer keys using the first r bits beginning with the most significant bit, and building the corresponding 2r buckets. During the creation of the buckets a counter array is built, too. Each entry in the counter array contains the number of keys in the corresponding bucket. (2) Communication of Counters. The local 2r counters are exchanged between the processors. After this step, each processor knows the total amount of elements per bucket. (3) Computation of bucket distribution . Each processor computes locally a distribution of the buckets to the processors. If it is not possible to achieve a good load balance, then each processor starts again with step 1 and sets the new radix to ir, where i is the number of iterations. By extending the radix, the algorithm tries to produce more and smaller buckets that may lead to a better load balance. Otherwise the algorithm continues with step 4. (4) All-to-All key communication. The buckets are sent in an all-to-all fashion. After this step no more communication is necessary. (5) Local sorting.
3
Improvements
As we can see in Section 2, the main problem with the C3-Radix algorithm is that the first 3 steps may have to be repeated several times. The number of iterations depends on the data distribution, the initial radix chosen and the size of the integer keys. The more iterations are necessary, the bigger the radix gets. Since the number of buckets as well as the number of counters is 2r , the allocated memory and the amount of data, that has to be communicated may increase the running time of the algorithm tremendously. What we want to improve is the way, the algorithm tries to distribute the data equally among the processors if more than 1 iteration is necessary. All other optimizations of the C3-Radix algorithms will not be changed. In order to explain the problem more detailed, we give an example. Let i be the number of iterations, then the accumulated number of counters i in each processor is j=1 2(j+1)r . In the experiments of [7] the radix is set to 5. If we assume there is a data distribution that leads to 6 iterations, then the number of counters broadcasted over the network is 1,108,378,656. If we further assume that a counter is a 32 bit integer (which is necessary for large N), then we see that each processor has to communicate about 4.129 GB of data. Each processor sends its counter array to all the other P −1 processors. Assuming 16 processors, the total data transferred by the network is 990.96 GB. Besides this communication problem, there might arise memory problems. At least each processor has to store the counter array and the buckets constructed locally. Although there are more data structures like for example the initial data (N/P elements) or different counter arrays for the communication operation, the memory demand is dominated by the counter array. Therefore, we take its estimated memory demand as a key figure. Since the counter array has size 230 in the sixth iteration and each entry stores one 32 bit integer, each processor needs at least 4 GB of main memory. Depending on the implementation of the counter communication (allreduce, broadcast, ..) the node has
888
M. Schmollinger
to buffer up to P counter arrays which leads to 64 GB in our example. For a huge part of supercomputers, namely the PC-based SMP- or workstation-clusters, this size is not manageable. The program will abort due to out of memory errors. In order to avoid these large data arrays, our idea is not to rebuild all the buckets and counters with a larger radix within each iteration, but only to rebuild those that lie directly on at least one border of two processor data domains. (The data domain of a processor is its part of the sorted sequence of N integers. The N/P smallest integers are the data domain of processor 0, the next smallest N/P integers are the data domain of processor 1, and so on). Hence, at most P − 1 buckets need to be rebuilt. In the first iteration the algorithm works as described in Section 2. The only difference is that after checking if another iteration is necessary, only selected buckets are rebuild. For further iterations the first 3 steps are replaced by the following steps. (1) Reverse Sorting of special buckets. Rebuild the buckets and counters that lie directly on the border of two processor data domains. (2) Communication of Counters. All-to-all communication of the new counter array. (3) Computation of bucket distribution. Check, if another iteration is necessary. If it is necessary, detect the buckets that have to be rebuilt with a higher radix. Otherwise proceed to step 4 of the algorithm described in Section 2. Now, we want to analyze the behavior of this new algorithm in order to judge the received improvement. In each iteration, a certain number of buckets have to be rebuild. We denote this number with aj , where 1 ≤ aj < P and j > 0 is the number of the iteration (a1 = 0). Let i be the number i amount i of iterations performed. The accumulated of counters in each processor is j=1 2r + aj 2r − aj = i2r + (2r − 1) j=1 aj . Looking at the scenario above (r = 5, i = 6, P = 16) and assuming the worst-case, where aj = P − 1 for j > 1 the number of counters broadcasted over the network is 2517. Again we assume that a counter is a 32 bit integer, then we see that each processor has to communicate about 9.83 KB. In each iteration each processor sends its counter array to the other P − 1 processors. Hence, the overall amount of counters transferred by the network is 2359.2 KB. The situation is also much better for the memory requirements. Storing the counter array in step 6 requires 19.66 KB per processor which normally is negligible compared to custom computer memory sizes. It should fit in the cache! Again, the maximum temporary buffer size needed for the communication and reduction step of the counters is 314.56 KB which is not critical, too. The analysis and the example show the potential of the formulated improvement. Until now, we have just looked how the number of counters increases in both methods depending on the number of iterations. But we also have to care about the way the new buckets are created locally in the step 1 (Reverse Sorting). C3-Radix scans its N keys and rebuilds all buckets using the new radix. While building the buckets the counter array is updated, too. The scan can be done in O(N ), and an update of the buckets and counters can be done in O(1). A practical drawback of this method is the following: As we know, the array of buckets and counters may get very big. Scanning the data means to access these arrays very irregularly. Hence, the memory hierarchy of the system is not
Improving Communication Sensitive Parallel Radix Sort for Unbalanced Data
889
used efficiently, and the time needed for step 1 will grow quickly beginning at a critical size of the radix. On the other side, the improved version does not always have to scan all local integer keys. It just scans the keys located in the buckets, which were decided to be rebuild. But in the worst case all local keys are contained in such buckets. Further, these N keys are not stored in one array consecutively, therefore, there is an additional overhead for switching between the buckets. After scanning the buckets and building new buckets out of them, these buckets have to replace the old buckets. Assuming that all buckets are stored in an array, this operation can be done in O(2r ), whereby r is the size of the initial radix in the first iteration. For small initial radix sizes N should be ≥ 2r , hence, the additional time needed is not a problem. For larger sizes, step 1 is not the bottleneck of the algorithm as we saw in the example. Despite of that, we see that there might be situations for step 1, where C3-Radix is better than the improved version and vice versa, depending on the data distribution and the size of the radix. In the following sections, we present experimental results that will show the behavior of the two algorithms for several data distributions. The improved algorithm is called balanced communication sensitive parallel radix sort (BCSP-Radix).
4
Experimental Tests
Our experimental tests were made on a Linux-SMP cluster with two Pentium III processors (650 MHz) and 1 GB main memory per node. The nodes are connected by a Myrinet 1.28 GBit/s switched LAN. The whole cluster consists of 98 nodes (Kepler-Cluster, [16]). As we showed in Section 3, it is sufficient to compare the first 3 steps of C3and BCSP-Radix , because the steps 4 and 5 are the same as in the C3-Radix algorithm. We implemented them with C++ and TPO++ [4], which is an object-oriented message-passing system build on MPI [11,12]. Concerning the data distributions, we use two kinds of data sets. The first type are distributions already used in [15,7,9,5] and, therefore, are called standard data distributions. The second type are data distributions, which leads to worst-case behavior. We will explain them later on page 891. Duplicates are allowed in all data sets. For all experiments we try to achieve the best load balance possible. Since duplicates are allowed, a perfect load balance is not always possible. In general, we accept deviations of ≤ 1% of N/P . Our four standard data distributions are defined as follows, in which MAX is (231 − 1) for the integer keys, see also [15,7,9,5]. Random [R], the data set is produced by calling the C library random number generator random() consecutively. The function returns integer values between 0 and 231 − 1. Gaussian [G], an integer key is generated by calling the random() function 4 times, adding the return values and dividing the result by 4. Bucket Sorted [B], the generated integer keys are sorted into P buckets, obtained by setting the first N/P 2 keys at each processor to be random numbers in the range of 0 to (MAX/P − 1), the second N/P 2 keys in the range of MAX/P to (2MAX/P − 1), and so forth. Staggered [S], if the processor index i is < P/2, then all N/P integer keys at the processor are random numbers between (2i + 1)MAX/P . Otherwise, all N/P keys are random numbers between (i−P/2)MAX/P and ((i−P/2+1)MAX/P −1).
890
M. Schmollinger
The main problem of using radix sort is that we do not know how the data is distributed and, therefore, with which size of the radix we should start. Our aim is to minimize the total running time. But without knowing details about the data it is not possible to choose the size of the radix optimally. The algorithm should grant that the running time should be close or equal to its optimum independent of the radix size and data distribution . Hence, in our test, we varied the initial size of the radix from 5 to 12. We made experiments using 16M integer keys and 8, 16 and 32 processors. All tests produced similar results, therefore, we give the explanation of the results by means of the 16 processors test. The running times of this test are presented in Table 1. Table 1. Test with 16 processors (8 nodes with 2 processors) and 16M integer keys using 4 different data distributions, and varying the initial radix in a range from 5 to 12.. The first number in each cell is the mean running time of all used processors for the first 3 steps. The second number is the number of iterations needed for the execution
radix [R] [G] [B] [S]
5 0.95|1 2.63|3 2.87|3 2.76|3
6 1.06|1 2.42|2 2.87|3 2.53|3
7 1.12|1 1.85|2 2.51|2 2.94|2
radix [R] [G] [B] [S]
5 0.96|1 2.87|3 2.47|3 2.49|3
6 1.05|1 1.63|2 4.45|3 4.23|3
7 1.16|1 1.92|2 1.67|2 1.75|2
BCSP-Radix 8 9 10 1.24|1 1.24|1 1.23|1 1.68|2 1.71|2 1.77|2 2.44|2 2.10|2 1.94|2 2.34|2 1.98|2 1.79|2 C3-Radix 8 9 10 1.23|1 1.21|1 1.19|1 2.83|2 4.65|2 10.74|2 2.09|2 3.74|2 10.02|2 2.05|2 3.62|2 9.62|2
11 1.20|1 1.07|1 0.93|1 1.01|1
12 total mean 1.15|1 1.15 1.11|1 1.78 0.96|1 2.08 1.09|1 2.05
11 1.16|1 1.06|1 0.93|1 0.99|1
12 total mean 1.14|1 1.14 1.12|1 3.35 0.96|1 3.29 1.08|1 3.23
Concerning the [R] distribution, both algorithms achieve similar running times for all sizes of the radix because, if only 1 iteration is necessary then they are the same. In this case, the best running time can be achieved by using the smallest radix, which is obvious, because then the data structures are small, too. Computation and communication benefit by that. In general, the overall best running time for all distributions can be achieved, if we use the smallest size of the radix, with which only 1 iteration is necessary. In our sample this is r = 11 for [G], [B] and [S]. Again, both algorithms are the same. If the radix is chosen < 11, then the situation is as follows. If the number of iterations needed to perform the algorithm is equal for several sizes of the initial radix, then C3Radix is better if the size of the radix is minimal and BCSP-Radix is better for larger sizes of the radix. The reason why C3-Radix gets slower for larger sizes of the radix is founded in the increasing sizes of the bucket and counter arrays. Communication and computation gets much higher if the radix size increases dramatically due to further iterations. BCSP-Radix is much more stable, because the more iterations are necessary, the smaller the buckets are that have to be rebuild. Furthermore, the size of the counter
Improving Communication Sensitive Parallel Radix Sort for Unbalanced Data
891
array for the communication is bounded by O(2r ) as we saw in Section 3. Therefore, the communication cost is stable, too. If BCSP-Radix is worse, this is because of the running time of step 1. In these cases, due to the small radix, the data is distributed in a small number of buckets (≤ P − 1). Hence, each bucket has to be rebuild and each processor has to scan all integer keys stored locally. The additional time needed for replacing all buckets by 2r new buckets leads to the worse behavior. An essential observation can be made looking at the mean of the processor mean times (see column total mean in Table 1). BCSP-Radix is better for [S], [B], [G] and of course equal for [R]. While C3-Radix has some runaways in all distributions, BCSP-Radix is much more stable. This is important, because normally we do not know the behavior of the data with respect to the chosen radix. By using BCSP-Radix it is guaranteed that the total running time of the whole algorithm is not destroyed by the first 3 steps. For the BCSP-Radix algorithm the worst-case data-distribution can be described as follows. After the first iteration, the data is partitioned into P − 1 buckets of equal size and the other buckets are empty. The algorithm decides to rebuild all P − 1 buckets with a larger radix. But the data is chosen in the way that the next bits are the same for all keys until the last r bits begin. That means, that in each iteration the P − 1 buckets remain unchanged until the last iteration. This is the worst case, because BCSP-Radix has to scan all keys in each iteration, and P −1 is the maximum number of buckets which achieve this situation. Using this data set with C3-Radix does also lead to the maximum number of iterations possible. This data set is not constructed very artificially. The keys are uniformly distributed within a small range of bits and duplicates are allowed. Hence, this might also occur in a well-distributed environment. As we know from the example in Section 3, we cannot perform all iterations for 32 bit keys with the C3-Radix, because the main memory of our SMP-nodes is limited to 1 GB (using only 1 processor per node!). Therefore, the data set is constructed in the way that the nodes of the cluster will not collapse (≤ 20 bits). But we will show the behavior of BCSP-Radix performing all possible iterations. Fig. 1 (left) presents an example of the comparison between the algorithms, where the data is distributed in the way that both algorithms have to perform 4 iterations, which means that the data can be partitioned looking at the first 20 bits. BCSP-Radix is much better than C3-Radix. The difference between both would even grow if further iterations were necessary. Unfortunately, C3-Radix cannot iterate further without aborting due to memory limitations. In Fig. 1 (right), the running times for BCSP-Radix making 6 iterations (30 bits) are illustrated. We have to adhere that in this case BCSP-Radix is even faster than C3-Radix performing only 4 iterations. Although this is a worst-case situation for BCSP-Radix, the running time is better than the worst time achieved with C3-Radix for [S], [B] and [G] and radix 10 where 2 iterations were necessary (see Table 1).
5
Conclusions
In the previous sections, we suggested improvements with respect to the data decomposition for the currently fastest parallel radix sort algorithm. Instead of rebuilding all buckets for each iteration with a larger radix size, BCSP-Radix only rebuilds those buck-
892
M. Schmollinger
Algorithm Step 1 Step 2 Step 3 C3 4.79 5.88 0.15 10.82 BCSP 5.35 0.02 < 0.01 5.37
Algorithm Step 1 Step 2 Step 3 BCSP 7.72 0.04 < 0.01 7.76
Fig. 1. Test with 16 processors, 16M integer keys, and an initial radix of 5. The numbers are the mean times (in sec.) of the processors for a worst case situation. (left) 4 iterations are necessary. (right) 6 iterations are necessary and only BCSP-Radix is able to terminate its execution
ets which help to find a better load balance for the proceeding steps of the algorithm. Due to this optimization, the computation, communication and memory requirements are much better. In experimental tests, we showed that the average general behavior of BCSP-Radix for standard data distributions is superior. While C3-Radix has several situations, where the execution time runs away, BCSP-Radix behaves much more stable and predictable. For worst-case data distributions the advantage of using BCSP-Radix is even bigger. While the non-optimized algorithm may not terminate due to memory constraints, BCSP-Radix is even in the position to achieve considerable running times. Even the memory requirements per processor are significantly within the capacity of custom workstations.
References 1. D. A. Bader and J. J´aJ´a. SIMPLE: A Methodology for Programming High Performance Algorithms on Clusters of Symmetric Multiprocessors (SMPs). Journal of Parallel and Distributed Computing, 58(1):92–108, 1999. 2. G. E. Blelloch, C. E. Leiserson, B. M. Maggs, C. G. Plaxton, S. J. Smith, and M. Zagha. A Comparison of Sorting Algorithms for the Connection Machine. In Proceedings of Sysmposium on Parallel Algorithms and Architectures, pages 3–16, July 1991. 3. A.V. Gerbessiotis and C. J. Siniolakis. Deterministic Sorting and Randomized Median Finding on the BSP Model. In Proceedings oth the 8th ACM Symposium on Parallel Algorithms and Architectures, pages pp.223–232, 1996. 4. T. Grundmann, M. Ritt, and W. Rosenstiel. Object-Oriented Message-Passing with TPO++. In EURO-PAR 2000, Parallel Processing, volume 1900 of LNCS, pages 1081–1084. Springer Verlag, 2000. 5. D. R. Helman, D. A. Bader, and J. J´aJ´a. Parallel Algorithms for Personalized Communication and Sorting With Experimental Study. In Proceedings of the IEEE Annual ACM Symposium on Parallel Algorithms and Architectures, pages 211–220, 1996. 6. D. R. Helman and J. J´aJ´a. Sorting on Clusters of SMPs. Informatica: An International Journal of Computing and Informatics, 23, 1999. 7. D. Jiminez-Gonzales, J. Larriba-Pey, and J. Navarro. Communication Conscious Radix Sort. In Proceedings of the International Conference on Supercomputing, pages 76–82. ACM, 1999. 8. D. Jiminez-Gonzales, J. Larriba-Pey, and J. Navarro. GI-Seminar: Algorithms for Memory Hierarchies, volume 2625 of LNCS, chapter Case Study: Memory Conscious Parallel Sorting, pages 358–378. Springer Verlag, 2003. Advanced Lectures. 9. D. Jiminez-Gonzales, J. Navarro, , and J. Larriba-Pey. Fast Parallel In-Memory 64 Bit Sorting. In Proceedings of the International Conference on Supercomputing, pages 114–122. ACM, 2001.
Improving Communication Sensitive Parallel Radix Sort for Unbalanced Data
893
10. D. Knuth. The Art of Computer Programming: Sorting and Searching, volume 3. AddisonWesley, 1973. 11. Message Passing Interface Forum. MPI: A Message-Passing Interface Standard. Technical Report CS-94-230, Computer Science Department, University of Tennessee, Knoxville, TN, May 1994. 12. Message Passing Interface Forum. MPI-2: Extensions to the Message-Passing Interface, July 1997. 13. R. Sedgewick. Algorithms. Addison-Wesley, 1992. 14. H. Shi and J. Schaeffer. Parallel Sorting by Regular Sampling. Journal of Parallel and Distributed Computing, 14:361–372, 1992. 15. A. Sohn and Y. Kodama. Load Balanced Parallel Radix Sort. In Proceedings of the International Conference on Supercomputing, pages 305–312. ACM, 1998. 16. University of T¨ubingen (SFB-382). http://kepler.sfb382-zdv.uni-tuebingen.de/.
Minimizing Global Communication in Parallel List Ranking Jop F. Sibeyn Institut f¨ur Informatik Universit¨at Halle-Wittenberg 06120 Halle, Germany. http://www.mpi-sb.mpg.de/˜jopsi/
Abstract. Consider a parallel computer on which it is considerably cheaper to perform communications within close neighborhoods than globally. In such a context, it is not necessarily a good idea to simulate optimal PRAM algorithms. In this paper consider the list-ranking problem and present a new divide-andconquer algorithm. For lists of length N , it performs O(N · log N ) work), but its communication is mainly local: there are only 6 · 2i communication operations within subnetworks of size P/2i .
1
Introduction
A linked list, hereafter just list, is a basic data structure: it consists of nodes which are linked together, so that every node has precisely one predecessor and one successor, except for the initial node, which has no predecessor, and the final node, which has no successor. The rank of a node is defined as the number of links that must be traversed to reach the final node of the list. An important problem connected to the use of lists is the list ranking problem: the determination of the ranks of all nodes. Typically, the set of links constitutes a set of lists, and in that case for each node one also wants to compute the index of the last node of its list. Parallel list-ranking is a key subroutine in more complex problems such as edge coloring bipartite graphs and in the lowest-common-ancestor problem. Parallel list ranking has been considered intensively [16,2,1,5,10,11,4,9,14], but there are still aspects which deserve further consideration. Imagine that one would like to solve a list-ranking problem on all the computers of a big institute or using even more distributed resources (using more computers also gives more main memory, for large problems this may avoid the considerable slow-down due to paging). Communication within small subnetworks may be performed without noticeable delay, while for more global communication the finite bandwidth of the network at its various levels may strongly reduce the processor-to-processor rate when all processors are communicating at the same time. In such a context, the focus of an algorithm should lie on minimizing the communication at the more global levels of the hierarchy, even if this implies some extra communication and work at the more local levels. In this paper we present an alternative algorithm with the following unique feature: there are only 6 · 2i operations within subnetworks of size P/2i . In other words: most communication is performed locally. H. Kosch, L. B¨osz¨orm´enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 894–902, 2003. c Springer-Verlag Berlin Heidelberg 2003
Minimizing Global Communication in Parallel List Ranking
2
895
Goal, Preliminaries, and Cost Model
The main purpose of this paper is to present and analyze an alternative list-ranking algorithm which implies an enrichment of the pool of available algorithms. The algorithm is formulated in a general recursive way with a cost expression given by a recurrence relation which is valid for any parallel computer model. Substituting the cost function for communication on the respective computer model allows to solve the recursion. In the full version of the paper we consider the performance on linear processor arrays and show that on this network the new algorithm is better than earlier ones and consequently also on all networks in which global communication is penalized even stronger. The input consists of an array succ[] of length N . The values in the array are all different. If succ[u] = v, with 0 ≤ v < N , then this is interpreted as v being the successor of u. In this case we will say that node u points to node v. If v < 0 or v ≥ N , this is interpreted as u being the last node of its list. In this case we will say that u points to null, in an implementation it is convenient if then v = u + N . There are P PUs and the information related to node u is stored in PU u · P/N . Throughout this paper, for all algorithms we are considering, we are assuming that either all nodes were randomly renumbered at the beginning of the algorithm, or that we had a list given by a random permutation to start with. The communicational complexity of such a randomization is low: essentially it costs the same as routing all data through the network once. Alternatively, the data might be allocated to the PUs with help of a suitable randomly selected hash function mapping {0, 1, . . . , N − 1} to {0, 1, . . . , P − 1} in a sufficiently regular way. This assumption allows us to work with the expected sizes of the packets, in the analysis of the communication cost. For sufficiently large packets (as one should have anyway in order to amortize the cost to start-up a packet) the deviations from the expected values are small. Our algorithm is based on the availability of two communication primitives. One is the balanced k-k routing, in which each processing unit, PU, sends k packets in total, k/P to each PU. The other is a balanced k-k swapping, in which, with respect to a given division of the network in two halfs, each PU in one half sends 2 · k/P packets to each PU in the other half. The time for these operations is denoted by Troute (P, k) and Tswap (P, k), respectively. On linear arrays Tswap (P, k) = 2 · Troute (P, k). On hierarchical networks the factor may be larger.
3
One-by-One Cleaning
All well-known parallel list-ranking algorithms perform are based on global all-to-all routings. Only the ‘one-by-one cleaning’ algorithm [13] solves the list ranking problem by a sequence of permutation routings. Its main ideas are repeated here, because the new algorithm is an alternative workout of the same underlying idea. We define the concepts ‘clean’ and ‘cleaning’. Let S1 and S2 be subsets of the PUs. We say that S1 is clean for S2 if none of the nodes stored in a PU from S1 has a successor in S2 . Cleaning S1 for S2 means to somehow achieve that S1 is clean for S2 . In this context we distinguish two operations: Autoclean(i): to clean PU i for itself; Altroclean(i, j): to clean PU i for PU j.
896
J.F. Sibeyn
For any PU i, autoclean(i) can be performed without any communication by following the links until a final node or a link pointing to a node in another PU is found. For any PU i and j, altroclean(i, j) can be performed by performing autoclean(j) followed by a single pointer-jumping step: for each node u in PU i with succ[u] = v in PU j, we perform succ[u] = succ[succ[u]] = succ[v]. Because we had first performed autoclean(j), we can be sure that v does not point to another node in PU j. In terms of autoclean and altroclean it is easy to formulate a list-ranking algorithm for a two-processor system, requiring only two routing operations independently of N : 1. For each i ∈ {0, 1}, autoclean(i); 2. For each i ∈ {0, 1}, altroclean(i, (i + 1) mod 2); 3. For each i ∈ {0, 1}, autoclean(i).
Initial Situation
After Autoclean
After Altroclean
Fig. 1. List ranking on a two-processor system. After the first autoclean, all links point to nodes in the other PU, except for the final node which has a successor value not corresponding to any node index. After the altroclean, all non-final nodes point to a node in the own processor. One more autoclean solves the problem.
The problem with extending this idea to arbitrary P is that when PU i is clean for PU j and we perform altroclean(i, k), for some other k, that then, if we do not take special care, we get again links to nodes stored in PU j, and PU i is no longer clean for PU j. In one-by-one cleaning, in each PU there are two sets of links, which are cleaned in different ways so that this problem is avoided. The algorithm consists of P − 1 rounds. An invariant is that after round t, 0 ≤ t < P , the ‘left-set’ of links in PU i is clean for {i, i − 1, . . . , i − t} and the ‘right-set’ of links is clean for {i, i + 1, . . . , i + t} (all indices are computed modulo P ). Then, in round t + 1, the left-set of PU i is cleaned for PU i − t − 1 with help of the right-set of PU i − t − 1, and the right-set of PU i is cleaned for PU i + t + 1 with help of the left-set of PU i + t + 1. One more autoclean achieves the invariant for t + 1. The unique feature of this algorithm is that in any communication operation each PU is sending to only one other PU, the whole pattern constituting a permutation.
Minimizing Global Communication in Parallel List Ranking
4
897
Divide and Conquer
The guiding idea in one-by-one cleaning is that if PU i is clean for PU j, one should be careful to keep it clean for PU j during the rest of the execution. This is a good strategy but also appears to require the given structure, not leaving room for further improvement. In this section we consider a divide-and-conquer algorithm using the same idea of alternating autoclean and altroclean. The algorithm is a variant of the ‘repeatedhalving’ algorithm [12] modified so as to minimize the number of global communication operations. With respect to a set S of the PUs, we will say that node u is final in S if the node succ[u] = v is not stored in a PU of S. In the following, if it is clear from the context which set S is meant, we will just say that node u is final. We say that the list-ranking problem has been solved for a subset S of the PUs, if each node stored in a PU from S knows a final node stored in a PU of S. More precisely, we specify that after solving the problem for S, succ[u] should be unchanged for each final node u in S, but for all non-final nodes in S, afterwards succ[u] = w, for some final node w in S. In the following we present a recursive algorithm for solving the problem for S1 ∪ S2 , for two subsets of the PUs S1 and S2 . The algorithm is actually quite simple, but it consists of many steps. These steps are divided in 6 phases. Two of these phases solve subproblems, the other four phases consist of a round of questions, answers and updates. Before the pseudo-code of the non-recursive phases a comment explains what is happening. 1. Solve the problem for S1 and for S2 ; /* As a result of the solving in each subset all non-final nodes now know a final node. Each final node in S1 and S2 which is not a final node in S1 ∪ S2 figures out the subsequent final node, and updates its succ-value accordingly. */ 2. In each set, for each final node u pointing to a node v in the other subset, send a question to node v = succ[u]; 3. In each set, for each non-final node v which received a question sent in step 2, send a question to node w = succ[v]; 4. In each set, for each node w which received a question sent in step 3 from a node v, send an answer back to node v. If w is a final node in S1 ∪ S2 , then the answer consists of a placeholder, otherwise w returns succ[w] and distance information; 5. In each set, for each node v which received a question sent in step 2 from a node u, send an answer back to node u. If v is a final node in S1 ∪ S2 , then the answer consists of a placeholder. Otherwise, if it is a final node in its own set or if it received a placeholder in step 4, v returns succ[v] and distance information, else v returns succ[w] = succ[succ[v]] and distance information; 6. In each set, for each final node u, which received a non-void answer sent in step 5 in reply to its question sent in step 2, update succ[u] and distance information; /* As a result of the update, a final node in S1 may now point to a non-final in S1 , and similarly in S2 . Such nodes now ask these non-final nodes for the final node they are pointing to, and update their succ-values accordingly. */
898
J.F. Sibeyn
7. In each set, for each formerly final node u, which has become non-final as a result of the update in step 6, send a question to node v = succ[u]; 8. In each set, for each node v which received a question sent in step 7 from a node u, send an answer back to node u. If v was a final node before the update in step 6, then the answer consists of a placeholder, otherwise v returns succ[v] and distance information. 9. In each set, for each formerly final node u, which received a non-void answer sent in step 8 in reply to its question sent in step 7, update succ[u] and the distance information; /* As a result of the updates, all final nodes point to final nodes again. A final node u in S1 points to a node v in S2 only when v is a final node in S1 ∪ S2 , and similarly for the final nodes in S2 . The subproblems given by the linking of the final nodes are now solved recursively. */ 10. Solve the problem for all formerly final nodes in S1 and S2 which became nonfinal as a result of the update in step 6. /* Now each final node in S1 is either a final node in S1 ∪ S2 or points to such a node, or points to a node which points to such a node, and similarly for the final nodes in S2 . A final node in this last category asks the node it is pointing to for its successor and updates its succ-value accordingly. */ 11. In each set, for each formerly final node u, which has become non-final as a result of the update in step 6, send a question to node v = succ[u]; 12. In each set, for each node v which received a question sent in step 11 from a node u, send an answer back to node u. If v is a final node in S1 ∪ S2 , then the answer consists of a placeholder, otherwise v returns succ[v] and distance information. 13. In each set, for each formerly final node u, which received a non-void answer sent in step 12 in reply to its question sent in step 11, update succ[u] and the distance information; /* Now each final node in S1 is either a final node in S1 ∪ S2 or points to such a node, and similarly for the final nodes in S2 . Each non-final node asks the final node it is pointing to for its successor and updates its succ-value accordingly. As a result all nodes are either final nodes in S1 ∪ S2 or are pointing to such a node. */ 14. In each set, for each non-final node u, send a question to node v = succ[u]; 15. In each set, for each node v which received a question sent in step 14 from a node u, send an answer back to node u. If v is a final node in S1 ∪ S2 , then the answer consists of a placeholder otherwise v returns succ[v] and distance information. 16. In each set, for each non-final node u, which received a non-void answer sent in step 15 in reply to its question sent in step 14, update succ[u] and the distance information. The steps of the algorithm are illustrated in Figure 2. The algorithm will be used by calling it with S1 and S2 each consisting of P/2 PUs, and then recursively dividing
Minimizing Global Communication in Parallel List Ranking
Initial Situation
P’ P/2
After Step 1
P/2
P’ P/2
After Step 6
P’ P/2
P/2
After Step 9
P/2
P’ P/2
After Step 10
P’ P/2
899
P/2
After Step 13
P/2
P’ P/2
P/2
After Step 16
P’ P/2
P/2
Fig. 2. The stages of the recursive list-ranking algorithm. On the right side of each picture only the nodes are shown which are still relevant for the nodes on the left side.
these sets. On a single PU, the problem is solved internally without further recursion. When answering a question (Step 4, 5, 8, 12, 15) from a node u we distinguish the case that the question was actually unnecessary because u was already pointing to a node with the desired property. In this case succ[u] should not be updated, which is signaled to u by sending a placeholder.
900
J.F. Sibeyn
The given algorithm constitutes an alternative generalization of the idea for the twoprocessor system depicted in Figure 1. Here we are solving (= searching for the last node in the own subset of PUs) rather than cleaning (= searching for the first node in the other subset of PUs) for efficiency reasons. This detail has a considerable impact on the recurrence relation we will get. The operation of the algorithm is summarized in the following lemma: Lemma 1 Let u be any node stored in a PU in S1 ∪ S2 , let v = succ[u] before applying the algorithm and let v = succ[u] after applying the algorithm. If v is stored in a PU in S1 ∪ S2 , then – v is stored in a PU in S1 ∪ S2 . – succ[v ] is not stored in a PU in S1 ∪ S2 , If v is not stored in a PU in S1 ∪ S2 , then v = v. Lemma 2 The algorithm performs 2 routing operations within the whole network and 6 · 2i routing operations within subnetworks of size P/2i , for all 1 ≤ i < log P . Proof: In Step 2 and 5 (2 steps) the communication is global, in Step 3, 4, 7, 8, 11, 12, 14 and 15 (8 steps), the communication is limited to half the network. There are two recursive calls performing the same operations in the halfs. Let T (i) denote the number of operations within subnetworks of size P/2i . The number of recursive calls in these subnetworks is 2i . T (i) = 2. For all i > 0, we have T (i) = 8 · 2i−1 + 2 · 2i = 6 · 2i . Here the first contribution is due to the routing operations from level i − 1, the second contribution to the global operations performed at level i of the recursion. Let Tsolve (P, P , α) denote the time for solving a problem on P PUs, with α list nodes per PU. Nodes, for which we already established that the successors lie in a set of P PUs, for some P ≥ P . In the following analysis we assume that in the algorithm S1 and S2 each consist of half of the PUs. Lemma 3 The time consumption Tsolve (P, P , α) of the algorithm is the sum of the following contributions: Step 1: Tsolve (P/2, P , α), Step 2+5: Tswap (P, α · β) · (2 + 2 · β), Step 3+4: Troute (P/2, α · β · β) · (2 + γ), Step 7+8: Troute (P/2, α · β · γ) · (2 + β), Step 10: Tsolve (P/2, P − P/2, α · β · γ), Step 11+12: Troute (P/2, α · β · γ) · (2 + γ), Step 14+15: Troute (P/2, α · β) · (2 + γ), Here β = (P/2)/P , γ = (P/2)/(P − P/2) and
Minimizing Global Communication in Parallel List Ranking
901
Proof: The result for step 1 is the simplest: we are recursing for P/2 PUs, each PU still holding the original number of participating nodes which are pointing to nodes stored in P different PUs. In step 2, all final nodes pointing to a node in the other subset are asking a question and get an answer in step 5. The questions consist of one integer each. The fraction of nodes pointing to a node in the other subset is (P/2)/P = β. Here, we are using that, according to the first claim in Lemma 1, the number of nodes pointing into another set of PUs does not increase. Of the nodes which receive a question, exactly a fraction P/P = 2·β points to a node in S1 ∪S2 and answer in step 5 with two integers, the other nodes answer with a single integer. These routing operations are the only two performed in the whole set of P PUs, all other operations are limited to the subsets of size P/2. In step 3, a fraction β of the questions is forwarded. The nodes which receive a question are final in their own subset of PUs. Thus, a fraction (P/2)/(P − P/2) = γ of them points to a node in the other subset of PUs and answer with two integers, the other nodes answer with a single integer. In step 7, all nodes which became non-final are asking a question. A node becomes non-final, if and only if it got in reply to its question from step 2 the index of a node stored in its own subset of PUs. Because we know that these indices are arbitrary except for the fact that they do not point to nodes in the other subset, the fraction with this property is (P/2)/(P − P/2) = γ. Of the nodes which receive a question a fraction (P/2)/P = β was non-final in their own subset of PUs and answer with two integers, the other nodes answer with a single integer. In step 10, the recursion is performed for precisely these α · β · γ nodes in each PU which became non-final in step 6. The successors of these nodes are distributed over P − P/2 different PUs. In step 11, the same nodes are asking a question as in step 7. The only thing we know of the nodes which receive a question is that they are final in their own subset of PUs. Thus, a fraction (P/2)/(P − P/2) = γ of them points to a node in the other subset of PUs and answer with two integers, the other nodes answer with a single integer. In step 14, all nodes which initially had their successors within the own subset of PUs ask a question. The fraction of these nodes is β. Of the nodes which receive a question we only know that originally they were final in their own subset. If they were pointing to a node in the other subset, which happens for a fraction γ of them, they answer with two integers, else with one. For any given network with P PUs, substituting the costs of the routing operations, Tsolve (P, P, N/P ) as expressed in Lemma 3 gives an exact estimate for the total communication time when solving a list-ranking problem of size N with the recursive algorithm.
5
Conclusion
We presented a new recursive parallel list-ranking algorithm. Due to its mainly local communication, it performs good on networks on which global communication is strongly penalized. The simplest example of such a network is the linea array, for which it can be shown that the new idea indeed leads to better performance than earlier algorithms.
902
J.F. Sibeyn
References 1. Anderson, R.J., G.L. Miller, ‘Deterministic Parallel List Ranking,’ Algorithmica, 6, pp. 859– 868, 1991. 2. Cole, R., U. Vishkin, ‘Approximate Parallel Scheduling, Part I: the Basic Technique with Applications to Optimal Parallel List Ranking in Logarithmic Time,’ SIAM Journal on Computing, 17(1), pp. 128–142, 1988. 3. Dehne, F., A. Fabri, A. Rau-Chaplin, Scalable Parallel Geometric Algorithms for Coarse Grained Multicomputers,’ Proc. 9th Computational Geometry, pp. 298–307, ACM, 1993. 4. Dehne, F., S.W. Song, ‘Randomized Parallel List Ranking for Distributed Memory Multiprocessors,’ Proc. Asian Computer Science Conference, LNCS 1179, pp. 1–10, 1996. 5. Hayashi, T., K. Nakano, S. Olariu, ‘Efficient List Ranking on the Reconfigurable Mesh with Applications,’ Proc. 7th International Symposium on Algorithms and Computation, LNCS 1178, pp. 326–335, Springer-Verlag, 1996. 6. J´aJ´a, J., An Introduction to Parallel Algorithms, Addison-Wesley Publishing Company, Inc., 1992. 7. Juurlink, B.H.H., H.A.G. Wijshoff, ‘The E-BSP Model: Incorporating General Locality and Unbalanced Communication inti the BSP Model,’ Proc. 2nd International Euro-Par Conference, LNCS 1124, pp. 339–347, Springer-Verlag, 1996. 8. Ranade, A., ‘A Simple Optimal List Ranking Algorithm,’ Proc. of 5th High Performance Computing, Tata McGraw-Hill Publishing Company, 1998. 9. Reid-Miller, M., ‘List Ranking and List Scan on the Cray C-90,’ Journal of Computer and System Sciences, 53(3), pp. 344–356, 1996. 10. Ryu, K.W., J. J´aJ´a, ‘Efficient Algorithms for List Ranking and for Solving Graph Problems on the Hypercube,’ IEEE Transactions on Parallel and Distributed Systems, Vol. 1, No. 1, pp. 83–90, 1990. 11. Sibeyn, J.F., ‘List Ranking on Meshes,’ Acta Informatica, 35, pp. 543–566, 1998. 12. Sibeyn, J.F., ‘Better Trade-offs for Parallel List Ranking,’ Proc. 9th Symposium on Parallel Algorithms and Architectures, pp. 221–230, ACM, 1997. 13. Sibeyn, J.F., ‘One-by-One Cleaning for Practical Parallel List Ranking,’ Algorithmica, 32, pp. 345–363, 2002. 14. Sibeyn, J.F., F. Guillaume, T. Seidel, ‘Practical Parallel List Ranking,’ Journal of Parallel and Distributed Computing, 56, pp. 156–180, 1999. 15. Valiant, L.G., ‘A Bridging Model for Parallel Computation,’ Communications of the ACM, 33(8), pp. 103–111, 1990. 16. Wyllie, J.C., The Complexity of Parallel Computations, PhD Thesis, Computer Science Department, Cornell University, Ithaca, NY, 1979.
Construction of Efficient Communication Sub-structures: Non-approximability Results and Polynomial Sub-cases Christian Laforest LaMI, CNRS, Universit´e d’Evry, 523, Place des Terrasses 91000 Evry, France
[email protected] Abstract. In this paper, we study the following problem. Given a graph G = (V, E) and M ⊆ V , construct a subgraph G∗M = (V ∗ , E ∗ ) of G spanning M , with the minimum number of edges and such that for all u, v ∈ M , the distance between u and v in G and in G∗M is the same. This is what we call an optimal partial spanner of M in G. Such a structure is ”between” a Steiner tree and a spanner and could be a particulary performant and low cost structure connecting members in a network. We prove that the problem cannot be approximated within a constant factor. We then focus on special cases: We require that the partial spanner is a tree satisfying additionnal conditions. For this sub problem, we describe a polynomial algorithm to construct such a tree partial spanner.
1
Introduction
We are interested in this paper by a graph problem that is between two well known graph problems: The Steiner tree and the spanner problems. Both are NP-complete for the majority of their formulations. The first one consists in constructing a tree of minimum weight connecting a subset of selected vertices and have been studied for a long time. Many approximation algorithms have been published (see for example [1,4] for general references on approximation algorithms and [5,16,17] for the Steiner problem). The objective of the second one is, given a graph G and a stretch factor α, to construct a subgraph of minimum weight such that for any pair of vertices their distances in the graph and in the spanner is the same, up to a given factor α. If α is a multiplicative (resp. additive) factor, the corresponding spanners are called multiplicative (resp. additive) spanners (see for example [3,8,11,12,13,15]). We can note that in the traditional Steiner problem, perserving distances between the selected vertices is not taken into account. Several recent works deal with maximal and/or average distances in approximated Steiner trees [6,9, 10,14] (see [7] for the same kind of problem for spanning trees). The weight of obtained substructure is good (compared to optimal) but it fails keeping exact (optimal) distances between the selected vertices. On the other hand, spanners deal with distances. But spanners are only considered as spanning subgraph that must include all the vertices of the graph. H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 903–910, 2003. c Springer-Verlag Berlin Heidelberg 2003
904
C. Laforest
In this paper, we try to generalise this concept and to consider spanners of minimum weight with multiplicative stretch 1 for subsets of selected vertices. This is what we call partial spanners. The study of this problem is motivated by network problems. The goal is to construct a subnetwork to connect members of a video-conference for example. The distance (latency) between participants and the total weight must be low to ensure good Quality of Service with a low cost. The paper is organized as follows. Section 2 gives definitions, describe the problem of partial spanner and gives a general bound. In section 3 we prove that the problem cannot be approximated within a constant. As the general problem is hard, we restrict our attention to particular cases of partial spanners. We require that they must be trees (for simplicity of routing mechanism) and additional properties that allow to implement these communication structures at the application layer of networks. Unfortunately such a partial spanner tree does not always exist. In section Section 4 we propose a polynomial algorithm that constructs this tree if and only if it exists.
2
Definitions, Problem, and General Bounds
In this paper, G = (V, E) is a graph where V is the set of vertices (representing the set of nodes), E the set of edges (representing the set of links). Graphs are undirected and connected (see [2] for undefined terms of graph). For each pair u, v of vertices of V , dG (u, v) is the distance between u and v; this is the number of edges in a shortest path between u and v in G. The weight of G, W (G) is |E|, its number of edges. Definition 1 (Partial spanner). Let G = (V, E) be a graph and M ⊆ V . The sub-graph S = (VS , ES ) of G is a partial spanner of M in G if – M ⊆ VS ⊆ V and ES ⊆ E. – For all u, v ∈ M , dS (u, v) = dG (u, v). Note that when M = V there is only one partial spanner, G itself. Here is now the main problem. Problem 1 (Partial spanner problem). Instance: G = (V, E) a graph and M ⊆ V . Solution: A partial spanner S of M in G. Measure: W (S), the weight of S. Let G = (V, E), M ⊆ V . A partial spanner of M in G of minimum weight is denoted by G∗M and is called an optimal partial spanner of M . The following result gives general tight bounds on the weight of an optimal partial spanner. Theorem 1. Let G = (V, E) and M ⊆ V . The following bounds for W (G∗M ) are tight: 1 DG (M ) = max{dG (u, v) : u, v ∈ M } ≤ W (G∗M ) ≤ dG (u, v) 2 u,v∈M
Construction of Efficient Communication Sub-structures
3
905
Complexity of Problem 1
In this section, we prove that the problem of constructing an optimal partial spanner cannot be approximated from a constant multiplicative factor. For that, we make a reduction with the following problem. Problem 2 (Minimal set cover). Instance: X a set of elements and F a set of subsets of X satisfying: X = F ∈F F . Solution: A subset C ⊆ F covering X: X = F ∈C F . Measure: |C|, the size of C. This problem is not APX (see [1], problem SP4). Theorem 2. Unless P = N P , for any constant α, there is no α approximation algorithm for Problem 1. Proof. We start the proof by describing a special instance of Problem 1, called four levels instance. The graph G = (V, E) of this instance must satisfy the following requirements. Vertex set V constains two special vertices r and Y and two non empty sets V1 and X plus additionnal vertices ensuring connectivity. Let m = |X| + 1. The connections in G are the following: – Vertex r is connected to each vertex of V1 by a path of length m − 1 (using additional vertices). – Each vertex ui ∈ V1 must be connected by a path of length 3 to at least one vertex of X. We note Ci the set of vertices of X connected to ui . To be a four levels instance we must have: Ci = X ui ∈V1
– Each vertex of X is connected (by an edge) to vertex Y . There is no other connection in a four levels instance. The instance of Problem 1 is the constructed graph G and M = X ∪ {r}. It is now easy to see that in polynomial time: – From any instance (X, F) of Problem 2, one can construct an associated four levels instance of Problem 1. – From any four levels instance of Problem 1, one can construct an instance (X, F) of Problem 2. Let us now consider a four levels instance G, M . We can note that for all u, v ∈ X, dG (u, v) = 2 and the only shortest path between u and v is via Y . Any partial spanner of M must contain all the m − 1 edges uY for all u ∈ V1 to ensure the condition on distances. The other edges to add in a partial spanner are just used to connect r to vertices of X. Note also that as for all u ∈ X, dG (r, u) = m + 2, vertex r must be connected to u by a direct path via only one vertex of V1 .
906
C. Laforest
Hence, any partial spanner of M must contain at least a path of length 3 for each u ∈ X (3(m − 1) edges) plus m − 1 edges Y u (u ∈ X) plus (m − 1)s edges of paths between r and ui (ui ∈ V1 ) where s is the number of vertices of V1 used to connect r to vertices of X in the partial spanner. The weight of a partial spanner of M must be at least 3(m − 1) + (m − 1) + (m − 1)s = (m − 1)(s + 4). Moreover, this number of edges is sufficient to construct a partial spanner of M . Suppose first that there exists a partial spanner S of M , whose weight is (m − 1)(s + 4) for a four levels instance of problem 1. Let us consider the s labels Ci of the vertices ui such that there is a path between r and ui in S. The union of these labels gives X (because of the previous remarks). Then this is a solution C of size s for the minimal set cover problem with X and F = {Ci : i = 1, . . . , k}. Let us see now the other direction of the reduction. Let X, F be any instance of Problem 2. Let C be a solution of size p. One can construct a partial spanner of M in the (associated four levels) graph by taking the following edges: The p paths of length m − 1 between r and ui for each i such that Ci ∈ C. For each u ∈ X a path (of length 3) between u and ui labelled Ci such that u ∈ Ci (such a i exists because C covers X). Finally, each edge Y u for all u ∈ M −{r}. The constructed graph is a partial spanner of M , and its weight is exactly (m − 1)(p + 4). Hence, we have shown that for all s, there exists a solution for a four levels instance for problem 1, of weight (m − 1)(s + 4) if and only if there exits a solution for an instance of problem 2, of size s. Now, let α be a constant, α ≥ 1. Suppose that there exists an α approximation algorithm Algo for the partial spanner problem. Let (X, F) be any instance of the minimal set cover problem. Let G = (V, E) and M ⊆ V be the four levels graph associated to the instance (X, F). Let S be a partial spanner constructed by algorithm Algo applied on G and M In addition to Algo we also suppress useless vertices of V1 (i.e. not connected to r by a path of lenght m − 1) and subtrees containing no vertex of M . As Algo is an α approximation algorithm, we have: W (S) ≤ αW (G∗M ). Let A be the set of vertices of V1 in S and a = |A|. As each vertex of A is in a shortest path from r to a vertex of X, by previous discussions we have: (m − 1)(a + 4) ≤ W (S). Now, let B be the set of vertices of V1 in G∗M and b = |B|. We have: W (G∗M ) = (m−1)(b+4). From these notations, we have (m−1)(a+4) ≤ α(m−1)(b+4) and a+4 ≤ α(b+4). Moreover, with the previous discussions, B = C ∗ , that is B is a minimum covering for the instance (X, F). Hence, by applying Algo, one can construct a covering A whose cardinality a satisfies: a + 4 ≤ α(|C ∗ | + 4). That is: a ≤ α|C ∗ | + 4(α − 1) ≤ (5α − 4)|C ∗ |. The covering problem can then be (5α − 4) approximated. This contradicts the fact that it is not an APX problem (see [1]).
4
Partial Spanners That Are Trees
Section 3 shown that the general partial spanner problem is very hard (even to approximate and even for unweighted graphs). However, partial spanners remain very interesting substructures to connect computers. In this section, we
Construction of Efficient Communication Sub-structures
907
will require, in addition to be a partial spanner for M , to have the following properties: 1. The constructed partial spanner must be a tree. 2. The nodes that route information in the tree must be in M . These two requirement are very useful to construct an efficient communication sub-structure because it can be realised at the application layer of network, i.e. without using intermediate routers to route messages or duplicate them, just by using tunnels. Unfortunately, given a graph G and M there is not always such a tree partial spanner. In this section we propose a polynomial time algorithm that constructs such a tree if and only if there exists one in G. To obtain this algorithm we need intermediate results and notions like Ltrees, S-trees and Steiner trees. The relationship between S-tree and tree partial spanners is given in Corollary 1. Definition 2 (Subtree of a graph). Let G = (V, E) be a graph. The tree T = (VT , ET ) is a subtree of G if VT ⊆ V and ET ⊆ E. Moreover T covers M ⊆ V if M ⊆ VT . Definition 3 (Steiner tree of M ). Let G = (V, E) be a graph and M ⊆ V . A tree T0 = (V0 , E0 ) is a Steiner tree of M in G if: – T is a tree covering M in G: M ⊆ V0 ⊆ V and E0 ⊆ E. – T0 is of minimum weight: W (T0 ) = min{W (T ) : T covering M in G} Definition 4 (L-tree). Let G = (V, E) be a graph and M ⊆ V . A tree T = (VT , ET ) (if it exists) is a L-tree of M if: – M ⊆ VT ⊆ V , ET ⊆ E and T has no subtree containing member of M . – For any u, v ∈ M, dT (u, v) = dG (u, v). Lemma 1. Let G = (V, E) be a graph and M ⊆ V . If G contains a L-tree T of M then T is a Steiner tree of M . Proof. Let T0 be a Steiner tree of M . Let r0 ∈ M . Let us apply a depth first search of T0 from r0 . Let r0 , . . . , rk (= r0 ) be the sequence of vertices of M visited during this tour (k ≥ |M |). In order to simplify the presentation, we use the notation i ⊕ j for i + j (mod k). As T is a L-tree, we have: dG (ri , ri⊕1 ) = dT (ri , ri⊕1 ) ≤ dT0 (ri , ri⊕1 ) for all i = 1, . . . , k. Moreover, we have a well known property: k i=0
dT0 (ri , ri⊕1 ) = 2W (T0 )
908
C. Laforest
The sequence of vertices contains several times each vertex of M . This leads to: 2W (T ) ≤
k i=0
dT (ri , ri⊕1 ) ≤
k
dT0 (ri , ri⊕1 ) = 2W (T0 )
i=0
Hence, W (T ) ≤ W (T0 ). As of course we have W (T0 ) ≤ W (T ) we get: W (T0 ) = W (T ). T is a Steiner tree of M . Definition 5 (S-tree). Let G = (V, E) be a graph and M ⊆ V . A tree T = (VT , ET ) (if it exists) is a S-tree of M if: – T is a L-tree of M . – S = {u : dT (u) ≥ 3} ∪ {(dT (u) ≥ 2 and u ∈ M )} ⊆ M The last condition says that what are usually called the ”steiner points” (including duplication nodes and internal members of M ) of T must be members of M only. The following result show the relationship between optimal partial spanners and L-trees. Corollary 1. Let G = (V, E) be a graph and M ⊆ V . Any L-tree (hence any S-tree) T of M is an optimal partial spanner of M . Proof. Definition 4 of a L- tree gives conditions on distances between members of M and Lemma 1 proves that such a covering tree is of minimum weight. In the following, we will use the notion of complete graph of distances of M in graph G. It is a complete graph (each pair of vertices is connected by an edge) with vertex set equal to M . Each edge e = uv (u, v ∈ M ) is weighted by w(e) = dG (u, v). By extension of notation, for any subgraph or set of edges G = (V , E ), we note: W (G ) = w(e) e∈E
Algorithm KS Input: G = (V, E) and M ⊆ V . Construct KM the complete graph of distances of M . Construct in KM a minimum weight spanning tree TP (with Prim algorithm for example). For each edge e = uv of TP construct a shortest path between u and v in G. Return the union of these paths. Lemma 2. Let G = (V, E) be a graph and M ⊆ V . If G contains a S-tree of M then KS applied to G and M returns a Steiner tree of M . Proof. From Lemma 1, a L-tree (hence a S-tree) T0 of M is a Steiner tree of M . Let T1 be the graph returned by KS(G, M ). The other notations are the same than in algorithm KS. We have: W (T0 ) ≤ W (T1 ) ≤ W (TP ). As T0 is a
Construction of Efficient Communication Sub-structures
909
S-tree, it is composed of m − 1 (with m = |M |) shortest paths of G between vertices of M . For each such shortest path, we associate a corresponding edge e of KM . These edges composed a tree T2 spanning KM . Its weight is then greater than W (TP ): W (TP ) ≤ W (T2 ) = W (T0 ). Combining the inequalities we have: W (T0 ) = W (T1 ). T1 is a Steiner tree of M . Theorem 3. Let G = (V, E) be a graph and M ⊆ V . G contains a S-tree of M ⇐⇒ KS(G, M ) returns a S-tree of M . Proof. The ⇐= part is trivial. Let us prove the other implication. Let T be a S-tree of M . As T is S-tree, it is composed of m − 1 (with m = |M |) shortest paths of G between vertices of M . At each of these shortest paths, we associate an edge e of KM . The set of these edges is a tree T spanning KM . As T is a S-tree, we have: W (T ) = W (T ). Like in the description of algorithm KS, let TP be the minimum weight spanning tree (Prim) of KM and T1 be the graph returned by KS(G, M ). As T is a spanning tree of KM and TP is a Prim tree of KM we have: W (TP ) ≤ W (T ). From Lemma 1 we know that T is a Steiner tree of M . From Lemma 2, we know that T1 is also a Steiner tree of M . Hence: W (T ) = W (T1 ). From proof of Lemma 2 we have: W (T1 ) = W (TP ) Combining all these results gives: (1) W (T ) = W (T ) = W (TP ) = W (T1 ) We must prove now that T = TP . In this case T1 is also a S-tree. Suppose that T = TP . For all e = uv ∈ E(TP ), let f (e) be the set of edges of E(T ) of the unique path between u and v in T (if e ∈ E(T ), f (e) = {e}). As T is a S-tree of M in KM , we have: (2) ∀e ∈ E(TP ), w(e) = W (f (e)) Moreover, as TP = T , there exists e ∈ E(TP ) such that |f (e)| > 1 and then: |f (e)| > |M | − 1 (3) e∈E(TP )
We also have:
∀e ∈ E(T ), ∃e ∈ E(TP ),
e ∈ f (e)
(4)
Indeed let e = uv ∈ E(T ) and let Vu (resp. Vv ) be the set of vertices of T in the subtree of T containing u (resp. v) obtained by deleting e from T . As TP spans KM , there exists an edge e ∈ E(TP ) with an extremity in Vu and the other in Vv . It is then clear that e ∈ f (e). From (3) we know that there exists e0 ∈ E(T ) such that e0 is in several f (e) (e ∈ E(TP )). Combining this fact with (1), (2) and (4) we have: w(e) = W (f (e)) ≥ w(e0 ) + w(e) > W (T ) W (T ) = e∈E(TP )
e∈E(TP )
that is a contradiction and T = TP .
e∈E(T )
910
C. Laforest
Knowing either a given G contains a S-tree for M or not is then polynomial; just apply KS and control (in polynomial time) that the returned graph is or is not a S-tree of M . Acknowledgements. Work supported by the French CNRS project AcTAM.
References 1. G. Ausiello, P. Crescenzi, G. Gambosi, V. Kann, A. Marchetti Spaccamela, and M. Protasi. Complexity and approximation. Springer, 1999. 2. J.A. Bondy and U.S.R Murty. Graph theory with applications. North-Holland, 1976. 3. M. Elkin and D. Peleg. The harness of approximating spanner problems. In Springer Verlag, editor, STACS, volume 1770 of Lecture notes in computer science, pages 370–381, 2000. 4. D. Hochbaum. Approximation algorithms for NP-hard problems. PWS publishing compagny, 1997. 5. S. Hougardy and H. Pr¨ omel. A 1.598 approximation algorithm for the steiner problem in graphs. In Proc. 10th Ann. ACM-SIAM Symp. on Discrete Algorithms, ACM-SIAM, pages 448–453, 1999. 6. A. Irlande, J.-C. K¨ onig, and C. Laforest. Construction of low-cost and low-diameter steiner trees for multipoint groups. In G. Proietti M. Flammini, E. Nardelli and P. Spirakis, editors, Procedings of Sirocco 2000, pages 197–210. Carleton Scientific, 2000. 7. S. Khuller, B. Raghavachari, and N. Young. Balancing minimum spanning trees and shortest-path trees. Algorithmica, (14):305–321, 1995. 8. G. Kortsarz and D. Peleg. Generating sparce 2-spanners. J. algorithms, (17):22– 236, 1994. 9. G. Kortsarz and D. Peleg. Approximating shallow-light trees. In Eighth ACMSIAM Symp. on Discrete Algorithms, pages 103–110, 1997. 10. C. Laforest. A good balance between weight and distance for multipoint trees. In International Conference On Priniciples Of Distributed Systems (OPODIS’02), pages 195–204, 2002. 11. C. Laforest, A. Liestman, D. Peleg, T. Shermer, and D. Sotteau. Edge-disjoint spanners of complete graphs and complete digraph s. Discrete mathematics, 203:133–159, 1999. 12. C. Laforest, A. Liestman, T. Shermer, and D. Sotteau. Edge-disjoint spanners of complete bipartite graphs. Discrete mathematics, 234:65–76, 2001. 13. A.L. Liestman and T. Shermer. Grid spanners. Networks, (23):123–133, 1993. 14. M. Marathe, R. Ravi, R. Sundaram, S. Ravi, Rosenkrantz, and H. D. Hunt III. Bicriteria network design problems. Journal of algorithms, 28:142–171, 1998. 15. D. Peleg and Sch¨ offer A. Graph spanners. J. Graph Theory, (13):99–116, 1989. 16. G. Robins and Zelikovsky. Improved steiner tree approximation in graphs. In SODA 2000. 17. H. Takahashi and A. Matsuyama. An approximate solution for the steiner problem in graphs. Math. Jap., (24):573–577, 1980.
c-Perfect Hashing Schemes for Binary Trees, with Applications to Parallel Memories (Extended Abstract) Gennaro Cordasco1 , Alberto Negro1 , Vittorio Scarano1 , and Arnold L. Rosenberg2 1
Dipartimento di Informatica ed Applicazioni “R.M. Capocelli”, Universit` a di Salerno, 84081, Baronissi (SA) – Italy {alberto,vitsca,cordasco}@dia.unisa.it 2 Dept. of Computer Science, University of Massachusetts Amherst Amherst, MA 01003, USA
[email protected] Abstract. We study the problem of mapping tree-structured data to an ensemble of parallel memory modules. We are given a “conflict tolerance” c, and we seek the smallest ensemble that will allow us to store any nvertex rooted binary tree with no more than c tree-vertices stored on the same module. Our attack on this problem abstracts it to a search for the smallest c-perfect universal graph for complete binary trees. We construct such a graph which witnesses that only O c(1−1/c) · 2(n+1)/(c+1) memory modules are needed to obtain the required bound on conflicts, and we prove that Ω 2(n+1)/(c+1) memory modules are necessary. These bounds are tight to within constant factors when c is fixed—as it is with the motivating application.
1
Introduction
Motivation. This paper studies the efficient mapping of data structures onto a parallel memory system (PMS, for short) which is composed of several modules that can be accessed simultaneously (by the processors of, say, a multiprocessor system). Mapping a data structure onto a PMS poses a challenge to an algorithm designer, because such systems typically are single-ported: they queue up simultaneous accesses to the same memory module, thereby incurring delay. The effective use of a PMS therefore demands efficient mapping strategies for the data structures that one wishes to access in parallel—strategies that minimize, for each memory access, the delay incurred by this queuing. Obviously, different mapping strategies are needed for different data structures, as well as for different ways of accessing the same data structure. As a simple example of the second point, one would map the vertices of a complete binary tree quite differently when optimizing access to levels of the tree than when optimizing access to root-to-leaf paths. H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 911–916, 2003. c Springer-Verlag Berlin Heidelberg 2003
912
G. Cordasco et al.
The preceding considerations give rise to the problem studied here. Given a data structure represented by a graph, and given the kinds of subgraphs one wants easy access to (called templates), our goal is to design a memory-mapping strategy for the items of a data structure that minimizes the number of simultaneous requests to the same memory module, over all instances of the considered templates. Our Results. This paper presents the first strategy for mapping a binary-tree data structure onto a parallel memory in such a way that any rooted subtree can be accessed with a bounded number of conflicts. Our results are achieved by proving a (more general) result, of independent interest, about the sizes of graphs that are “almost” perfect universal for binary trees, in the sense of [5]. Related Work. Research in this field originated with strategies for mapping two-dimensional arrays into parallel memories. Several schemes have been proposed [3,4,8,10] in order to offer conflict-free access to several templates, including rows, columns, diagonals and submatrices. While the strategies in these sources provide conflict-free mappings, such was not their primary goal. The strategies were actually designed to accommodate as many templates as possible. Strategies for mapping tree structures considered conflict-free access for one elementary template—either complete subtrees or root-to-leaf paths or levels [6,7] or combinations thereof [2], but the only study that even approaches the universality of our result—i.e., access to any subtree—is the C-template (“C” for “composite”) of [1], whose instances are combinations of different numbers of distinct elementary templates. The mapping strategy presented for the Ctemplates’ instances of size K, with M memory modules, achieves O(K/M + c) conflicts. Background. For any binary tree T : Size (T ) is its number of vertices; Size (T, i) is its number of level-i vertices; Hgt (T ) is its height (= number of levels). For any positive integer c, a c-contraction of a binary tree T is a graph G that is obtained from T via the following steps. 1. Rename T as G(0) . Set k = 0. 2. Pick a set S of ≤ c vertices of G(k) that were vertices of T . Remove these vertices from G(k) , and replace them by a single vertex that represents the set S. Replace all of the edges of G(k) that were incident to the removed vertices by edges that are incident to S. The graph so obtained is G(k+1) . 3. Iterate step 2 some number of times. A graph Gn = (Vn , En ) is c-perfect-universal for the family Tn of n-vertex binary trees if every c-contraction of an n-vertex binary tree is a labeled-subgraph of Gn . By this we mean the following. Given any c-contraction G(a) = (V, E) of an n-vertex binary tree T , the fact that G(a) is a subgraph of Gn is observable via a mapping f : V → Vn for which each v ∈ V is a subset of f (V ). The simplest 1-perfect-universal graph for the family Tn is the height-n complete binary tree, T n , which is defined by the property of having all root-to-leaf paths of common length Hgt (T n ) = n. The perfect-universality of T n is witnessed by the identity map f of the vertices of any n-vertex binary tree to the
c-Perfect Hashing Schemes for Binary Trees
913
vertices of T n . Of course, T n is a rather inefficient perfect-universal graph for Tn , since it has 2n − 1 vertices, whereas each tree in Tn has only n vertices. It is natural to ponder how much smaller a 1-perfect-universal graph for Tn can be. The size—in number of vertices—of the smallest such graph is called the perfection number of Tn and is denoted Perf(Tn ). In [5], Perf(Tn ) is determined exactly, via coincident lower and upper bounds. Theorem 1 ([5]) Perf(Tn ) = (3 − (n mod 2)) 2(n−1)/2 − 1. In this paper, we generalize the study of storage mappings for trees in [5] by allowing boundedly many collisions in storage mappings. We thus relax the “one to one” demands of perfect hashing to “boundedly many to one.”
2
Close Bounds on Perfc (Tn )
Our study generalizes Theorem 1 to c-perfect-universality, by defining Perfc (Tn ), for any positive integer c, to be the size of the smallest c-perfect-universal graph for Tn ; in particular, Perf1 (Tn ) = Perf(Tn ). We first derive our upper bound on Perfc (Tn ) by explicitly constructing a graph Gc that is c-perfect-universal for Tn . Theorem 2 For all integers n and c > 1, Perfc (Tn ) < 2 + 2c1−1/c 2(n+1)/(c+1) + O(n).
(1)
We construct our c-perfect-universal graph Gc via an algorithm Ac that colors the vertices of T n in such a way that the natural vertex-label-preserving embedding of any n-vertex tree T into T n (which witnesses T ’s being a subgraph of T n ) never uses more than c vertices of any given color as homes for T ’s vertices. When we identify each like-colored set of vertices of T n —i.e., contract each set to a single vertex in the obvious way—we obtain the c-perfect-universal graph Gc , whose size is clearly an upper bound on Perfc (Tn ). Algorithm Ac proceeds in a left-to-right pass along each level of T n in turn, assigning a unique set of colors, Ci , to the vertices of each level i, in a roundrobin fashion. Ac thereby distributes the 2i level-i vertices of T n equally among the |Ci | level-i vertices of Gc . Clearly, thus, Size (Gc ) = i |Ci |. The remainder of the section is devoted to estimating how big the sets Ci must be in order for Gc to be c-perfect-universal for Tn . Auxiliary results. We begin by identifying some special subtrees of T n . Let m and i be integers such that1 log m ≤ i ≤ n, and let x be a binary string of length i − log m . Consider the subtree T(i,m) (x) of T n constructed as follows. 1. Generate a length-(i − log m ) path from the root of T n to vertex x. 1
All logarithms are to the base 2.
914
G. Cordasco et al.
2. Generate the smallest complete subtree rooted at x that has at least m leaves; easily, this subtree—call it T (x)—has height log m . 3. Finally, prune the tree so constructed, removing all vertices and edges other than those needed to incorporate the leftmost m leaves. Easily, every tree T(i,m) has exactly m leaves, all of length i. Lemma 1. The trees T(i,m) are the smallest rooted subtrees of T n that have m leaves at level i. (Sketch) For any level j of any binary tree T , we have Size (T, j − 1) ≥ Proof. 1 2 Size (T, j) . One verifies easily from our construction that the trees T(i,m) achieve this bound with equality. Lemma 2. For all i and m,
2m + i − log m − 1 ≤ Size T(i,m) ≤ 2m + i − 1.
(2)
Proof. (Sketch) We merely sum the sizes of the complete subtree on T (x), plus the size of the various paths needed to “capture” them. Thus we have Size T(i,m) = (i − hpm−1 + 1) + 2m − pm (3) where pm is the number of complete subtree on T (x) and hpm−1 is the height of the smallest complete subtree on T (x). We obtain the bound of the lemma from the exact, but nonperspicuous, expression (3), from the facts that there is at least one 1 in the binary representation of m and that pm ≥ 1. Let us now number the vertices at each level of T n from left to right and say that the distance between any two such vertices is the magnitude of the difference of their numbers. Lemma 3. For any integers 0 ≤ i ≤ n and 0 ≤ δi < i, the size of the smallest δi rooted subtree of T n that has m leaves at leveli of T n , each at distance ≥ 2 from the others, is no smaller than Size T(i,m) + (m − 1)δi . Proof. If two vertices on level i are distance ≥ 2δi apart, then their least common ancestor in T n must be at some level ≤ i − (δi + 1). It follows that the smallest rooted subtree T of T n that satisfies the premises of the lemma must consist of a copy of some T(i−δi ,m) , with m leaves at level i − δi , plus m vertex-disjoint paths (like “tentacles”) from that level down through each of the δi levels i − δi + 1, i − δi + 2, . . . , i of T n . Equation (3) therefore yields: Size (T ) ≥ mδi + Size T(i−δi ,m) = (m − 1)δi + Size T(i,m) . The upper bound proof. We propose, in the next two lemmas, two coloring schemes for Ac , each inefficient on its own (in the sense that neither yields an upper bound on Perfc (Tn ) that comes close to matching our lower bound), but which combine to yield an efficient coloring scheme. We leave to the reader the simple proof of the following Lemma.
c-Perfect Hashing Schemes for Binary Trees
915
Lemma 4. Let T n be colored by algorithm Ac using 2δi colors at each level i. If each def 2δi ≥ κi = 2i /c , then any rooted n-vertex subtree of T n engenders at most c collisions. Lemma 5. Let T n be colored by algorithm Ac using 2δi colors at each level i. If each2
n + log(c + 1) − i 1 def , (4) exp2 2δi ≥ λi = 4 c then any rooted n-vertex subtree of T n engenders at most c collisions. Proof. We consider a shallowest tree T that engenders c+1 collisions at level i of T n . Easily, the offending tree T has c + 1 leaves from level i of T n . By the design of Algorithm Ac , these leaves must be at distance 2δi from one another. We can, therefore, combine Lemma 3, the lower bound of (2) (both with m = c + 1), and (4) to bound from below the size of the offending tree T . Size (T ) ≥ Size T(i,c+1) + cδi
n − 2c + log(c + 1) − i ≥ 2(c + 1) + i − log(c + 1) − 1 + c ≥ n + 1. c Since we care only about (≤ n)-vertex subtrees of T n , the lemma follows. A bit of analysis verifies that the respective strengths and weaknesses of the coloring schemes of Lemmas 4 and 5 are mutually complementary. This complementarity suggests the ploy of using the κi -scheme to color the “top” of T n and the λi -scheme to color the “bottom.” A natural place to divide the κi ≈ λi . Using this “top” of T n from the “bottom” would be at a level i where
n−c+1 def intuition, we choose level i = + log c to be the first level of c+1 the “bottom” of T n . (Since c ≤ n, trivial calculations show that n > i .) Using our hybrid coloring scheme, then, we end up with a c-perfect-universal graph i −1 n−1 κj + λk . Evaluating the two summations Gc such that Size (Gc ) = j=0
k=i
in turn, we find the following, under the assumption that c > 1 (since the case c = 1 is dealt with definitively in [5]; see Theorem 1). i −1
j=0
κj =
i −1
j
2 /c ≤
j=0
1 ≤ exp2 c 2
i −1
j=0
(2j /c + 1) =
1 i 1 2 + i − c c
n−c+1 + log c + O(n) < 2 · 2(n+1)/(c+1) + O(n)(5) c+1
To enhance the legibility of powers of 2 with complicated exponents, we often write exp2(X) for 2X .
916
G. Cordasco et al.
n−1 (c + 1)1/c (n−k)/c n + log(c + 1) − k −2 < · λk = exp2 2 c 2 k=i k=i k=i i +c−1 n − c + 1 log c n (n−k)/c − + < 2 2 ≤ 2c · exp2 c c(c + 1) c n−1
n−1
k=i
< 2 c1−1/c · 2(n+1)/(c+1) .
(6)
The bounds (5, 6) yield the claimed upper bound (1) on Perfc (Tn ). Because of space limitations, we defer the proof of the following lower bound to the complete version of this paper. Theorem 3 For all c > 1 and all n, Perfc (Tn ) > exp2
n + 1 11 − c+1 3
.
Acknowledgment. The research of A.L. Rosenberg was supported in part by US NSF Grant CCR-00-73401.
References 1. V. Auletta, S. Das, A. De Vivo, M.C. Pinotti, V. Scarano, “Optimal tree access by elementary and composite templates in parallel memory systems”. IEEE Trans. Parallel and Distr. Systs., 13, 2002. 2. V. Auletta, A. De Vivo, V. Scarano, “Multiple Template Access of Trees in Parallel Memory Systems”. J. Parallel and Distributed Computing 49, 1998, 22–39. 3. P.Budnik, D.J. Kuck. “The organization and use of parallel memories”. IEEE Trans Comput., C-20, 1971, 1566–1569. 4. C.J.Colbourn, K.Heinrich. “Conflict-free access to parallel memories”. J. Parallel and Distributed Computing, 14, 1992, 193–200. 5. F.R.K. Chung, A.L. Rosenberg, L. Snyder. “Perfect storage representations for families of data structures.” SIAM J. Algebr. Discr. Meth., 4, 1983, 548–565. 6. R. Creutzburg, L. Andrews, “Recent results on the parallel access to tree-like data structures – the isotropic approach”, Proc. Intl. Conf. on Parallel Processing, 1, 1991, pp. 369–372. 7. S.K. Das, F. Sarkar, “Conflict-free data access of arrays and trees in parallel memory systems”, Proc. 6th IEEE Symp. on Parallel and Distributed Processing, 1994, pp. 377–383. 8. D.H.Lawrie. “Access and alignment of data in an array processor”. IEEE Trans. on Computers, C-24, 1975, 1145–1155. 9. R.J. Lipton, A.L. Rosenberg, A.C. Yao, “External hashing schemes for collections of data structures.” J. ACM, 27, 1980, 81–95. 10. K.Kim, V.K.Prasanna. “Latin Squares for parallel array access”. IEEE Trans. Parallel and Distributed Systems, 4, 1993, 361–370. 11. A.L. Rosenberg and L.J. Stockmeyer, “Hashing schemes for extendible arrays.” J. ACM, 24, 1977, 199–221. 12. A.L. Rosenberg, “On storing ragged arrays by hashing.” Math. Syst. Th., 10, 1976/77, 193–210.
A Model of Pipelined Mutual Exclusion on Cache-Coherent Multiprocessors Masaru Takesue Dept. Electronics and Information Engr., Hosei University, Tokyo 184-8584 Japan
[email protected] Abstract. This paper proposes a model of pipelined mutual exclusion on cache-coherent multiprocessors that allows processors to concurrently access different memory locations of shared data in a pipelined manner. The model converts the effective unit of mutual exclusion from shared data to one memory location, by producing the total order of requests for each location while preserving the total order for shared data in the order for location. Two implementations of the model are evaluated. Keywords: Models, mutual exclusion, pipelining.
1
Introduction
In cache-coherent multiprocessors, communication is performed through synchronization. Mutual exclusion is the synchronization operation for ensuring write atomicity on a set D of shared data. Since it serializes the accesses to D, its latency with a naive scheme such as the test&set is surprisingly great when a large number of processors are competing for D, leading to a crucial obstacle to fine-grain parallel computation. The latency is greatly reduced, especially when the contention rate on D is large, by pipelining mutual exclusion so that processors can concurrently access different memory locations for D in a pipelined manner [1]; the pipelining is implemented with a software-hardware hybrid scheme. The contribution of this paper is to formalize a model of the pipelined mutual exclusion. The key idea behind the model is to convert the effective unit of mutual exclusion from D to one memory location, ensuring mutual exclusion on D. For this end, the model produces a total order of requests for each memory location of D, preserving the total order of requests for D in the total order for each location. For simplicity of discussion, we restrict a memory location to a memory-block location in the paper, though it can be a memory-word location [1]. Related work: Of the previous work for mutual exclusion [2]-[8], software queue-based algorithms are most promising in heavily contentious cases [3]-[5], since the produced total order of requesters allows them to spin locally for the access right on D. The QOLB [6] is one of the hardware queue-based schemes [8,6], and outperforms over the software counterparts [7]. However, none of the previous work addresses location-based pipelined mutual exclusion. H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 917–922, 2003. c Springer-Verlag Berlin Heidelberg 2003
918
M. Takesue
A recently proposed scheme [9] allows multiple threads to concurrently access falsely shared locations of D by speculatively removing unnecessary locking; so this scheme addresses a very restricted case in our model. As for the memory consistency models, the data-race-free-1 model [10] can release each memory block before the release lock on D; this leads to the pipelining but needs a complex hardware scheme to match the arrival of the block with the access on it. The release consistency model [11] can overlap the writes in the critical section (CS) for a D and the reads in the CS for another D in one processor. Notice that this is not the pipelining among the CSs for a same D in separate processors as addressed in this paper. The rest of the paper is organized as follows: Section 2 introduces the model. Sections 3 evaluates a software and a software-hardware hybrid implementations of the model, comparing with the most efficient non-pipelined software [3] and hardware [6] queue-based algorithms. Section 4 concludes the paper.
2
The Model
Let a set D of shared data be allocated to n memory blocks, B0 to Bn−1 . Each processor is assumed to selectively access some memory blocks of D that are required in the processor. Let TOD denote the total order of processors requesting for D, and TOLi be the total order of processors requesting for block Bi ; the set {TOLi |0 ≤ i ≤ n − 1} is denoted by {TOL}. To preserve the TOD in each TOL, we introduce a tree data structure, called order preserving tree (OPT). The logical structure of an OPT node is shown in Fig. 1-(a). The tail pointer, tail, has the identifier of the tail processor in a total order (described shortly). An example of the OPT for a set D of 4 memory blocks is shown in Fig. 1-(b). The memory blocks for D are depicted by the dashed boxes since those are not included in the OPT but associated with the leaf nodes. struct OPTnode { enum boolean nonleaf; struct { tail; /∗ of a TOC/TOL ∗/ struct OPTnode ∗ptr; } child[C]; }; (a) The logical structure of an OPT node
N0 Z ~ Z N1 • • N2 • • BBN BBN •
B0
B1
•
B2
B3
(TOD) ({TOC}) ({TOL})
(b) An example of the OPT
Fig. 1. The logical structure of an OPT node, and an OPT for 4-block shared data
In reality, we map the tail pointers onto an array so that those in one node are packed in one memory block, and reorganize the ptr’s and arity of the tree as another data structure. Then the tail in a node indicates the tail processor in the total order, TOC, for its child node (memory block) if it is a nonleaf, or in the TOL for the associated memory block otherwise. For instance, in Fig. 1-(b), nonleaf node N0 has the tails of the TOCs for the child nodes N1 and N2 , and leaf node N1 has the tails of the TOLs for the associated blocks B0 and B1 .
A Model of Pipelined Mutual Exclusion on Cache-Coherent Multiprocessors
919
With the OPT, the CS entry routine, enterCS, for D produces the {TOL} from the TOD, preserving the TOD in each TOC and especially in each TOL. A processor accesses a memory block in the blockwise critical section, BwCS, for the block. Before and after the BwCS, we need an enterBwCS and an exitBwCS for the block to acquire and release the access right on the block. The enterCS is shown in Fig. 2. The root points to the root node of the OPT. The SCANQ is a queue for scanning the tree in breadth-first order. A requesting processor I is put into the TOD by the serialize. The for-loop corresponds to the BwCS for memory block myP allocated to an OPT node. The enterBwCS and exitBwCS acquire and release the access right on block myP. In the BwCS, the serializeChild puts identifier I into the TOC or TOL for block childP (i.e., for the child node or the associated block of D) if it is required in the processor. enterCS(root,I) { serialize(root,I); /∗ TOD ∗/ enqSCANQ(root); while (nonemptySCANQ()) { myP = deqSCANQ(); enterBwCS(myP); for (i=0; ichild[i].ptr; childT = myP->child[i].tail; serializeChild(childP,childT,I); /∗ TOC/TOL ∗/ updateTail(myP->child[i].tail,I); if (myP->nonleaf) enqSCANQ(childP); } exitBwCS(myP); } } Fig. 2. The enterCS routine
The CS routine, CS, for D consists of multiple tuples each of enterBwCS, BwCS, and exitBwCS. The number of tuples is equal to the number of memory blocks required in the processor, and the elements of one tuple can interleave with those of another tuple. No exit-CS routine is necessary because of the blockwise release of the access right in the exitBwCS. Now we define the model M of pipelined mutual exclusion, and prove that the pipelining is possible. Definition 1. Model M consists of the TOD, {TOC}, {TOL}, OPT, enterCS, and CS that comprises multiple tuples each of enterBwCS, BwCS, and exitBwCS. Theorem 1. Model M ensures and pipelines mutual exclusion on a set D of shared data. Proof. Assume that processor I is a predecessor of processor J in the TOD and j) of D. If processor I is a predecessor of both access blocks Bi and Bj (i =
920
M. Takesue
processor J in the total order for an OPT node, the enterCS puts the former before the latter in the TOC for its child node or in the TOL for the associated memory block, since the former receives the OPT node earlier than the latter. So by induction, processor I is put before processor J, especially in the TOLs for blocks Bi and Bj . Thus processor I certainly accesses those blocks earlier than processor J. This ensures mutual exclusion on D. As for the pipelining, since a processor releases the access right on a memory block basis, the successors in the TOL for the block can access it on receiving the access right. Theorem 2. Model M works with the relaxed memory consistency models. Proof. A program correctly synchronized under the sequential consistency model produces the same result as with any of the relaxed memory models (such as the release consistency model) [11,10]. In M, the pair of an enterBwCS(B) and an exitBwCS(B) always synchronizes the memory operations on each block B.
3
Evaluation
This section describes our simulation environment, and shows the potential performance and overhead of the pipelined mutual exclusion. 3.1
Simulator and Evaluated Algorithms
We evaluate with an RTL (register transfer level) simulator of our system with up to 256 processing nodes (PNs). The system parameters are listed in Table 1. Four mutual exclusion algorithms are evaluated; the pipelined algorithms, PpS and PpH, based on model M with software and hardware queues, respectively, and the non-pipelined algorithms, NpS and NpH, with software and hardware queues. The software queue is implemented on an array by using the Fetch&Inc [3]. The hardware queue is organized in the requesting caches by specific instructions and an extended cache protocol; see [1] for the PpH, and [6] for the NpH. Table 1. System parameters Item processor (PE) cache network (NW) cycle time [clock]
Description extended DLX [12] unified, blocking, 32-byte line, set-associative hypercube, wormhole routing, 1 byte/hop·cycle PE/cache/NW-switch: 1, memory unit (in each PN): 10
For the NpS and NpH, we acquire the locks on all memory blocks of D in the CS (due to no OPT), while we access the blocks and release the locks on a block basis outside the CS, because this may lead to a sort of the pipelining, though the degree of overlap in the pipelining, i.e., the number of processors that are concurrently in the individual CSs for D, may be small.
A Model of Pipelined Mutual Exclusion on Cache-Coherent Multiprocessors
3.2
921
Results
The speedup of the PpS and NpS over the NpH is shown in the left part of Fig. 3; the degrees of overlap obtained with the NpH is alway one (i.e., no overlapping). The PpS improves the performance of NpS by a factor of about 1.0 to 2.23, leading to a speedup of about 0.25 to 1.38 over the NpH. Surprisingly, the speedup reaches up to over 1.0 with no hardware support except of the Fetch&Inc. The PpH’s speedup over the NpH is about 1.0 to 88.2 as shown in the right part of Fig. 3. As the sizes of data and system increase, a greater speedup is achieved owing to a larger degree of overlap up to about 91.2. Speedup (PpH) 80
the figure on top of the bar: degree of overlap with PpH
91.2
Speedup (PpS and NpS) : NpS : PpS x/y: degree of overlap with NpS/PpS
1.5 1.0 0.5
5.2/ 157.6 4.7/ 4.8/ 5.4/ 5.0/ 5.3/ 243.6 57.5 62.8 14.4 12.7 2.8/ 2.8/ 2.7/ 1.0/1.8/ 4.3 1.0/1.7/4.2 1.0/1.8/ 3.9 1.9 1.9 2.0 1.0 1.0 1.0
data size [block] 1 2 4 16 64 system size [PN] 16
1 2 4 16 64
1 2 4 16 64
64
256
60
53.6
40 20
23.5 13.1 6.6 14.2 6.5 4.3 2.9 2.2 1.4 1.0 1.0
1
23.1 6.3 3.7 1.4 1.0
4 64 1 4 64 1 4 64 2 16 256 2 16 256 2 16 256 16
64
256
Fig. 3. Speedup of the PpS and NpS (left), and PpH (right) over the NpH
The overhead of PpS and PpH, i.e., their performance relative to the NpH’s in light-load cases, is shown in Fig. 4, where np (= 1, 2, or 4) number of PNs are contesting for D on the 256-PN system. The relative execution time of PpS and PpH is about 1.8 to 5.7, and about 0.3 to 1.0. Normalized Execution Time 6 5 4 3 2 1
: PpS : PpH
data size [block] 1 2 4 16 64 1 2 4 16 64 number of contesting PNs (np ) 2 1
1 2 4 16 64 4
Fig. 4. Performance of the PpH and PpS relative to the NpH’s in light-load cases
922
4
M. Takesue
Conclusions
Our model of pipelined mutual exclusion converts effectively the unit of mutual exclusion from shared data to one memory location, ensuring mutual exclusion on shared data, so that processors can concurrently access different locations of shared data. The model is orthogonal to memory consistency models since it correctly synchronizes memory operations on a location basis. A remarkable result of evaluation is that the software implementation (omitted for space, and will be presented in a future paper) of the model outperforms over a hybrid non-pipelined algorithm when the contention rate is high; then the hybrid implementation [1] is much more effective than the software one. A scheme for pipelining conditional synchronization [2] will be reported in another paper [13]. For a smaller communication latency, we are designing a scheme for dynamically clustering the requests for the pipelined synchronization.
References 1. Takesue, M.: Pipelined Mutual Exclusion on Large-Scale Cache-Coherent Multiprocessors. Proc. the 14th IASTED Int. Conf. on Parallel and Distributed Computing and Systems (2002) 745–754 2. Dinning, A.: A Survey of Synchronization Methods for Parallel Computers. IEEE Computer 22 (1989) 3–43 3. Anderson, T. E.: The Performance of Spin Lock Alternatives for Shared-Memory Multiprocessors. IEEE Trans. on Parallel and Distributed Systems, 1 (1990) 6–16 4. Graunke, G., Thakkar, S.: Synchronization Algorithms for Shared-Memory Multiprocessors. IEEE Computer 23 (1990) 60–69 5. Grummey, J. M. M., Scott, M. L.: Synchronization Without Contention. Proc. 4th Int. Conf. on Architectural Support for Programming Languages and Operating Systems (1991) 269–278 6. Goodman, J. R., Vernon, M. K., Woest, P. J.: Efficient Synchronization Primitives for Large-Scale Cache-Coherent Multiprocessors. Proc. 3rd Int. Conf. on Architectural Support for Programming Languages and Operating Systems (1989) 64–75 7. K¨ agi, A., Burger, D., Goodman, J. R.: Efficient Synchronization: Let Them Eat QOLB. Proc. 24th Int. Symp. on Computer Architecture (1997) 170–180 8. Lee, J., Ramachandran, U.: Synchronization with Multiprocessor Caches. Proc. 17th Int. Symp. on Computer Architecture (1990) 27–37 9. Rajwar, R., Goodman, J. R.: Speculative Lock Elision: Enabling Highly Concurrent Multithreaded Execution. Proc. 34th Int. Symp. on Microarchitecture (2001) 294– 305 10. Adve, S. V., Hill, M. D.: A Unified Formalization of Four Shared-Memory Models. IEEE Trans. on Parallel and Distributed Systems 4 (1993) 613–624 11. Gharachorloo, K., Lenoski, D., Laudon, J., Gibbons, P., Gupta, A., Hennessy, J.: Memory Consistency and Event Ordering in Scalable Shared-Memory Multiprocessors. Proc. 17th Int. Symp. on Computer Architecture (1990) 15–26 12. Hennessy, J. L., Patterson, D. A.: Computer Architecture: A Quantitative Approach. Morgan Kaufmann Publishers (1990) 13. Takesue, M.: Pipelined Conditional Synchronization on Large-Scale CacheCoherent Multiprocessors. Proc. the 16th ISCA Int. Conf. on Parallel and Distributed Computing Systems (2003) accepted for presentation
Efficient Parallel Multiplication Algorithm for Large Integers Viktor Bunimov and Manfred Schimmler Institute for Computer Engineering and Communication Networks, Technical University of Braunschweig, Germany
[email protected],
[email protected] Abstract. A new algorithm for calculating A*B is presented. It is based on precomputation of a small number of values that allow for an efficient parallel implementation of multiplication. The algorithm requires O(logn) time and 2 2 O(n /logn ) area with small constant factors which makes it feasible for practical implementations. Keywords: Computer arithmetic, large number arithmetic, redundant numbers, carry save addition, parallel multiplication, integer multiplication.
1
Introduction
Integer multiplication is one of the most important operations in computer arithmetric. There are many algorithms and architectures for integer multiplication. In classical algorithm for integer multiplication the bits of one operand are multiplied with shifted copies of the second operand and the results are added [5], [13]. If the operands are very long (the number of bits n>128), the algorithm requires too many additions (n). 2QHDGGLWLRQKDVWLPHFRPSOH[LW\RI logn) [13] and area (hardware) complexity of Q FRQVHTXHQWO\WKHWLPHFRPSOH[LW\RIWKLVPXOWLSOLFDWLRQDOJRULWKPLV2 n*logn) and the AT-complexity (area-time complexity) is O(n2logn). There are a many other algorithms for integer multiplication: Karatsuba multiplication [4], Toom multiplication [11], FFT multiplication, Schönhage-Strassen multiplication [8] and others. All these are sequential algorithms with good AT-complexity, but, compared to parallel methods which can be implemented in hardware, the time complexity is log3 not good: The Karatsuba multiplication has time complexity of O(n ). Toom 1/2 multiplication has time complexity of O(n*log n). Winograd [14], Cook [5], Zuras [15] and Knuth [5] developed versions of Toom multiplication with different improvements. Another interesting development is multiplication using Fast Fourier Transform (FFT) [2],[3]. One of the best algorithms is the Schönhage-Strassen multiplication with the time complexity of O(nlognloglogn). But the time complexity of every sequential algorithm is asymptotically greater than O(logn). The fastest parallel algorithms for integer multiplication has time complexity O(logn) [9], [12], [13] which has been proven to be optimal. One of the possibilities to speed up the multiplication is the parallelisation of the suboperations of multiplication. There are a many parallel algorithms for integer multiplication: Wallace-tree multipliers [1], [10], [12] use a parallelisation of the H. Kosch, L. Böszörményi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 923–928, 2003. © Springer-Verlag Berlin Heidelberg 2003
924
V. Bunimov and M. Schimmler
classical algorithm for multiplication with redundant numbers for intermediate representation. These provide an optimal time complexity of O(log n), but the area 2 2 complexity is O(n ); consequently, the AT-complexity is O(n log n). The parallel algorithm for the integer multiplication in [9] has an optimal time complexity of 2 O(log n) too, and the AT-complexity is reduced to O(n ). The best parallel algorithm for parallel multiplication is the parallelisation of the Schönhage-Strassen multiplication [8],[13] with time complexity of O(log n) and area complexity of 2 O(nlognloglogn), thus the AT-complexity is O(nlog nloglogn). But this algorithm is not used in practical applications, because the constants (which are hidden in the Onotation) are too large. In this paper we present a new parallel algorithm for integer multiplication with 2 optimal time complexity of O(log n) and AT-complexity of O(n /log n). In contrast to earlier time- and area-optimal algorithms, our method is suited for real world implementations because of its small constant. This paper is organised as follows. In section 2, the idea of the new algorithm is presented. Section 3 gives a short introduction into fast adder technology using redundant number representations for intermediate results. The final version of our new algorithm exploiting these techniques to optimally speed up the algorithm of section 2 is given in Section 4. Finally, in Section 5 we show the exact complexity figures for an implementation example which concludes the paper.
2
Basic Version of the New Algorithm
Our goal is the computation of P = X*Y , where X and Y are large n-bit numbers (typical for our applications: n ≥ 1024). We want to do this in optimal time, i.e. T = 2 2 O(log n) and with limited area A = O(n /log n). In addition we are interested in keeping the constants small, since the well known algorithms with optimal ATcomplexity are not used in practical systems because their constant factors are too 2 large. To shorten the notation throughout this paper we define two variables k := log n 1/2 and t := log n . Initially, we subdivide X into intervals of k bits. There are a n/k intervals of k bits. We subdivide each interval of k bits into subintervals of t bits: t bits t bits
k/t intervals
t bits
t bits t bits
t bits
k/t intervals
t bits t bits
t bits
k/t intervals
n/k intervals
Fig. 1. Any interval of k bits is subdivided into intervals of t bits each.
Now our method proceeds in three steps: Step 1 calculates the products of Y with every possible t-bit value and stores the results in registers for further usage. Step 2 calculates the products of Y with all the k-bit-pieces of X. Step 3 combines these partial products to get the product of X and Y. 1/2 2 We will show that each step can be performed with O(n + n/log n) adders (and
Efficient Parallel Multiplication Algorithm for Large Integers
925
registers) in at most log n steps, which meets the time and area requirements stated above. t Step 1: There are 2 partial products to be computed. This operation is performed t completely in parallel. 2 adders are responsible for one value each. Each adder computes the product of one individual t-bit-number and Y. This multiplication is performed according to the classical school method, such that log t shifted copies of Y or 0, respectively, are added to each other. 1/2 This operation requires log n additions of length n+t < 2n . The results are stored in a set of registers in order to wait for subsequent computation in step 2. t t In the step 1 we use 2 adders and a lookup table with 2 entries. Thus the area complexity of step 1 is t
3/2
A1 = O(n*2 ) = O(n )
(1) 1/2
The time complexity is given by the number of additions which is log n therefore
and
T1 = O(log n) Tadd
(2)
Step 2: In this step n/k partial products (the products of Y times the k-bit pieces of X) are computed in parallel. n/k adders are responible for this task, each one for one partial product. Each computation consists of k/t additions, where the operands are taken from the registers whose values have been computed in Step 1. The process for each k-bit piece is purely sequential. The adders start with an initial value of 0 as intermediate result. In the first step each adder reads the value from one of the registers which corresponds to the least significant t bits of its k-bit interval of X. It is added to the intermediate result. After that the next t bits of the k-bit piece is taken. The corresponding value is taken from the appropriate register, it is shifted by t bits, and added to the intermediate result. In the same way the complete k-bit substring of X is worked through in k/t substeps corresponding to the k/t pieces of t bits. The final result is the product of Y and the k-bit substring of X. Since we need n/k adders of size 2n each, the area respectively time complexity of step 2 is 2
2
2
2
1/2
A2 = O(n /k)=O(n /log n), T2 = O(k/t)Tadd = O(log n/logn )Tadd = O(logn)Tadd
(3)
Step 3: The n/k partial products of step 2 are added to one final result. n/2k adders are used to implement this addition according to a binary tree. Like in Step 2, each partial product is shifted with respect to the position of its corresponding k-bit substring of X. The area complexity of Step 3 results from the number of adders required. Again, each of them has a width of 2n bits. 2
2
2
A3 = O(n /2k) = O(n /log n)
(4)
The time complexity is determined by the number of additions that one adder has to perform. It is equal to the depth of the binary tree. T3 = O(logn/k) Tadd ⊆ O(logn) Tadd
(5)
The result is P=X*Y. The time complexity of the complete algorithm is obviously T1+T2+T3 = O(logn) Tadd
(6)
926
V. Bunimov and M. Schimmler
The area complexity is 2
2
3/2
2
2
A1+A2+A3 = O(n /log n+n ) = O(n /log n)
(7)
Consequently, the AT-complexity is 2
2
2
AT = O(logn*n /log n) = O(n /logn) Tadd
3
(8)
Carry Save Adders and Redundant Number Representation
The central operation of most algorithms for integer multiplication including ours is addition. There a many different methods for a addition: ripple carry addition, carry select addition, carry look ahead addition and others [7]. The disadvantage of all of these methods is the carry propagation which makes the latency of the addition depend on the length of the operands. This is not a big problem for operands of size 32 or 64 but if the operand size is in the range of 1024. The resulting delay has a significant influence on the time complexity. Carry save addition [6] is a method for an addition without carry propagation. It is simply a parallel collection of n full-adders without any interconnection. Its function is to add three n-bit integers X, Y, and Z to produce two integers C and S as results such that C+S=X+Y+Z th
(9)
st
The i bit of the sum si and the (i+1) bit of carry ci+1 is calculated using the boolean equations si = xi ⊕ yi ⊕ zi, ci+1 = xiyi ∨ xizi ∨ yizi ,
(10)
in other words, a carry save adder cell is just a full-adder cell. When carry save adders are used in an algorithm one uses a notation of the form (S, C) = X + Y + Z
(11)
to indicate that two results are produced by the addition. The results are now represented in two binary n-bit words. Of course, this representation is redundant in the sense that we can represent one value in several different ways. This redundant representation has the advantage that the arithmetic operations are fast, because there is no carry propagation. An n-bit carry save adder consists of n full adders (a circuit for a 1-bit wide addition). We define the area of one full adder as 1. Therefore, the area of an n-bit carry save adder is n. We define the time complexity as the number of steps necessary to do the job. The time complexity of one full adder is TFA. Thus, the time complexity of carry save adder is TFA too, because all the full adders operate in parallel. We define a new kind of redundant adder as an adder with 4 inputs and 2 outputs. This is a cascade of two carry save adders. (S, C) = S1 + C1 + S2 + C2 = (S1, C1) + (S2, C2) We will use a notation
(12)
Efficient Parallel Multiplication Algorithm for Large Integers
S’ = S’1 + S’2
927
(13)
whereby S’, S’1, and S’2 are redundant numbers with S’ = (S, C), S’1 = (S1, C1), and S’2 = (S2, C2)
(14)
The area complexity of a redundant n-bit adder of this kind is 2n. The time complexity is 2TFA.
4
Optimised Version of the New Algorithm Using Redundant Representation of Numbers
The AT-complexity of our algorithm is 2
O(n /logn)
(15)
The standard addition requires a time complexity of O(logn)
(16)
Nor there is straightforeward way to speed up our algorithm by using the carry save addition with constant time complexity. Thus, we use redundant adders instead of standard adders. We calculate the product P’=X*Y,
(17)
where P’ is a redundant number with P’=(S, C)
(18)
The time complexity and the area complexity is equal to the time and area complexity of the basic algorithm in section 2. The result consists now of two numbers S and C. Finally, we add S and C with a standard adder with area complexity of O(n)
(19)
O(logn)
(20)
and time complexity of
The time complexity of the new algorithm is O(logn+logn) = O(logn)
(21)
The area complexity of the new algorithm is 2
2
2
2
O(n /log n+n) = O(n /log n)
(22)
Consequently, the AT-complexity is 2
2
2
O(n /log n*logn) = O(n /logn)
(23)
The redundant representation doubles the amount of register space for all intermediate results, because every nonredundant n-bit number is now represented by two n-bit numbers. But this increase of a factor of two in area in not visible here due to the O-notation.
928
5
V. Bunimov and M. Schimmler
Conclusion
In this paper we have given a simple parallel algorithm for multiplication of large 2 2 integers with optimal time (O(logn)) and low area (O(n /log n)) complexity. It is well suited for an implementation in real world applications. We give an example for the precise complexity: For multiplication of two 1024-bit numbers it requires 104 nonredundant registers, 64 carry save adders and 1 standard adder. The time complexity of the new algorithm is 74TFA, where TFA is the delay of a full adder.
References [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15]
Ciminiera, L., Montuschi, P.: Carry-Save Multiplication Schemes Without Final Addition, IEEE Transactions on Computers 45 (1996), 1050–1055. Cooley, J. W., Tukey, J. W.: An Algorithm for the machine calculation of complex Fouries series, Mathematics of Computation 19 (1965), 297-301, MR 31 #2843. Gentleman, W. M., Sande, G.: Fast Fourier transforms–for fun and profit, AFIPS 1966 Fall Joint Computer Computer Conference, Spartan Books, Washington, 1966, 563– 578 Karatsuba, A. A., Ofman, Y.: Multiplication of multidigit numbers on automata, Sovjet Physics Doklady 7 (1963), 595–596. Knuth. D. E.: The art of computer programming, volume 2: seminumerical algorithm, nd 2 edition, Addison-Wesley, Reading Massachusetts, 1981. Koc, C. K.: RSA Hardware Implementation, RSA Laboratories, RSA Data Security, Inc. August 1995, http://security.ece.orst.edu/koc/papers/reports.html Lee, S. C.: Digital Circuits and Logic Design, Prentice-Hall, Inc. Englewood Cliffs, New Jersey, 1976. Schönhage, A., Strassen, V.: Schnelle Multiplikation großer Zahlen, Computing 7 (1971), 281–292. MR #1431. Singer, B., Saon, G.: An efficient algorithm for parallel integer multiplication, Journal of Network and Computer Applications 19 (1996), 415–418. Stelling, P. F., Martel, C. U., Oklobdzija, V. G., and Ravi, R.: Optimal Circuits for Parallel Multipliers, IEEE Transactions on Computers 47 (1998), 273–285. Toom, A. L.: The complexity of a scheme of functional elements realizing the multiplication of integers, Sovjet Physics Doklady 3 (1963), 714–716. Wallace, C. S.: A Suggestion for a Fast Multipliers, IEEE Transactions Electronic Computing, 13 (1964), 14–17. Wegener, I.: Efficiente Algorithmen für grundlegende Funktionen, Chapter 3, 75-125, B. G. Teubner, 1989. Winograd S.: Arithmetic complexity of computations, CBMS-NSF Regional conference Series in Applied Mathematics, Philadelphia, 1980. ISBN 0-89871-163-0. MR 81k:68039. Zuras, D.: More on squaring and multiplying large integers, IEEE Transactions on Computers 43 (1994), 899–908.
Topic 14 Routing and Communication in Interconnection Networks Jose Duato, Olav Lysne, Timothy Pinkston, and Hermann Hellwagner Topic Chairs
Communication networks, protocols, and application programming interfaces (APIs) are crucial factors for the performance of parallel and distributed computations. This topic of Euro-Par 2003 is therefore devoted to all aspects of communication in on-chip interconnects, parallel computers, networks of workstations, and more widely distributed systems such as grids. Papers were solicited that examine the design and implementation of interconnection networks and communication protocols, advances in system area and storage area networks, routing and communication algorithms, and the communication costs of parallel and distributed algorithms. On-chip and power-efficient interconnects, I/O architectures and storage area networks, switch architectures as well as multimedia and QoS-aware communication were new topics introduced in this year’s Call for Papers (CfP). The CfP attracted 24 submissions to this topic, of which eight papers (33%) were selected for publication and presentation at the conference. Six of them were accepted as full papers, two as short ones. The selected papers cover a wide scope, ranging from convenient and efficient communication library designs to congestion control algorithms, QoS routing, packet forwarding in the InfiniBand Architecture, and deflection routing in unbuffered all-optical networks. We would like to thank all the authors for their submissions to this Euro-Par topic. We owe special thanks to the 25 external referees who provided competent and timely review reports. Their effort has ensured the high quality of this part of Euro-Par 2003. We trust you will find this topic to be highly stimulating and quite informative.
H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, p. 929, 2003. c Springer-Verlag Berlin Heidelberg 2003
Dynamic Streams for Efficient Communications between Migrating Processes in a Cluster Pascal Gallard and Christine Morin IRISA/INRIA – PARIS project-team
[email protected] http://www.kerrighed.org
Abstract. This paper presents a communication system designed to allow efficient process migration in a cluster. The proposed system is generic enough to allow the migration of any kind of stream: socket, pipe, char devices. Communicating processes using IP or Unix sockets are transparently migrated with our mechanisms and they can still efficiently communicate after migration. The designed communication system is implemented as part of Kerrighed, a single system image operating system for a cluster based on Linux. Preliminary performance results are presented.
1
Introduction
Clusters are now more and more widely used as an alternative to parallel computers as their low price and the performance of micro-processors make them really attractive for the execution of scientific applications or as data servers. A parallel application is executed on a cluster as a set of processes which are spread among the cluster nodes. In such applications, processes may communicate and exchange data with each other. In a traditional Unix operating system, communication tools can be streams like pipe or socket for example. For loadbalancing purpose, a process may be migrated from one node to another node. If this process communicates, special tools must be used in order to allow high performance communication after the migration. This paper presents a new communication layer for efficient migration of communicating processes. The design of this communication layer assumes that processes migrate inside the cluster and do not communicate with processes running outside the cluster. In the Kerrighed operating system [5], depending on the load-balancing policy, processes may migrate at any time. The remainder of this paper is organized as follows. Sect. 3 describes the dynamic stream service providing the dynamic stream abstraction and Kerrighed sockets. Sect. 4 shows how the dynamic stream service can be used to implement distributed Unix sockets. Sect. 5 presents performance results obtained with Kerrighed prototype. Conclusions and future works are presented in Sect. 6. H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 930–937, 2003. c Springer-Verlag Berlin Heidelberg 2003
Dynamic Streams for Efficient Communications between Migrating Processes
2
931
Background
The problem of migrating a communicating process is difficult and this explains why several systems, such as Condor [4], provide process migration only for non communicating processes. The MOSIX[1] system uses deputy mechanisms in order to allow the migration of a communicating process. When a process migrates (from a home-node to a host-node), a link is created between this process and the deputy. Every communication from/to this process is transmitted to the deputy that acts as the process. In this way, migrated processes are not able to communicate directly with other processes and thus communication performance decreases after a migration. The Sprite Network operating system[3] uses similar mechanisms in order to forward kernel calls whose results are machine-dependent. Several works like MPVM[2] or Cocheck[9] allow the migration of processes communicating by message passing. However, these middle-wares are not transparent for applications. Mobile-TCP[7] provides a migration mechanism in the TCP protocol layer using a virtual port linked to the real TCP socket. Mobility is one of the main features of IPv6[6] but communications can migrate only if the IP address migrates. In this case, one process must be attached to one IP address and each host must have several IP addresses (one for each running communicating process). Even in this case, only one kind of communication tools (inet sockets) can migrate. Another case of communication migration is detailed in M-TCP [11] where a running client, outside of the cluster, can communicate with a server through an access point. If processes on servers migrate, access points can hide the communication changes. None of these proposals offer a generic and efficient mechanism for migrating streams in a cluster allowing a migrating process to use all standard communication tools of a Unix system. We want to avoid message forwarding between cluster nodes when processes migrate. We propose a generic approach in order to provide standard local communication tools like pipe, socket and char -devices compliant with process migration (decided for load-balancing reasons or due to configuration changes in the cluster – addition/eviction of nodes). This approach has been implemented as part of Kerrighed project at the operating system level and in this way provides full migration transparency to communicating applications.
3
Dynamic Streams
Our work aims at providing standard communication interfaces such as Unix sockets or pipes to migrating processes in a cluster. Migrating a process should not alter the performance of its communications with other processes. A communication comprises two distinct aspects: the binary stream between two nodes, and the set of meta-data describing the state of the stream and how to handle it. Our architecture is based on this idea. We propose the concept of dynamic stream on which standard communication interfaces are built. We call the endpoints of these streams as ”KerNet
932
P. Gallard and C. Morin
Sockets” and these can be migrated inside the cluster. Dynamic streams and KerNet sockets are implemented on top of a portable high performance communication system providing a send/receive interface to transfer data between different nodes in a cluster. Socket
KerNet
Direct
Dynamic stream
Low−level Point−to−Point communication system
Netdevice
Pipe
FIFO
Char
FIFO
LIFO
Myrinet
TCP/IP
Fig. 1. Kerrighed network stack
The proposed architecture is depicted in Figure 1. Low-level Point-to-Point (PtP) communication service can be based on device drivers (such as myrinet), the generic network device in Linux kernel (netdevice) or a high-level communication protocol (such as TCP/IP). It is reliable and provides messages orders. On top of the low level point-to-point layer, we provide 3 kinds of dynamic streams: direct, FIFO and LIFO streams. We use these dynamic streams, implemented by the KerNet layer to offer dynamic version of standard Unix stream interfaces (sockets, pipe. . . ). It is a distributed service which provides global stream management cluster wide. In the remainder of this paper, we focus on the design and implementation of the KerNet layer and the Unix socket interface. 3.1
Dynamic Stream Service
We define a KerNet dynamic stream as an abstract stream with two or more defined KerNet sockets and with no node specified. When needed, a KerNet socket is temporarily attached to a node. For example, if two KerNet sockets are attached, send/receive operations can occur. A KerNet dynamic stream is mainly defined by several parameters: – Type of stream: it specifies how data is transfered using the dynamic stream. A stream can be: • DIRECT for one to one communication, • FIFO or LIFO for stream with several readers and writers. – Number of sockets: number of existing sockets in the stream – Number of connected sockets: it specifies the current number of attached sockets. – Data filter: it allows modification of all data transmitted with the stream (in order to have cryptography, backup. . . ). Streams are managed by a set of stream managers, one executing on each cluster node. Kernel data structures related to dynamic streams are kept in a global directory which is distributed on cluster nodes.
Dynamic Streams for Efficient Communications between Migrating Processes
3.2
933
KerNet Sockets
The KerNet service provides a simple interface to allow upper software layers implementing standard communication interface to manage KerNet sockets: – create/destroy a stream, – attach: to get an available KerNet socket (if possible), – suspend: to unattach temporarily a socket (and to give an handle in order to be able to reclaim the KerNet socket later), – wakeup: to attach a previously unattach KerNet socket, – unattach: to release an attached KerNet socket, – wait: to wait for the stream to be ready to be used (all required attachments completed). KerNet provides two other functions (send, recv) for I/O operations. The dynamic stream service is in charge of allocating KerNet sockets when it is needed, and of keeping track of these KerNet sockets. When the state of one KerNet socket changes, the stream’s manager takes part in this change and updates the other KerNet sockets related to the stream. With this mechanism, each KerNet socket has got the address of each corresponding socket’s node. In this way, two sockets can always communicate in the most efficient way. At the end of a connection, a process is unattached from the stream. Depending on the stream type, the stream may be closed. 3.3
Example of Utilization of the KerNet API in the OS
Let us consider two kernel processes (P1 and P2 ) communicating with each other using a dynamic stream. They execute the following program: (1) (2) (3) (4) (5)
Process P1 stream = create(DIRECT, 2); socket1 = attach(stream); wait(stream); ch = recv(socket1);
(6) (7) send(socket1, ch);
Process P2 socket2 = attach(stream); wait(stream); send(socket2, ch); sec2 = suspend(socket2); Process P3 socket3 = wakeup(stream, sec2); ch = recv(socket3);
Initialization of a KerNet stream: P1 creates the stream and requests a KerNet socket. Next P1 wait for its stream to be ready, that is to say the two sockets to be attached. Assuming P2 is running after the stream creation and has the stream identifier, it can get a KerNet socket, and then, wait for the correct state of the stream. The stream’s manager sends the acknowledgement to all waiting KerNet sockets and provides the physical address of the other socket. With such information, KerNet sockets can communicate directly and send /receive communication can occur efficiently. Migrating process using KerNet streams: If a process wants to migrate (or transfers its KerNet sockets to another process), it just uses the suspend function. When a process migration is needed, the KerNet socket is suspended on
934
P. Gallard and C. Morin
the departure node and re-attached on the arrival node. In our example, P2 suspends the socket, which is latter re-attached when P3 executes. The dynamic stream service is in charge of ensuring that no message (or message piece) is corrupted or lost between the suspend and re-attached time. The stream manager updates the other KerNet socket so that it stops its communication until it receives new information from the stream manager. When the suspended socket is activated again, its new location is sent to the other KerNet socket and direct communication between the two KerNet sockets is restarted.
4
Implementation of Standard Communication Interface Using Dynamic Streams
User mode
Obviously, standard distributed applications do not use KerNet sockets. In order to create a standard environment based on dynamic streams and KerNet sockets, an interface layer is implemented at kernel level (see Figure 2). Each module of the interface layer implements a standard communication interface relying the interface of the KerNet service. The main goal of each interface module is to manage the standard communication interface protocol (if needed).
User process
Send
Receive
sys_Socket
Open
sys_Open
IOctl
sys_IOctl
KerNet Interface
Point−to−Point communication system
Attach
sys_Receive
Create
KerNet system
sys_Send
Kernel mode
SysCall
Socket
Network
Fig. 2. Standard environment based on KerNet sockets
KerNet interfaces are the links between the standard Linux operating system and the Kerrighed dynamic communication service. The Kerrighed operating system is based on top of a lightly-modified Linux kernel. All the different services, including the communication layer, are implemented as Linux kernel modules. In Kerrighed operating system, the communication layer is made of two parts. A static high-performance communication system that provide a node to node service. On top of this system, the dynamic stream service manages the migration of streams’s interfaces. Finally, the interface service replaces the standard functions for a given communication tool.
Dynamic Streams for Efficient Communications between Migrating Processes
935
In the remainder of this section, we describe the Unix socket[10] interface on the KerNet sockets. We aim at providing a distributed Unix socket, transparent to the application. In the standard Linux kernel, Unix sockets are as simple as unqueuing some packets from the sending socket and queuing them in the receiving socket. In this case the (physical) shared memory allows the operating system to access to the system structures of the two sockets. In the same way, the protocol management can be done easily. Obviously, in our architecture, the operating system of one node may not have access to the data structure of the other socket. Based on the KerNet services, the KerNet Unix sockets interface must manage the standard Unix sockets communication protocol. When a new interface is defined, a corresponding class of stream is registered in the dynamic stream service. A class of stream is a set of streams that share the same properties. This registering step defines general properties of streams associated to this interface (stream type, number of sockets. . . ) and returns it a stream class descriptor. When a process makes an accept on a Unix socket, the Unix socket interface creates a new stream, attaches the first KerNet socket and waits for the stream allocate the other KerNet socket (as in P1). When another process (may be on another node) executes a connect on the Unix socket, an attach attempt is made (process P2). On success, the stream is completed and the two KerNet sockets (and by this way the two Unix sockets) can communicate directly. The accept/connect example is a good representation of how we implement Unix sockets. With the same approach we have designed and implemented other standard socket functions like poll (in order to be able to use select syscall), listen, and so on. Send and receive functions are directly mapped on the send and receive KerNet socket functions. Based on this interface, we may provide other standard interfaces such as pipe, inet socket and even some access to char device. When a migration occurs (decided by the global scheduler or by the program itself), the migration service calls the suspend function and attaches the socket on the new node (such as P3).
5
Performance Evaluation
In the current implementation, KerNet provides standard Inet and Unix sockets interfaces. In order to have some performance evaluation of our communication system, we used the NetPipe[8] application. This benchmark is a ping-pong application with several packet’s size. We use the vanilla-TCP version and a KerNet one in order to use Unix socket. Several physical networks are used inside the cluster: the loopback interface (Fig. 3(a)), FastEthernet (Fig. 3(b)), and Gigabits Ethernet (Fig. 3(c)). In the FastEthernet and Gigabits Ethernet networks each node is a Pentium III (500MHz, 512KB cache) with 512MB of memory. The Kerrighed system used is an enhanced 2.2.13 Linux kernel.
936
P. Gallard and C. Morin
In addition, we provide in all cases, the performance of our point to point communication layer (without any dynamic functionality). We must notice that, these measures represent PtP communications from KerNet to kernel with buffers physically allocated in memory. In this case, buffers are in contiguous memory area and their size are lower than 128KB. Thus, the PtP low-level performance is a maxima for our KerNet stream. 900
100
800
90 80
700
70 Bandwitdh (Mbits/s)
Bandwitdh (Mbits/s)
600
500
400
60 50 40
300 30 200
20 TCP-like with kernet TCP Vanilla kernel Unix Vanilla kernel Distributed unix socket
100
TCP-like with kernet TCP Without kernet TCP Vamilla kernel Point to Point communication Distributed unix socket
10
0
0 1
10
100
1000
10000 100000 Packet size (bytes)
1e+06
1e+07
1e+08
1
10
(a) Loopback
100
1000 10000 Packet size (bytes)
100000
1e+06
1e+07
(b) FastEthernet
450
350
400 300 350 250 Bandwitdh (Mbits/s)
Bandwitdh (Mbits/s)
300
250
200
200
150
150 100 100 50 50
TCP-like with kernet TCP Vanilla kernel Point to Point communication
TCP-like, before migration TCP-like, after migration
0
0 1
10
100
1000 10000 Packet size (bytes)
100000
(c) GigaBit Ethernet
1e+06
1e+07
1
10
100
1000 10000 Packet size (bytes)
100000
1e+06
1e+07
(d) GigaBit Ethernet w. migration
Fig. 3. Throughput of several communication systems
First, we notice that the interfaces have a low impact on the dynamic stream: Unix socket and TCP sockets have nearly the same results. With a FastEthernet network, the KerNet dynamic stream bandwidths are nearly the same than PtP low-level one. On GigaBit ethernet network, transfers between user-space and kernel-space are more perceptible. When two communicating processes are on the same node, dynamic streams outperform the standard TCP sockets. This is mainly due to the small network stack in KerNet: IP stack provides some network services which are useless in a cluster or already performed by our low-level communication layer. However we do not reach (on a single node) the performance of a Unix socket. This is mainly due to the design of the low-level communication layer which as been designed for inter-node communications without any optimization for local communications. When two communicating processes are not on the same node, KerNet outperformed TCP socket again. The reason are the same as above.
Dynamic Streams for Efficient Communications between Migrating Processes
937
Other experiments have been performed to evaluate the impact of a migration on the dynamic stream performance. The NetPIPE application for TCP (NPtcp) has been modified to trigger a KerNet socket migration. Figure 3(d) shows that there is no overhead after a migration of a communicating process.
6
Conclusion
In this paper we have described the design and implementation of a distributed service allowing efficient execution of communicating processes after migration. We have introduced the concept of dynamic stream and mobile sockets. We have shown on the example of Unix sockets how standard communication interfaces can take advantage of these concepts. The proposed scheme has been implemented in the Kerrighed operating system. We currently study communications in the context of the migration of standard MPI processes without any modification of the application and of the MPI library. In future works on dynamic streams we plan to provide other stream standard communication interfaces like pipes and access to char devices. We also plan to study fault-tolerance issues in the framework of the design and implementation of checkpoint/restart mechanisms for parallel applications in Kerrighed cluster operating system.
References 1. A. Barak, S. Guday, and R. G. Wheeler. The MOSIX Distributed Operating System, volume 672 of Lecture Notes in Computer Science. Springer, 1993. 2. J. Casas, D. Clark, R. Konuru, S. Otto, R. Prouty, and J. Walpole. MPVM: A migration transparent version of PVM. Technical Report CSE-95-002, 1, 1995. 3. F. Douglis and J. Ousterhout. Transparent process migration: Design alternatives and the Sprite implementation. Software–Practice & Experience, 21(8), August 1991. 4. M. Litzkow, T. Tannenbaum, J. Basney, and M. Livny. Checkpoint and migration of UNIX processes in the Condor distributed processing system. Technical Report 1346, University of Wisconsin-Madison Computer Sciences, April 1997. 5. C. Morin, P. Gallard, R. Lottiaux, and G. Vall´ee. Towards an efficient single single system image cluster operating system. In ICA3PP, 2002. 6. C. E. Perkins and D. B. Johnson. Mobility support in IPv6. In Mobile Computing and Networking, pages 27–37, 1996. 7. Xun Qu, J. Xu Yu, and R. P. Brent. A mobile TCP socket. Technical Report TR-CS-97-08, Canberra 0200 ACT, Australia, 1997. 8. Q. Snell, A. Mikler, and J. Gustafson. Netpipe: A network protocol independent performace evaluator, 1996. 9. G. Stellner. Cocheck: Checkpointing and process migration for mpi. In International Parallel Processing Symposium, 1996. 10. R. Stevens. Unix Network Programming, volume 1-2. Prentice Hall, 1990. 11. F. Sultan, K. Srinivasan, D. Iyer, and L. Iftode. Migratory TCP: Highly available internet services using connection migration. In Proceedings of The 22nd International Conference on Distributed Computing Systems (ICDCS), July 2002.
FOBS: A Lightweight Communication Protocol for Grid Computing Phillip M. Dickens Department of Computer Science, Illinois Institute of Technology, 235-C Stuart Building, Chicago, Illinois 60616
[email protected] Abstract. In this paper, we discuss our work on developing an efficient, lightweight application-level communication protocol for the high-bandwidth, high-delay network environments typical of computational grids. The goal of this research is to provide congestion-control algorithms that allow the protocol to obtain a large percentage of the underlying bandwidth when it is available, and to be responsive (and eventually proactive) to developing contention for system resources. Towards this end, we develop and evaluate two applicationlevel congestion-control algorithms, one of which incorporates historical knowledge and one that only uses current information. We compare the performance of these two algorithms with respect to each other and with respect to TCP.
1 Introduction The national computational landscape is undergoing radical changes as a result of the introduction of cutting-edge networking technology and the availability of powerful, low-cost computational engines. This combination of technologies has led to an explosion of advanced high performance distributed applications that, because of the limited bandwidth and best-effort nature of the Internet1 environment, were heretofore infeasible. Concurrently, research efforts have focused on the development of Grid computing, a fundamentally new set of technologies that create large-scale distributed computing systems by interconnecting geographically distributed computational resources via very high-performance networks. The advanced applications being developed to execute in Grid environments include distributed collaboration, remote visualization of terabyte (and larger) data sets, large-scale scientific simulations, Internet telephony, and advanced multimedia applications. Arguably, Grid computing will reach its vast potential if, and only if, the underlying networking infrastructure (both hardware and software) is able to transfer vast quantities of data across (perhaps) quite long distances in a very efficient manner. Experience has shown, however, that advanced distributed applications executing in existing large-scale computational Grids are often able to obtain only a very small fraction of the available underlying bandwidth (see, for example, [21, 6, 11]). The reason for such poor performance is that the Transmission Control Protocol (TCP [3]), the communication mechanism of choice for most distributed applications, was not designed and is not well suited for a high-bandwidth, high-delay network H. Kosch, L. Böszörményi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 938–946, 2003. © Springer-Verlag Berlin Heidelberg 2003
FOBS: A Lightweight Communication Protocol for Grid Computing
939
environment [21, 11, 17]. This issue has led to research aimed at improving the performance of the TCP protocol itself in this network environment [21, 1, 5, 10], as well as the development of application-level techniques that can circumvent the performance problems inherent within TCP [13, 12, 16, 19, 20]. Despite all of the research activity undertaken in this area, significant problems remain in terms of obtaining available bandwidth while being in some sense fair to competing flows. While TCP is able to detect and respond to network congestion, its very aggressive congestion-control mechanisms result in poor bandwidth utilization even when the network is lightly loaded. User-level protocols such as GridFTP[2], RUDP[16], and previous versions of FOBS [7-9], are able to obtain a very large percentage of the available bandwidth. However, these approaches rely on the characteristics of the network to provide congestion control (that is, they generally assume there is no contention in the network). Another approach (taken by SABUL [20]) is to provide an application-level congestion-control mechanism that is closely aligned with that of TCP (i.e. using the same control-feedback interval of a single round trip time). We are attempting to approach the problem from a somewhat different perspective. Rather than attempting to control the behavior of the protocol on a time scale measured in milliseconds, we are interested in developing approaches that can operate on a much larger time scale while still providing very effective congestion control. We believe the best way to achieve this goal is to use the historical information that is available to the application to help drive the congestion-control algorithms. The work presented in this paper is aimed at showing that our approach is both feasible and useful. The rest of the paper is organized as follows. In Section 2, we provide a brief overview of the components of the FOBS data transfer system. In Section 3, we discuss the design of two user-level congestion control algorithms. In Section 4, we discuss the design of the experiments developed to evaluate the performance of our control mechanisms and to compare their performance with that of TCP. In Section 5, we discuss the results of these experiments. We discuss related work in Section 6, and provide our conclusions and future research directions in Section 7.
2 FOBS FOBS is a simple, user-level communication mechanism designed for large-scale data transfers in the high-bandwidth, high-delay network environment typical of computational Grids. It uses UDP as the data transport protocol, and provides reliability through an application-level acknowledgment and retransmission mechanism. Experimental results have shown that FOBS performs extremely well in a computational Grid environment, consistently obtaining on the order of 90% of the available bandwidth across both short- and long-haul network connections [7-9]. Thus FOBS addresses quite well the issue of obtaining a large percentage of the available bandwidth in a Grid environment. However, FOBS is in-and-of-itself a very aggressive transport mechanism that does not adapt to changes in the state of the endto-end system. Thus to make FOBS useful in a general Grid environment we have developed congestion control mechanisms that are responsive to changes in system
940
P.M. Dickens
conditions while maintaining the ability to fully leverage the underlying highbandwidth network environment. 2.1
Rate-Controlled FOBS
Rate-Controlled FOBS enhances the basic FOBS mechanism by placing simple control agents at the communications endpoints. The agents collect and exchange information related to the state of the transfer and cooperatively implements the congestion control algorithms. The data transfer engine itself is (at least conceptually) reasonably simple. While it is beyond the scope of this paper to discuss in detail the implementation of FOBS and the technical issues addressed (the interested reader is directed to [8, 9, 18] for such a discussion), it is worthwhile to briefly discuss the implementation of the reliability mechanism. FOBS employs a simple acknowledgment and retransmission mechanism. The file to be transferred is divided into data units we call chunks, and data is read from the disk, transferred to the receiver, and written to disk in units of a chunk. We have set the chunk size to 100 MBs, this number being chosen based on extensive experimentation. Each chunk is subdivided into segments, and the segments are further divided into packets. Packets are 1,470 bytes (within the MTU of most transmission mediums), and a segment consists of 10,000 packets. The receiver maintains a bitmap for each segment in the current chunk depicting the received/notreceived status of each packet in the segment. A 10,000-packet segment requires a bitmap of 1,250 bytes, which will also fit in a data packet within the MTU of most transmission media. These bitmaps are sent from the data receiver to the data sender at intervals dictated by the protocol, and trigger (at a time determined by the congestion/control flow algorithm) a retransmission of the lost packets. The data to be transferred uses UDP sockets and the bitmaps are sent on TCP sockets. There are two advantages of using this approach. First, such segmentation of the data de-couples the size of an acknowledgment packet from the size of the chunk. For example, a chunk size of 100 MBs would require on the order of 8,700 bytes for the acknowledgment packet (assuming one bit per data packet). While the size of the acknowledgment packet could be decreased with the use of negative acknowledgments, it is not difficult to imagine that it may still require bitmaps of a size (perhaps significantly) larger than the MTU of a given transmission medium. This makes it more difficult to deliver the feedback in a timely fashion (it would have to be fragmented along the way), which would have a very detrimental impact on performance. Perhaps more importantly, these bitmaps provide a precise and detailed picture of the packet-loss patterns caused by the current state of the end-to-end system. We term these bitmaps packet-loss signatures, and preliminary research suggests that as the events that drive packet loss change the packet-loss signatures themselves also change. Currently, the packet-loss signatures are fed to a visualization system as the transfer is taking place, and we are attempting to develop an understanding of these signatures. The goal is to incorporate information gleaned from these bitmaps into the congestion-control mechanisms.
FOBS: A Lightweight Communication Protocol for Grid Computing
941
3 Congestion Control Algorithms We are interested in evaluating approaches to congestion control that operate under a different set of assumptions than current techniques. One issue in which we are interested is whether the feedback-control interval must be pegged to roundtrip times in order to provide effective congestion control and ensure fairness. This approach is problematic for two reasons: First, a small spike in packet loss may well represent an event that has dissipated by the time the sender is even aware of its occurrence. Thus the congestion control mechanism may be reacting to past events that will not reoccur in the immediate future. Secondly, such an approach can become unstable. That is, the algorithm can get into cycles of increasing the sending-rate due to low packet loss, followed by a spike in packet-loss due to increasing the rate, followed by backing off in reaction to increased loss, and so forth. For these reasons, we believe it is important to explore alternative approaches. We have developed and tested two alternative approaches to congestion control, one where the feedback-control interval is significantly lengthened and the increase/decrease parameters for the send-rate are linear. This approach incorporates historical knowledge into the design of the algorithm. The other approach is statebased, where the behavior of the algorithm depends only upon the current state and the current feedback. It is important to note the way in which historical knowledge is incorporated into the first protocol. In particular, the algorithm was developed based on empirical observations made during the development and testing of the FOBS system. Thus the historical patterns were deduced by direct observation, and deducing such information from statistical measures would of course be significantly more complex. However, it does point to the fact that historical information can be important. One observation was that once the protocol found a sending rate that resulted in negligible packet-loss, it could remain at that rate for a reasonably long period of time (which in general was longer than it took to successfully transfer one complete chunk of data). Another observation was that once such a rate was established, the best approach was to stay at that rate. That is, there appeared to be an “optimal” sending rate such that being more aggressive tended to result in non-proportional increases in packet loss, and becoming less aggressive did not result in a meaningful decrease in packet loss. Finally, given a current “optimal” sending rate S, it was observed that the next such sending rate was quite often close to S. Such characteristics were not observed on all connections all the time. However, it was a pattern that emerged frequently enough to be noticed. 3.1
Historical Knowledge
These observations were incorporated into the protocol in the following ways. First, the feedback-control interval was extended to the time required to successfully transfer one complete chunk of data. The mean loss rate for an entire chunk was calculated once the data had been successfully transferred, and if this rate exceeded a threshold value the send-rate was decreased. The threshold value was ½ of 1%, and the decrease parameter was a constant 5 Mbs on a 100 Mbs link and 50 Mbs on Gigabit connection. This slow decrease in the send rate is clearly not appropriate when the current conditions are causing massive packet loss and a reduction in sending rate (or several reductions) does not have a significant impact on the loss rate.
942
P.M. Dickens
We are investigating techniques to detect such conditions and respond by either raising the decrease parameter or perhaps by switching to another protocol. The increase parameter for this approach was also linear, but the algorithm does not necessarily increase the sending rate each time the loss-rate falls (or remains) below the current threshold value. Rather, the protocol keeps track of the current rate, the number of times the sending rate has been incremented from the current rate, and the number of times such an increase has resulted in a significant increase in packet loss. Based on this sample, the probability that increasing the sending rate will result in increased packet loss is computed, and the protocol then uses this probability to determine if the sending rate will be increased or remain unchanged. It is important to note that this protocol incorporates two different types of historical knowledge. One type of knowledge was the empirical observations that suggested the basic framework of the algorithm. The other source of knowledge was obtained by tracking the historical relationship between an increase in the sending rate (from a given rate A to A + increase_amount) and an increase in the loss-rate. Thus the protocol captured information obtained both within a given session and between multiple sessions. 3.2
State Based Approach
We also investigated what may be termed a state-based approach, where decisions regarding the send rate are based solely on the current state and current feedback information. We have developed such an algorithm and have implemented three states thus far: Green, Yellow, and Red. These states represent excellent, moderate, and poor system conditions respectively. Each state has its own minimum and maximum sending rate, its own increase and decrease parameters, and its own set of conditions under which it will change state. The feedback-control interval is the amount of time required to successfully transfer one segment of data to increase the responsiveness of the protocol to current conditions. The parameters are set such that the sending rate is increased at a high rate when there is little packet loss, and decreased very quickly in the event of massive packet loss.
4 Experimental Design Our experiments were conducted on links between Argonne National Laboratory and the Center for Advanced Computing Resource (CACR), and links between Argonne National Laboratory and the National Center for Supercomputing Applications (NCSA). The round trip time between ANL and CACR (as measured by traceroute) was approximately 64 milliseconds, which we loosely categorize as a long-haul connection. The round trip time between ANL and NCSA was on the order of eight milliseconds, which we (again loosely) categorize as a short-haul connection. All links were connected through the Abilene backbone network. The slowest link between ANL and NCSA was an OC-12 connection from NCSA to Abilene; both ANL and CACR had to Gigabit Ethernet connections to Abilene. Two hosts were tested within NCSA: a 64 processor SGI Origin 2000 running IRIX 6.5 and a 128 node Itanium cluster running Linux 2.4.16. Each node in the
FOBS: A Lightweight Communication Protocol for Grid Computing
943
Itanium cluster consisted of an Intel 800MHz Itanium CPU (duel processor), and each processor of the SGI was a 195MHz MIPs R10000. The host at CACR was a fourprocessor HP N4000 running HP-UX 11. The tests at Argonne National Laboratory were conducted on a dual processor Intel i686 running Linux RedHat 6.1. All experiments were conducted using a single processor. We simulated the transfer of a 10-Gigabyte file from Argonne National Laboratory to the machines at both NCSA and CACR. That is, we performed the number of iterations necessary to complete a transfer of that size but did not actually perform the read from and write to disk (largely because of disk quotas on some hosts). We were unable to send in the opposite direction because of firewalls at ANL. The tests were conducted during normal business hours (for all hosts involved) to ensure there would be some contention for network and/or CPU resources. We were interested in the mean throughput and the overall loss percentage1. We were also interested in comparing the performance of Rate-Controlled FOBS with that of TCP, and conducted experiments between these same links to make such a comparison. The results obtained for TCP were extremely poor due to the small buffer size on the host at Argonne National Laboratory (64 KB), which could only be changed with root permissions (that we did not have). We present below these results, as well as results obtained in previous experiments where the Large Window extensions for TCP were enabled.
5
Experimental Results
The results of these experiments are shown in Table 1. The major difference between the two user-level algorithms is the higher loss rate experienced by the state-based approach on the link between ANL and CACR. This larger loss rate was a reflection the instability of the state-based approach on that particular link in the presence of contention. The instability arose from the fact that the algorithm immediately increases the sending rate whenever the loss rate fell (or remained) below the threshold value. Thus it spent a significant amount of time oscillating between sending at the “optimal” rate (as discussed in Section 3), and a rate that was not significantly higher but that resulted in significantly increased packet loss nonetheless. This phenomenon was due to the fact that negligible packet loss was experienced on the short-haul connections, and the algorithm thus rarely had to decrease the send rate. Another interesting result was that FOBS was not able to obtain a very large percentage of the OC-12 or the Gigabit Ethernet connections we tested. There were two primary factors contributing to this relatively low utilization of network bandwidth. First, the experiments were run during the day when contention for CPU and network resources was quite high. The throughput was much higher when the experiments were run at night when there was little competition for CPU or network cycles (e.g., on the order of 94% of the maximum bandwidth on the OC-12 connection between ANL and NCSA). 1
The loss percentage was calculated as follows. Let N be the total number of packets that had to be sent (assuming no packet loss), and let T be the total number of packets actually sent. Then the loss rate R = (T – N) / N.
944
P.M. Dickens
The second factor has to do with inefficiencies of the FOBS implementation. Currently, the select() statement is used to determine those sockets ready for either sending or receiving. This is a heavyweight system call that is used repeatedly in the current implementation. A better approach would be to use threads to determine when a socket becomes available, and we are currently implementing a multi-threaded version of FOBS. Table 1. This table shows the throughput and packet-loss percentage for FOBS using historical information (FOBS HI), FOBS using only current state information (FOBS CS) and TCP
Link
ANL to Itanium Linux Cluster at NCSA
ANL to SGI Origin 2000 at NCSA
ANL to HP N4000 at CACR
Protocol
Throughput
Loss
FOBS HI
190 Mbs
< 0.1%
FOBS CS
192 Mbs
< 0.1%
TCP
7.2 Mbs
----
FOBS HI
161 Mbs
< 0.1%
FOBS CS
156 Mbs
< 0.1%
TCP
6.5 Mbs
----
FOBS HI
86
Mbs
< 0.1%
FOBS CS
84 Mbs
2.4 %
TCP
1.6 Mbs
----
It is also quite clear that the application-level approach obtained performance significantly higher than that obtained by TCP across all experiments with very little packet loss. As noted above, the primary reason for TCP’s very poor performance has to do with the very small TCP buffers available on terra (which has a significant negative impact on TCP’s performance). In previous experiments that we conducted [7-9], using the same links but different hosts (that supported the Large Window extensions for TCP [17]), the results were significantly better. In particular, on the link between ANL and NCSA TCP obtained on the order of 90 Mbs, and obtained on the order of 50 Mbs on the link between ANL and CACR. However, even with the Large Window extensions, TCP’s performance was significantly less than that obtained by FOBS.
6 Related Work The work most closely related to our own are RUDP [16] and SABUL [20], both of which have been developed at the Electronics Visualization Laboratory at the
FOBS: A Lightweight Communication Protocol for Grid Computing
945
University of Illinois Chicago. RUDP is designed for dedicated networks (or networks with guaranteed bandwidth), making the issue of congestion control largely irrelevant. However, since FOBS is designed to operate in a general Grid environment the issues of congestion control must be addressed. SABUL does provide congestion control that uses a feedback-control interval pegged to round trip times. We believe this interval is too small for large-scale data transfers across high-performance networks, and the research presented in this paper is an attempt to move away from such a model of congestion control. Research related to application-level scheduling and adaptive applications is also related (e.g. the Apples project [4] and the Grads project [14]). However, this body of research is focused on interactions between an application scheduler and a heavyweight Grid scheduler provided in systems such as Globus [13] and Legion [15]. FOBS, on the other hand, is a very lightweight communication protocol that operates outside of the domain of such large systems. However, an interesting extension of our work would be to modify FOBS such that it could interact with Gridlevel schedulers to negotiate access to system resources.
7 Conclusions and Future Research In this paper, we described a lightweight application-level communication protocol for computational Grids. The protocol is UDP-based, with a simple acknowledgment and re-transmission mechanism. We discussed two different application-level congestion-control mechanisms, and showed that Rate-Controlled FOBS was able to significantly outperform TCP across all connections tested with very minimal packet loss (especially when historical knowledge was incorporated into the algorithm). There are many ways in which this research can be extended. The results provide herein suggest that integrating historical knowledge into the congestion control mechanism can significantly enhance its performance. This of course brings up the issue of how to automate the collection of historical data, which data has the best chance of providing insight into future behavior, and how such data can be incorporated into the control mechanisms. These are all issues we are currently pursuing. It is our goal to make FOBS a widely used transport protocol for data transfers within computational Grids. Towards this end, we have developed an easy-to-use drag-and-drop interface, a 32-bit CRC for increased reliability, and a sophisticated system for visualization of the packet-loss signatures. Currently, we are developing a multi-threaded version of the protocol to increase efficiency and a version of FOBS capable of performing cluster-to-cluster data transfers.
References [1] List of sack implementations. Web Page of the Pittsburgh Supercomputing Center. http://www.psc.edu/networking/all_sack.html. [2] Allcock, W., Bester, J., Breshahan, J., Chervenak, A., Foster, I., Kesselman, C., Meder, S., Nefedova, V., Quesnel, D., and Tuecke, S. Secure, Efficient Data Transport and Replica Management for High-Performance Data_Intensive Computing. In Proceedings of IEEE Mass Storage Conference, 2001.
946
P.M. Dickens
[3] Allman, M., Paxson, V., and W.Stevens. TCP Congestion Control, RFC 2581. http://www.faqs.org/rfcs/rfc2581.html [4] The AppLeS Homepage http://apples.ucsd.edu/ [5] Automatic TCP Window Tuning and Applications. National Laboratory for Advanced Networking Research Web Page http://dast.nlanr.net/Projects/Autobuf_v1.0/autotcp.html [6] Boyd, E.L., Brett, G., Hobby, R., Jun, J., Shih, C., Vedantham, R., and Zekauska, M. E2E piPEline: End-to-End Performance Initiative Performance Environment System Architecture. July, 2002. http://e2epi.internet2.edu/e2epipe11.shtml [7] Dickens, P. A High Performance File Transfer Mechanism for Grid Computing. In Proceedings of The 2002 Conference on Parallel and Distributed Processing Techniques and Applications (PDPTA). Las Vegas, Nevada, 2002. [8] Dickens, P., and Gropp, B. An Evaluation of Object-Based Data Transfers Across High Performance High Delay Networks. In Proceedings of the 11th Conference on High Performance Distributed Computing, Edinburgh, Scotland, 2002. [9] Dickens, P., Gropp, B., and Woodward, P. High Performance Wide Area Data Transfers Over High Performance Networks. In Proceedings of The 2002 International Workshop on Performance Modeling, Evaluation, and Optimization of Parallel and Distributed Systems., 2002. [10] Enabling High Performance Data Transfers on Hosts: (Notes for Users and System Administrators). Pittsburgh Supercomputing Center http://www.psc.edu/networking/perf_tune.html#intro [11] Feng, W., and Tinnakornsrisuphap, P. The Failure of TCP in High-Performance Computational Grids. In Proceedings of Proceedings of Super Computing 2000 (SC2000). [12] The FOBS Data Transfer System http://www.csam.iit.edu/~pmd/FOBS/fobs_index.html [13] The Globus Project http://www.globus.org [14] The Grid Applications Development Software Project Homepage http://juggler.ucsd.edu/~grads/ [15] Grimshaw, A., and Wulf, W. Legion-a view from 50,000 feet. In Proceedings of Proceedings of High-performance Distributed Computing Conference (HPDC), 1996. [16] He, E., Leigh, J., Yu, O., and DeFanti, T. Reliable Blast UDP : Predictable High Performance Bulk Data Transfer. In Proceedings of Proceedings of the IEEE Cluster Computing, Chicago, Illinois, September 2002. [17] Jacobson, V., Braden, R., and Borman., D. TCP Extensions for high performance. RFC 1323, May 1992. [18] Kannan, V., Dickens, P., and Gropp, W. A Performance Study of Application Level Acknowlegment and Retransmission Mechanisms. In Proceedings of The International Conference on High Performance Distributed Computing and Applications, Reno, Nevada, 2003. [19] Sivakumar, H., Bailey, S., and Grossman, R. PSockets: The Case for Application-level Network Striping for Data Intensive Applications using High Speed Wide Area Networks. In Proceedings of Super Computing 2000 (SC2000). [20] Sivakumar, H., Mazzucco, M., Zhang, Q., and Grossman, R. Simple Available Bandwidth Utilization Library for High Speed Wide Area Networks. Submitted to Journal of SuperComputing. [21] The Web100 Homepage http://www.internet2.edu/e2epi/web02/p_web100.shtml
Low-Fragmentation Mapping Strategies for Linear Forwarding Tables in InfiniBandTM Pedro L´ opez, Jos´e Flich, and Antonio Robles Dept. of Computing Engineering (DISCA) Universidad Polit´ecnica de Valencia P.O.B. 22012, 46071 - Valencia, SPAIN {plopez,jflich,arobles}@gap.upv.es
Abstract. The InfiniBand Architecture (IBA) supports distributed routing by using forwarding tables stored in each switch, which only consider the destination local identifier (LID) of the packet for routing. Each LID is mapped to a different table entry. Additionally, the IBA specifications allow each destination port to be assigned up to 2n consecutive virtual addresses by masking the n least significant bits of its LID. Each range of virtual addresses must be mapped to consecutive table entries when IBA linear forwarding tables are used. However, the fact that each port may require a different number of virtual addresses and the fact that this number may not be a power of two could lead to waste some table entries, causing a fragmentation of the forwarding tables as a consequence of an inefficient mapping strategy of LIDs. Fragmentation of the forwarding tables could become critical as far as it reduces the number of available table entries to map LIDs, limiting, in turn, the number of ports that can be placed in the network. In this paper, we propose two effective mapping strategies to tackle the fragmentation effect on IBA forwarding tables. The first strategy is able to remove the fragmentation effect when the number of virtual addresses is a power of two for all destinations, introducing a fragmentation percentage lower than 20% in all cases. On the other hand, the second strategy is able to almost completely eliminate the possible fragmentation effect.
1
Introduction
InfiniBand [2] has been recently proposed as a standard for communication between processing nodes and I/O devices as well as for interprocessor communication. The InfiniBand Architecture (IBA) is designed around a switch-based interconnect technology with high-speed point-to-point links. An IBA network is composed of several subnets interconnected by routers, each subnet consisting of one or more switches and end node devices (processing nodes and I/O devices). Each end node contains one or more channel adapters (referred to as HCA in
This work was supported by the Spanish MCYT under Grants TIC2000–1151–C07 and 1FD97-2129, by the JJ.CC. de Castilla-La Mancha under Grant PBC-02-008, and the Generalitat Valenciana under grant CTIDIB/2002/288.
H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 947–957, 2003. c Springer-Verlag Berlin Heidelberg 2003
948
P. L´ opez, J. Flich, and A. Robles
0 1 2
Virtual LMC addr. 3 8 3 8 3 8
...
...
...
8191
3
8
HCA
Forwarding Table output port for ID 0 output port for ID 1 ... output port for ID 7 output port for ID 8 output port for ID 9 ... output port for ID 15 output port for ID 16 output port for ID 17 ... output port for ID 23 ... output port for ID 65528 output port for ID 65529 ... output port for ID 65535
(a)
HCA 0
Virtual LMC addr 1 2
1
3
8
2 3
0 1
1 2
...
...
...
Forwarding Table output port for ID 0 output port for ID 1 ... output port for ID 8 output port for ID 9 ... output port for ID 15 output port for ID 16 output port for ID 17 output port for ID 18 output port for ID 19 ...
wasted entries
wasted entry
(b)
Fig. 1. Mapping of virtual addresses to a linear forwarding table. (a) LMC=3 for all HCAs and (b) different LMC values for each HCA.
the case of processor nodes and TCA in the case of I/O devices)1 . Each channel adapter contains one or more ports. Each port has a unique local identifier (LID) assigned by the subnet manager which is used by the subnet to forward packets towards their destination (the destination local identifier or DLID). Routing in IBA subnets is distributed, based on forwarding tables stored in each switch, which only consider the packet DLID for routing [3]. IBA routing is deterministic since the routing tables only provide one output port per DLID. Forwarding tables in InfiniBand can be linear or random. In a linear table, the DLID of the incoming packet is used as an index into the table in order to obtain the output port. On the other hand, random tables are content-addressable memories. Thus, every entry in the table contains two fields: the DLID and the output port to be used. Clearly, the hardware cost of a linear table is lower than for a random table, as the access to the table is straightforward. In the random tables, additional logic must be used to compare the DLID of the packet within all the DLIDs stored in the table. The logic complexity increases with table size, leading to high access times and could prevent the use of large forwarding tables. Hence, from our point of view, linear tables will be the preferred choice for manufacturers. In this paper, we will focus on linear forwarding tables. Additionally, IBA provides multiple virtual ports within a physical HCA port by defining a LID Mask Control or LMC [3]. The LMC specifies the number of least significant bits of the DLID that a physical port masks (ignores) when it validates that a packet DLID matches its assigned LID. Up to seven bits can be masked (3-bit field). As these bits are not ignored by the switches, from the subnet point of view, each HCA port has been assigned not a single address but a valid range of addresses (up to 2LM C consecutive addresses). Moreover, each HCA port will accept all packets destined for any valid address within its range. This IBA feature is originally intended to provide alternative routing paths within a subnet. The sender HCA port selects one of them by choosing the 1
In what follows, we will use the term HCA port to refer to HCA ports and TCA ports indistinctly.
Low-Fragmentation Mapping Strategies for Linear Forwarding Tables
949
appropriate DLID when injecting a packet into the network. Additionally, this feature can also be used for other purposes. For instance, in [6] we proposed the Destination Renaming technique, which uses the virtual addressing feature to allow the implementation on IBA of any routing that take into account the input port for routing. The Destination Renaming technique uses a new virtual address for a particular destination HCA port whenever it encounters a routing conflict (two different routing options at the same switch leading to the same destination HCA port through different output ports). Moreover, in [4,5] virtual addressing is used to allow distributed adaptive routing in InfiniBand switches. By assigning several addresses to the same HCA port, several output ports are supplied by the forwarding tables. The virtual addresses generated by the subnet manager for each HCA port must be mapped to the forwarding table of every switch. Figure 1.a shows the mapping of virtual addresses to a linear forwarding table when the same value of LMC is used by all HCA ports (LMC=3). In this case, the mapping is quite simple. As the LMC value is equal for all the HCA ports, all the ranges of virtual addresses for each HCA port are placed sequentially, starting at an entry whose index has its least LMC bits set to zero (entries 0, 8, 16, and so on). However, all the HCA ports may not require the same number of virtual addresses. For instance, if HCA 0 in Figure 1.a only needed 2 virtual addresses, then DLIDs 2 through 7 would neither be used by HCA 0 nor by any other HCA port. To solve this problem, InfiniBand specifications state that each HCA port can be programmed with its own LMC value. Therefore, each HCA port adjusts its LMC value to the number of required virtual ports. Figure 1.b shows an example where each HCA port uses a different LMC value.
2
Motivation
In order to justify the motivation of this paper, in this Section we analyze what problems may arise when different LMC values are used by the HCA ports. As the least LMC bits of the DLID are ignored by the HCA port, the range of virtual addresses for a particular HCA port must start at an address which has all its least LMC significant bits set to zero. Care must be taken in order to assign addresses to each port. Otherwise, gaps between two range of addresses for consecutive HCA ports may be created, leading to waste some entries of the forwarding table. As an example, Figure 1.b shows a straightforward mapping of the virtual addresses of different HCA ports with different LMC values to the forwarding table. The mapping is performed sequentially and searching in the forwarding table for the first suitable range of entries for each HCA port. As we can see, the first HCA port has a LMC value of 1, thus requiring only 2 virtual addresses. Therefore, the first two entries of the table are used to map the two virtual addresses of HCA 0 (DLID 0 and 1). However, as HCA 1 has a LMC value of 3, it needs a range of addresses that starts with the three least significant bits set to zero. The first available range of entries with such property is the range from DLIDs 8 through 15. Therefore, a gap has been created between the entry
950
P. L´ opez, J. Flich, and A. Robles
for DLID 1 and the entry for DLID 8. Moreover, the entry for DLID 17 is also wasted, because HCA 2 only needs one table entry (LMC=0), whereas HCA 3 needs two entries (LMC=1). As a result, in our example, in order to map 13 virtual addresses for 4 HCA ports, 20 entries in the forwarding table have been used, 7 of them being wasted. In particular (50% more entries than the strictly needed are used). We say that the forwarding table is fragmented. The fragmentation of the forwarding table may limit the number of HCAs we can connect in the subnet, and thus, it may limit the system scalability. If linear tables are implemented with full capacity for all the possible DLIDs, 49152 DLIDs can be used2 . In this situation, when 8 HCA ports per host are used and an average of 4 virtual addresses (LMC=2) per HCA port are needed, the system theoretically can support up to 1536 hosts. However, if a 35% of fragmentation was produced, the number of hosts that the system can support would drop to 998 hosts. In a more realistic scenario, IBA switches are shipped with smaller linear tables. In these situations, the fragmentation effect on forwarding tables could seriously affect system scalability. The fragmentation will depend on the way mapping is performed. For instance, in Figure 1.b, if the mapping order of HCA ports is changed to HCA 1, HCA 0, HCA 3, and HCA 2, then no fragmentation is produced at all. In all cases, the least LMC significant bits of the starting DLID are set to zero. Therefore, by using a simple ordering of HCA ports, we can eliminate the fragmentation. However, another situation can occur that will cause fragmentation even when using the previous ordered mapping. It occurs when one or some HCA hosts require a number of virtual addresses that is not a power of 2. Therefore, the LMC value used for that particular HCA port will cover all the needed virtual addresses plus a few more virtual addresses that are not really needed by the HCA port. Also, the fragmentation will depend on the required number of virtual addresses per port. This number will also depend on the applied routing algorithm, the network size and the network topology. Therefore, it is worth analyzing the impact on forwarding table fragmentation when different network configurations are used and different requirements of virtual addresses are imposed. In this paper, we try to maximize the utilization of the forwarding tables by proposing two mapping strategies that will reduce the fragmentation effect in linear forwarding tables. The first one will be referred to as DLOM (Decreasing LMC-Ordered Mapping) and will eliminate the fragmentation effect in the case all the virtual addresses covered by the LMC values are used (i.e. if the number of virtual addresses required by each HCA port is a power of two). The second one will be referred to as FF (First Fit) and can eliminate all the possible fragmentation effect even if some virtual addresses covered by the LMC value are not really used by a particular HCA port (i.e. if the number of virtual addresses required by some HCA ports is not a power of two). 2
Although 65536 DLIDs can be used, only 49152 DLIDs are available for unicast routing, bounding the number of HCA ports to 49152.
Low-Fragmentation Mapping Strategies for Linear Forwarding Tables
951
Mapping LMCa bits HCA port A (Pa ) LMCa
VA a
2
0−3
xxxxxxxxxxxxxx00 SDLID a ... xxxxxxxxxxxxxx11 EDLID a
gap (when
HCA port B (Pb) LMCb 3
)
xxxxxxxxxxxxx111 SDLID b− 1 xxxxxxxxxxxxx000 SDLID b ...
VA b 0−7
xxxxxxxxxxxxx111 EDLID b LMC bits b
Fig. 2. DLOM strategy.
The rest of the paper is organized as follows. In Section 3, the proposed mapping strategies are presented. In Section 4 an analysis of the fragmentation caused by each mapping strategy is performed. Finally, in Section 5, some conclusions are drawn.
3
Mapping Strategies
The first mapping strategy will be referred to as DLOM strategy. It stands for Decreasing LMC-Ordered Mapping. As its name states, it is based on a previous ordering of HCA ports according to their LMC values. This strategy is able to fully eliminate fragmentation when all the virtual addresses covered by the LMC value are used by the HCA port. The key idea of the strategy is shown in Figure 2. A gap in the forwarding table may be created between two consecutively mapped sets of virtual addresses (V Aa and V Ab ) which correspond to two HCA ports (Pa and Pb , respectively) when their LMC values (LM Ca and LM Cb , respectively) are different. The mapping of V Aa should be performed with a starting DLID (SDLIDa ) whose least LM Ca significant bits are set to zero. Also, V Ab should be mapped with a starting DLID (SDLIDb ) whose least LM Cb significant bits are set to zero. Thus, the address immediately before SDLIDb (SDLIDb -1) will have its least LM Cb significant bits set to ones. Also, the ending DLID of Va (EDLIDa ) will have its least LM Ca significant bits set to ones because all the virtual addresses covered by the LMC value are used. No gap will be produced between both set of virtual addresses if the EDLIDa has its least LM Cb bits set to ones. A sufficient condition to ensure this is that LM Ca >= LM Cb . Effectively, if LM Ca < LM Cb (as in Figure 2), then the (LM Cb − LM Ca ) bits after the LM Ca least significant bits may be zeros depending on the location where the V Aa addresses have been mapped.
952
P. L´ opez, J. Flich, and A. Robles
Forwarding Table
HCA Virtual addr 0 3 1 5
LMC 2 3
2 3 4
2 1 1
1 −
5 6
5 2
3 1
−
wasted entry
(a)
00000 00001 00010 00011 00100 00101 00110 00111 01000 01001 01010 01011 01100 01101 01110 01111 10000 10001 10010 10100 10101 10110 10111 11000 11001 11010 11011
Forwarding Table
HCA Virtual addr 0 3 1 2 3
5 2 1
4 5 6
1 5 2
LMC 2 3 1 − −
3 1
Available to mappings with LMC=0
Available to other mappings
00000 00001 00010 00011 00100 00101 00110 00111 01000 01001 01010 01011 01100 01101 01110 01111 10000 10001 10010 10100 10101 10110 10111 11000 11001 11010 11011
(b)
Fig. 3. (a) DLOM fragmentation. (b) FF technique.
Therefore, the DLOM strategy ensures that, for all the pairs of HCA ports mapped consecutively, Pa and Pb , it will be accomplished that LM Ca >= LM Cb . This can be achieved by sorting the HCA ports in decreasing orders of their LMCs before mapping. As previously commented, there can be cases where an HCA port does not need to use all the addresses that can be covered by its LMC value. Remember that a given LMC comprises 2LM C addresses. In these cases, the DLOM mapping strategy does not guarantee a full defragmented table, as shown in Figure 3.a. In order to avoid wasting addresses, a second mapping strategy is proposed, which is referred to as First Fit (FF) strategy. Actually, this strategy will try to reuse the addresses that are not needed by a particular HCA port which have been reserved, though, due to its LMC value. Unlike the DLOM strategy, the FF strategy does not need to previously sort the HCA ports by their LMC values. The FF strategy will reserve a valid range of DLID addresses to a particular HCA port from the first range of DLIDs not previously assigned. In other words, for each HCA port, it will search for the first suitable fit. Figure 3.b shows an example of the FF strategy. The first HCA port (HCA 0) needs only three virtual addresses. The FF strategy reserves DLIDs 0 through 2 for HCA 0. The next HCA port (HCA 1) needs 5 virtual addresses (LMC=3). The next range of addresses that starts with its three least significant bits set to zero is DLID 8 through 12. Therefore, a free range has been created from DLID 3 through DLID 7. The next HCA port (HCA 2) needs only 2 virtual addresses. The FF strategy chooses a valid range in the non-reserved zone. In particular, DLID 4 through DLID 5. The next two HCA ports (HCA 3 and HCA 4) need only one virtual address each one. The FF strategy uses two non-assigned DLIDs in the non-reserved zone. In particular, DLIDs 3 and 6 are chosen for HCA 3
Low-Fragmentation Mapping Strategies for Linear Forwarding Tables
953
Table 1. Renumbering HCAports after mapping. Input data DLOM Renumbering HCA port LMC Mapping order Starting DLID Ending DLID New HCA port 0 2 2 8 11 8 1 3 1 0 7 0 2 1 4 16 17 16 3 2 3 12 15 12
and HCA 4, respectively. Notice that after mapping the seven HCA ports, only two virtual addresses will be left unassigned. As more HCA ports were mapped, those requiring only one virtual address would be mapped to these locations, thus leading to a fragmented-free mapping strategy. A possible problem of the FF strategy is that it creates some DLIDs that are recognized as valid by several HCA ports. For instance, in Figure 3.b, DLID address 14 is a valid one for both HCA 1 and HCA 6 ports. Remember that a given HCA accepts packets destined to any address within its 2LM C assigned ones. Hence, HCA 1 will accept any packet destined to DLIDs 8 to 15. This should not be a problem because DLID address 14 is really assigned to HCA 6, and, thus, forwarding tables have been programmed to forward packets whose DLID is 14 to HCA 6. But, if due to some network routing errors, a packet with DLID 14 arrives to HCA 1, it will be accepted, because it is in the range defined by its LMC. Unfortunately, the FF strategy could not be applied according to the current IBA specifications (1.0.a) because the subnet manager does not allow overlapping ranges of DLIDs. Once the mapping is performed, each HCA port has an assigned range of DLIDs for its use. However, in order to each HCA port to use them correctly, the HCA port identifier must be set to the starting DLID. This is because the LMC bits of the packet DLID will be masked at destination to match the destination HCA port. Therefore, once the mapping is computed, the HCA port identifiers must be renumbered. Notice that the renumbering of HCA ports must be performed regardless of the applied mapping function. In Table 1 we can see an example of network with four HCA ports that have been mapped to the forwarding tables using the DLOM strategy. Firstly, the HCA ports are sorted by their LMC values. After sorting the HCA ports, the mapping is performed by assigning starting and ending DLIDs. As can be observed, for the original HCA port 1, its first address will be zero. Therefore, the HCA port must be renumbered to zero. The rest of HCA ports are renumbered in the same way according to the starting assigned address.
4
Evaluation of the Proposed Mapping Strategies
In this section, we evaluate the DLOM and FF strategies presented in this paper. For comparison purposes, we will evaluate the straightforward mapping strat-
954
P. L´ opez, J. Flich, and A. Robles 40000
40000 35000 ’Min’ ’Max LMC’ ’No mapping’ ’FF’ ’DLOM’
30000 25000
Table size (entries)
Table size (entries)
35000
20000 15000
25000 20000 15000
10000
10000
5000
5000
0
’Min’ ’Max LMC’ ’No mapping’ ’FF’ ’DLOM’
30000
0 0
20 40 60 80 100 120 140 Maximum number of virtual addresses per HCA port
(a)
0
20 40 60 80 100 120 140 Maximum number of virtual addresses per HCA port
(b)
Fig. 4. Number of forwarding table entries required for different number of virtual addresses required. 256 hosts. In (a) 50% of hosts require more than one virtual address and in (b) all the hosts require more than one virtual address.
egy consisting of setting the LMC values of all the HCA ports to the highest LMC value (LM Cmax ) used for a particular HCA port (the easiest mapping strategy). With this mapping strategy, a high fragmentation is introduced as a large number of addresses is wasted. For a given HCA port with a required LMC, up to 2LM Cmax −LM C addresses are wasted. This strategy will be referred to as M axLM C. We will also compare the strategies with the basic sequential (no ordering) mapping consisting of selecting the next valid range of free addresses. This strategy will be referred to as No Mapping. In addition, results for an optimal strategy that assigns the minimum number of addresses to HCA ports without any fragmentation is evaluated. The number of required addresses is simply the sum of the virtual addresses required by all the HCA ports. It will be referred to as Min. In a first study, we will analyze the impact of the applied mapping strategy used on the fragmentation of the forwarding tables under different scenarios. These scenarios will be defined by the number of HCA ports in the system, the maximum possible number of virtual addresses required by any HCA port and the percentage of HCAs requiring more than one virtual address. In particular, we will analyze networks with 256 HCA ports. Regarding the percentage of HCA ports requiring more than one virtual address, we will show results for 50% and 100%. Finally, for each HCA port that requires more than one virtual address, we assign a number randomly selected between 2 and a maximum number. We will vary the maximum number of virtual addresses required by a particular HCA from 2 up to 128 (the upper bound allowed by IBA). If virtual addresses are used to supply alternative paths to destinations, with the last two parameters we model the number of such paths. Notice that a large number of virtual addresses per HCA port and a high percentage of the number of HCA ports requiring more than one virtual address models a system with a high connectivity. In the analysis, we will measure fragmentation as the percentage of wasted entries relative to the minimum number of entries required in the table. In other words,
Low-Fragmentation Mapping Strategies for Linear Forwarding Tables
955
fragmentation keeps track of the additional useless entries in the forwarding tables. Figure 4 shows the number of entries in the forwarding tables when different mapping strategies are used. In particular, Figure 4.a shows the number of entries when the percentage of HCA ports requiring more than one virtual address is 50%, whereas Figure 4.b show the results when all hosts use more than one virtual address. As can be seen, when no mapping strategy is used (No mapping) more than 60% of fragmentation is caused (assuming high connectivity). The easiest mapping strategy (Max LMC) exacerbates the fragmentation as it uses the maximum LMC used in the network for all HCA ports. On average, this strategy increases fragmentation up to 300% (≈33000 vs. ≈8000 in Figure 4.a). As we can see, the fragmentation is reduced when the DLOM strategy is used. In particular, the fragmentation percentage introduced by DLOM decreases down to 30% in the worst case (Figure 4.a). Remember that the only fragmentation incurred by the DLOM strategy is due to the virtual addresses not used by HCAs but covered by the LMC values (i.e., when the required number of virtual addresses is not a power of two). This can be verified by the less fragmentation achieved by the FF strategy. As it can be observed, the FF strategy introduces no fragmentation in any case as it uses the minimum number of required entries. In a second study, we will obtain results for topologies with different number of HCA ports. In particular, we will analyze irregular networks of 8, 12, 16, 24, and 32 switches randomly generated, taking into account some restrictions. First, we will assume that every switch in the network has the same degree. Four ports at each switch will be used to connect to HCA ports. The remaining ports will be used to connect to other switches. Switches with 7, 8, and 10 ports will be used. Second, each switch will be connected to other switch by exactly one link. Five different topologies will be generated for each network size and average results will be shown. The applied routing algorithm is the smart-routing [1] and only one path for each pair of source-destination ports will be used. The destination renaming technique proposed in [6] will be used to correctly compute the forwarding tables, which leads to the need of some virtual addresses per each HCA port. Notice that if several paths per each source-destination pair of hosts were allowed, more virtual addresses would be required. Table 2 shows the average number of required entries and the average percentage of incurred fragmentation when using smart-routing. Also, results for different network connectivities are shown (different number of links connecting switches). As it can be observed, the FF mapping strategy is the only one that introduces no fragmentation in all the evaluated topologies. Also, it can be observed that the DLOM strategy does not achieve a very high reduction in the fragmentation when compared with No mapping. This is because most of the fragmentation is caused by virtual addresses not used by HCA ports but covered by their LMC value. Anyway, DLOM incurs in lower fragmentation than No Mapping in all cases.
956
P. L´ opez, J. Flich, and A. Robles
Table 2. Average fragmentation of forwarding tables when using smart-routing under different topologies and network connectivity. Sw Links Min entries 8 3 36.00 8 4 68.20 8 6 98.40 12 3 74.40 12 4 85.00 12 6 108.00 16 3 92.20 16 4 104.60 16 6 139.00 24 3 194.60 24 4 244.20 24 6 255.80 32 3 282.00 32 4 323.40 32 6 456.47
5
Max LMC entries % 44.80 24.44 128.00 87.68 128.00 30.08 134.40 80.65 153.60 80.71 192.00 77.78 204.80 122.13 204.80 95.79 256.00 84.17 384.00 97.33 537.60 120.15 537.60 110.16 512.00 80.92 512.00 58.32 682.67 48.90
FF entries % 36.00 0.00 68.20 0.00 98.40 0.00 74.40 0.00 85.00 0.00 108.00 0.00 92.20 0.00 104.60 0.00 139.00 0.00 194.60 0.00 244.20 0.00 255.80 0.00 283.00 0.00 323.40 0.00 458.47 0.00
No mapping entries % 36.00 0.00 78.00 14.37 121.60 23.58 78.00 4.84 86.40 1.65 124.40 15.19 96.00 4.12 110.60 5.74 157.60 13.38 214.00 9.97 293.20 20.07 316.40 23.69 329.60 16.47 389.60 20.47 563.87 22.99
DLOM entries % 36.00 0.00 75.20 10.26 121.60 23.58 77.40 4.03 86.20 1.41 121.00 12.04 95.40 3.47 109.80 4.97 153.40 10.36 209.40 7.61 286.20 17.20 303.00 18.45 319.60 12.93 380.20 17.56 550.07 19.98
Conclusions
IBA specifications allow each HCA port to be assigned a number of virtual addresses or LIDs, which must be mapped to a range of consecutive entries in the forwarding tables. However, the use of an inappropriate mapping strategy could lead to waste some table entries, causing fragmentation in the forwarding tables. This fact becomes critical as far as it reduces the number of HCA ports that can be placed in the network and/or avoids the implementation of forwarding tables on switches with a given table size. In this paper, we have proposed two effective mapping strategies to eliminate the fragmentation effect in linear forwarding tables. The DLOM strategy is based on a previous ordering of the HCA ports according to their number of required virtual addresses. However, this strategy only eliminates the fragmentation effect when the number of virtual addresses required by each HCA port is a power of two. To overcome this problem, we have proposed the FF strategy, which is able to eliminate the fragmentation effect in all cases. This strategy will reserve a valid range of LID addresses to a particular HCA port from the first range of LIDs not previously assigned, trying to reuse the addresses not needed by a particular HCA port which, otherwise, would be reserved according to the LMC. The evaluation of the proposed mapping strategies shows that their impact on the reduction in the fragmentation of the forwarding tables is more significant as far as the network size increases and a high number of virtual addresses are required by each HCA port. In particular, the fragmentation percentage introduced by DLOM decreases down to 20% whereas no fragmentation is introduced by applying the FF strategy. Moreover, when using smart routing under different topologies and network connectivities, the average fragmentation of the
Low-Fragmentation Mapping Strategies for Linear Forwarding Tables
957
forwarding tables introduced by using DLOM is always lower than that introduced when no mapping strategy is applied. Again, in all cases, no fragmentation is introduced by applying the FF strategy.
References 1. L. Cherkasova, V. Kotov, and T. Rokicki, “Fibre channel fabrics: Evaluation and design,” in Proc. of 29th International Conference on System Sciences, Feb. 1995. 2. InfiniBandTM Trade Association. http://www.infinibandta.com. 3. InfiniBandTM Trade Association, InfiniBand T M Architecture. Specification Volume 1. Release 1.0.a. Available at http://www.infinibandta.com. 4. J.C. Martinez, J. Flich, A. Robles, P. L´ opez, and J. Duato, “Supporting Adaptive Routing in InfiniBand Networks,” in 11th Euromicro Workshop in Parallel Distributed and Network-Based Processing, February, 2003. 5. J.C. Martinez, J. Flich, A. Robles, P. L´ opez, and J. Duato, “Supporting Fully Adaptive Routing in InfiniBand Networks,” in Proc. of the International Parallel and Distributed Processing Symposium, April 2003. 6. P. L´ opez, J. Flich, and J. Duato, Deadlock-free Routing in InfiniBandT M through Destination Renaming, in Proc. of 2001 International Conference on Parallel Processing (ICPP’01), Sept. 2001.
A Robust Mechanism for Congestion Control: INC Elvira Baydal and Pedro L´opez Dept. of Computer Engineering (DISCA) Universidad Polit´ecnica de Valencia Camino de Vera s/n, 46071 – Valencia, SPAIN elvira,
[email protected] Abstract. Several techniques to prevent congestion in multiprocessor interconnection networks have been recently proposed. Unfortunately, they either suffer from a lack of robustness or detect congestion relying on global information that wastes a lot of transmission resources. This paper presents a new mechanism that uses only local information to avoid network saturation in wormhole networks. It is robust and works properly in different conditions. It first applies preventive measures of different intensity depending on the estimated traffic level; and if necessary, it uses message throttling during predefined time intervals that are extended if congestion is repeatedly detected. Evaluation results for different network loads and topologies show that the proposed mechanism avoids network performance degradation. Most important, without introducing any penalty for low and medium network loads.
1
Introduction
Massively parallel computers provide the performance that scientific and commercial applications require. Their interconnection networks offer the low latencies and high bandwidth that is needed for different kinds of traffic. Usually, wormhole switching with virtual channels and adaptive routing is used. However, multiprocessor interconnection networks may suffer from severe saturation problems with high traffic loads, which may prevent reaching the wished performance (see Figure 1). With low and medium network loads, the accepted traffic rate is the same as the injection rate. But if traffic increases and reaches (or surpasses) certain level (the saturation point), accepted traffic falls and message latency increases considerably. Performance degradation appears because with high network traffic several packets compete for the same resources (physical or virtual channels), but as only one packet can use them, the rest of the packets stay stopped in the network, thus blocking other packets and so on. When this situation is generalized, the network is congested and performance degradation appears. One of the most popular approaches to solve this problem is to avoid that network traffic reaches the saturation point. Message throttling [4] (stop or slow down the injection rate of messages) has been the most frequently used method. Several mechanisms for multiprocessors have already been proposed but they have important drawbacks [2].
This work was supported by by the Spanish MCYT under Grants TIC2000-1151-C07 and 1FD97-2129
H. Kosch, L. B¨osz¨orm´enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 958–968, 2003. c Springer-Verlag Berlin Heidelberg 2003
Accepted Traffic (flits/node/cycle)
Latency since generation (cycles)
A Robust Mechanism for Congestion Control: INC
Uniform Complement
10000
1000
100 0.1
0.2
0.3
0.4
0.5
0.6
Injected Traffic (flits/node/cycle)
0.7
959
0.7 0.6 0.5
Uniform Complement
0.4 0.3 0.2 0.1 0.1
0.2
0.3
0.4
0.5
0.6
0.7
Injected Traffic (flits/node/cycle)
Fig. 1. Performance degradation in a 3-cube 8-ary (512 nodes) with a deadlock recovery-based fully adaptive routing algorithm with 3 virtual channels per physical channel. Uniform and complement traffic patterns. 16-flit messages.
In this paper, we propose and evaluate a new mechanism to avoid network saturation that tries to overcome these drawbacks. It is a robust mechanism that applies preventive measures of different intensity depending on the network traffic. Moreover, it progressively reduces the bandwidth available to inject messages, either by reducing the number of injection channels or, if it is necessary, by applying increasing intervals of forbidden message injection. The mechanism has been exhaustively tested, achieving good results in all the evaluated conditions and improving recent proposals [2], [15]. The rest of the paper is organized as follows. In order to design a good congestion control mechanism, Section 2 presents the main desirable features of such a mechanism. Section 3 describes the new proposal. Section 4 presents some simulation results. Finally, some conclusions are obtained.
2
Motivation
First, the mechanism should work properly for different conditions (robustness): different message destination distributions, message sizes, network topologies and even routing algorithms. As we can see in Figure 1, the network saturation point depends on the traffic pattern. Hence, a suitable mechanism for one distribution, may not work properly with another one. The same problem may appear when we change network topology. However, many of the previously proposed mechanisms have been analyzed with only one network size [1], [15] and for the uniform distribution of message destinations [4], [8], [13]. Other mechanisms do not achieve good results all traffic patterns considered [14]. On the other hand, some mechanisms are based on the idea that all the virtual channels of a physical channel are used in the same way [1], [2], thus being well suited for deadlock recovery schemes but may not for deadlock avoidance ones. Second, the mechanism should not penalize the network when it is not saturated. Notice that this is the most frequent situation [12]. Thus, when with low and medium network traffic, the mechanism should not restrict or delay message injection into the network. However, some of the previous proposals increase message latency before the saturation point [4], [14], also reducing the network throughput.
960
E. Baydal and P. L´opez
Finally, the new mechanism should not generate new problems in the network. Those proposals that need to send extra information across the network increase network traffic and may make worse the congestion situation [8]. Moreover, when congestion detection relies on global information, mechanisms do not scale well since the amount of network status information increases with the network size [15].
3 The New Mechanism: Injection and Network Congestion As we stated in Section 1, when network traffic is near to the saturation point, many packets block. Indeed, as wormhole switching is used, these packets stay stopped in the network, thus blocking other packets. In addition, as physical channels are multiplexed into more virtual channels, message advance speed decreases. The new mechanism uses message advance speed in order to detect network congestion. In particular, it measures the number of transmitted flits during a fixed time interval tm . Each virtual channel has an associated counter that increases each time one flit is transmitted. At the end of each interval, the channel is considered congested if it is busy and the number of transmitted flits is lower than a threshold fc (flit counter). When the header of a message reaches a congested area, either it will block or advance at a low speed. As a consequence, the remaining flits of the message will also block or decrease their advance speed. By tracking this speed, we try to detect congestion in network areas remote to the current node, but using only local information. Once congestion is detected, the applied measures are different depending on whether the node is currently injecting messages towards the congested area. In particular, each node manage two different flags: the injection and the network flags. The injection flag is set when some of the virtual channels that the node is using to inject messages into the network are congested. On the contrary, the network flag is set when congestion is detected in any virtual channel that is being used by messages injected by other nodes. The mechanism will referred to as INC, Injection and Network Congestion detection. When either the injection or network flags are set, message injection restrictions are applied. These measures are more restrictive when the injection flag is set than when only network flag does. The injection flag setting means that the node is actively contributing to network congestion. On the contrary, when only the network flag is set, although the current node is not directly generating the problem, as congestion propagates quickly through the network, some preventive measures should be applied. In both cases, if congestion is repeatedly detected in later intervals, the measures get more restrictive. Thus, the mechanism works in a progressive way. In particular, when injection congestion is detected, the number of enabled injection channels is divided (integer division) by a factor r1 . If this quotient reaches zero, injection is completely stopped but only during some interval fii (forbidden injection interval). This interval starts when the last message that the node is currently injecting is completely injected into the network. After fii cycles, one injection channel is enabled regardless of the detected network traffic, and the first pending message, if any, is injected. As network traffic is estimated periodically, injection restrictions get harder or, on the contrary, are reduced. The mechanism uses limited forbidden injection intervals for two reasons. First, it avoids starvation and, second, it allows that other nodes also detect congestion and
A Robust Mechanism for Congestion Control: INC injCh : Enabled injection channels AllinjC: Maximum number of injection channels
Wait tm cycles
Yes
961
Injection congestion? No
injCh := injCh/r1
Network congestion? No Yes
fii > fii_min?
No
injCh := injCh/r2
injCh = 0?
No
Yes
Yes
injCh = 0? No
fii := fii + inc_inj
Yes
Wait until the last message is injected
fii := fii + inc_net Yes
fii = fii−inc_net If injCh < AllInjCh injCh := injCh + 1
Injecting messages? No
Wait fii cycles Inject pending message
Fig. 2. Operation of the congestion control mechanism INC.
apply message injection limitation. If only network congestion is detected, the enabled injection channels are reduced by r2 , with r2 < r1 . Forbidden injection intervals are also used if network congestion is repeatedly detected, but they increase more slowly. The values of r1 and r2 depend on the network radix k. Notice that when k increases also does the average distance traversed by the messages, which makes worse network congestion. Therefore injection restrictions have to be harder. After some tests, we have found that r1 = k/4 and r2 = k/8 work well, k being the network radix. The forbidden injection interval is initialized to some value fiimin . Every time congestion is detected, it is incremented. This increment again depends on the network radix k. We have chosen it higher for injection than for network congestion. After several tests, we have found that values incinj = (fiimin × k)/16 and incnet = (fiimin × k)/32 work well, for injection and network congestion, respectively. Finally, injection bandwidth has to be recovered when congestion it not longer detected. However, injection restrictions are also reduced gradually. After an interval without detecting any congestion, injection limitation is smoothed. First, injection bandwidth is increased by reducing the forbidden interval (by incnet ), and later (when it reaches the minimum value fiimin ), by increasing (by one channel at a time) the number of injection channels. Notice that the reduction of injection bandwidth is faster than the later recovering. Figure 2 summarizes the behavior of the mechanism.
4
Evaluation
In this section, we will evaluate by simulation the behavior of the INC mechanism. The evaluation methodology is based on the one proposed in [2]. The most important performance measures are latency (time required to deliver a message, including the time spent at the source queue, measured in clock cycles) and throughput (maximum traffic
962
E. Baydal and P. L´opez
accepted by the network, measured in flits per node per cycle). On the other hand, as message latency at saturation grows with time, another valuable measure of performance is the required time to deliver a given number of messages. 4.1
Network Model
Each node has a router, a crossbar switch and several physical channels. Routing time and both transmission time across the crossbar and across a channel are all assumed to be equal to one clock cycle. Each node has four injection/ejection channels. We explained the advantages of having several injection/ejection channels in [2]. Concerning deadlock handling, we have evaluated the new mechanism both with deadlock avoidance and recovery techniques. For deadlock recovery, we use softwarebased deadlock recovery [11] and a True Fully Adaptive routing algorithm (TFAR) [12,11]. This routing algorithm allows the use of any virtual channel of those physical channels that forwards a message closer to its destination. Deadlocks are detected with the mechanism proposed in [10] with a threshold equal to 32 cycles. With deadlock avoidance, we use the fully adaptive routing algorithm proposed in [6]. In both cases, we have tested the mechanism with 3 and 4 virtual channels per physical channel. We have evaluated the performance of the proposed congestion control mechanism on different bidirectional k-ary n-cubes. In particular, we have used the following network sizes: 256 nodes (n=2, k=16), 1024 nodes (n=2, k=32), 512 nodes (n=3, k=8), and 4096 nodes (n=3, k=16). 4.2
Network Load
Each node generates messages independently, according to an exponential distribution. Destinations are chosen according to the Uniform, Uniform with locality, Butterfly, Complement, Bit-reversal, and Perfect-shuffle traffic patterns. Uniform distribution is the most frequently used one in the analysis of interconnection networks. The other patterns take into account the permutations that are usually performed in parallel numerical algorithms [9]. For message length, 16-flit, 64-flit and 128-flit messages are considered. We have performed two kinds of experiments, using constant and dynamic loads. Experiments with constant loads are the usual evaluation tool for interconnection networks. In this case, the message generation rate is constant and the same for all network nodes. We analyze all the traffic range, from low load until saturation. Simulations finish after receiving 500,000 messages with the smallest networks (256 and 512 nodes) and 1,200,000 with the largest ones (1024 and 4096 nodes), but only the last 300,000 (1,000,000) messages are considered to calculate average latencies. We will also evaluate the mechanism behavior when the traffic conditions change dynamically. In particular, we use a bursty load that alternates periods of a high message injection rate Lsat (that corresponds to a saturation interval), with periods of low traffic Llow . These loads are repeated twice (Lsat Llow Lsat Llow ). Let Ls be the message generation rate just before the network reaches saturation. We have chosen a generation rate Llow = 0.5 × Ls . On the other hand, in order to analyze the mechanism behavior at different saturation levels, we have tested two different values of the saturation load: Lsat = 1.2 × Ls and Lsat = 1.8 × Ls . Concerning the number of sent messages, for
A Robust Mechanism for Congestion Control: INC
963
a given generation rate, each node generates the same number of messages. When a node has generated all the messages for one of the injection rates, then, it starts generating traffic with the next load level in sequence. Finally, when it has generated all the messages for the complete sequence, it does not generate new messages at all. The simulation finishes when all the injected messages arrive to their destinations. We show results with 1, 000 messages per node generated with the Lsat injection rate, and two different values for the load under the saturation (Llow ): Mlow = 555 and Mlow = 1111 messages. The number of messages generated with the Llow rate establishes the elapsed time between two traffic bursts, and it allows us to analyze how the mechanism recovers the network after a congestion period. 4.3
Performance Comparison
In this section, we will analyze the behavior of the INC mechanism proposed in this paper. For comparison purposes, we will also evaluate the U-channels ([2]) and SelfTuned ([15]) mechanisms. U-channels also uses only local information to avoid network saturation. Each node locally estimates traffic by using the percentage of free virtual output channels that can be used to forward a message towards its destination. When this number is below a threshold value, congestion network is assumed to exist and message throttling is applied. In the Self-Tuned mechanism, nodes detect network congestion by using global information about the number of full buffers in the network. If this number surpasses a threshold, all nodes apply message throttling. The use of global information requires to broadcast data among all the network nodes. A way of transmitting this control information is to use a sideband network [15]. As the mechanism proposed in this paper does not need to exchange control messages, to make a fair comparison, the bandwidth provided by the sideband network should be considered as additional available bandwidth in the main interconnection network. However, in the results that we present we do not consider this fact. If this additional bandwidth were considered, the differences, not only in throughput but also in latency, between Self-Tuned and the rest of the mechanisms would be greater than the ones shown. As it was shown in [2], U-channels outperforms Self-Tuned with deadlock recoverybased TFAR routing, but it is not well suited for deadlock avoidance since, in this case, all the virtual channels are not used in the same way. For this reason, we compare INC with U-channels when TFAR routing is used, while we use Self-Tuned as the comparison with deadlock avoidance. Finally, in all cases, results without any congestion control mechanism (No-Lim) are also shown. First of all, the INC mechanism has to be tuned. We have evaluated its behavior with different message destination distributions, message sizes and network topologies. In all cases, the number of transmitted flits per virtual channel is sampled every tm = 20 cycles. As an example, Figure 3 shows the average message latency versus traffic for different threshold values for the uniform and perfect-shuffle traffic patterns with the TFAR routing algorithm based on deadlock recovery. Notice that both latency and accepted traffic are dependent variables on offered traffic. As you can see, threshold adjustment is not critical. In fact, several of them work fine since the mechanism tries to find the optimal injection bandwidth by applying progressive measures. Table 1 shows the values chosen for the evaluated networks, different number of virtual channels per physical channel,
964
E. Baydal and P. L´opez
Perfect-shuffle
No-lim fc1fii2 fc1fii8 fc3fii2 fc3fii8
10000
1000
100 0.05
0.1 0.15 0.2 0.25 0.3 0.35 Accepted Traffic (flits/node/cycle)
Latency since generation (cycles)
Latency since generation (cycles)
Uniform 10000
1000
100
No-lim fc1fii2 fc1fii8 fc3fii2 fc3fii8 0.03
0.06 0.09 0.12 0.15 0.18 Accepted Traffic (flits/node/cycle)
Fig. 3. Average message latency vs. accepted traffic for different threshold values for the INC mechanism. 16-ary 3-cube (4096 nodes). 3 virtual channels per physical channel. TFAR algorithm based on deadlock recovery. Table 1. Threshold values for different topologies and number of virtual channels per physical channel. Nodes Virt. ch. 256 1024 512 4096
3 4 3 4 3 4 3 4
Deadlock recovery fc fiimin incinj incnet 1 4 4 2 1 2 2 1 4 2 4 1 2 1 2 1 1 4 2 1 1 1 0 0 1 2 2 1 1 2 2 1
Deadlock avoidance fc fiimin incinj incnet 1 4 4 2 1 2 2 1 6 8 16 8 5 2 4 2 1 1 0 0 1 1 0 0 2 2 2 1 1 2 2 1
and deadlock recovery and avoidance strategies. As you can see, the optimal values depend mainly on the topology radix k and the number of virtual channels per physical channel. This is not a problem, as these are design parameters that are fixed once the machine is built. Notice that for 512 nodes and 4 virtual channels, the fii is held constant (incinj = incnet = 0). In this case, only reducing the number of injection channels is enough to avoid congestion. Once tuned the new mechanism, we can compare it with the other ones. For the sake of shortness, we will only show some results for the 4096-node network (k=16, n=3) and 3 virtual channels per physical channel. The complete evaluation for all the tested networks, traffic patterns and different number of virtual channels per physical channel can be found in [3]. Figure 4 shows these results. As we can see, the new mechanism INC avoids the performance degradation in all the evaluated cases, without increasing latencies for low and medium loads. In general, the results obtained with deadlock recovery are quite similar to the U-channels ones. On the other hand, with deadlock avoidance the INC mechanism improves in all the cases the Self-tuned. Although the Self-Tuned mechanism helps in alleviating network
A Robust Mechanism for Congestion Control: INC
Bit-reversal, Deadlock recovery
10000
1000
No-lim INC U-channels
100 0.05
0.1 0.15 0.2 0.25 0.3 0.35 Accepted Traffic (flits/node/cycle)
Latency since generation (cycles)
Latency since generation (cycles)
Uniform, Deadlock recovery
10000
1000
100
100 0.2
0.25
Accepted Traffic (flits/node/cycle)
0.3
Latency since generation (cycles)
Latency since generation (cycles)
No-Lim INC Self-Tuned
0.15
0.1 0.15 0.2 0.25 Accepted Traffic (flits/node/cycle)
Bit-reversal, Deadlock avoidance
1000
0.1
No-lim INC U-channels 0.05
Uniform, Deadlock avoidance
0.05
965
10000
1000 No-Lim INC Self-Tuned 100 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 Accepted Traffic (flits/node/cycle)
Fig. 4. Average message latency vs. accepted traffic. 16-flit messages. 8-ary 3-cube (4096 nodes).
congestion, it reduces network throughput and increases network latency with medium loads in some cases. Figure 5 shows some results for dynamic loads with a uniform distribution of message destinations, Lsat = 1.8 × Ls and Mlow = 1111 messages per node with Llow . As we can see, with the INC mechanism, the network accepts the injected bursty traffic without problems. On the contrary, when no congestion control mechanism is applied, as soon as the first burst is applied into the network, congestion appears. As a consequence, latency strongly increases and accepted traffic falls down. Later, after some time injecting with low rate, network traffic starts recovering but the arrival of a new traffic burst prevents it. Concerning Self-Tuned mechanism, we can see that it excessively limits injection rate, significantly reducing the highest value of accepted traffic, therefore increasing the time required to deliver all the injected messages. As we stated before, this time is another performance measure that is strongly affected by network saturation. Figure 5 shows that INC delivers the required number of messages in nearly half the time than No-Lim (74.36% of improvement), while Self-Tuned achieves a slightly smaller improvement (70.51%). Table 2 shows the percentage of improvement in the time required to deliver all the injected messages, for different network sizes, when we apply the INC and U-channels congestion control mechanisms versus when no congestion control mechanism is used. Deadlock recovery is used. We have got improvements in the simulation time up to 156%
E. Baydal and P. L´opez
100000
10000
No-Lim Self-tuned INC
1000
100 0e+00 1e+05 2e+05 3e+05 4e+05 5e+05 6e+05 Simulation time (cycles)
Accepted traffic (flits/node/cycle)
Latency since generation (cycles)
966
0.3 0.25
No-Lim Self-tuned INC
0.2 0.15 0.1 0.05 0e+00 1e+05 2e+05 3e+05 4e+05 5e+05 6e+05 Simulation time (cycles)
Fig. 5. Performance with dynamic loads. Uniform traffic pattern. 16-flit messages. 16-ary 3-cube (4096 nodes), 3 virtual channels. Fully adaptive algorithm with deadlock avoidance. Table 2. Improvement (%) in the time required to deliver all the injected messages for the INC and U-channels mechanisms with different topologies, TFAR algorithm with deadlock recovery and 3 virtual channels. Uniform traffic pattern. Nodes
Mechanism
256 (16 × 16) 1024 (32 × 32) 512 (8 × 8 × 8) 4096 (16 × 16 × 16)
INC U-channels INC U-channels INC U-channels INC U-channels
Lsat = 1.2 × Ts Lsat = 1.8 × Ts Mlow = 1111 Mlow = 555 Mlow = 1111 Mlow = 555 48.35 71.82 64.07 101.09 47.71 71.70 63.24 100.83 85.00 112.00 101.98 141.79 84.19 109.85 100.96 142.64 22.85 44.23 34.69 69.33 22.29 42.47 33.94 66.97 96.29 125.59 111.28 156.19 94.86 122.92 109.59 152.64
with the uniform distribution of message destination and up to 86% with the complement traffic pattern1 . In general, the results are slightly better for the INC mechanism than for the U-channels mechanism. However, as the differences between both mechanisms do not exceed 5%, we can conclude that the progressive reduction/recovery provided by INC it is not an improvement but adds complexity. Table 3 shows the same results when we use the INC and Self-tuned congestion control mechanisms versus No-Lim. Deadlock avoidance is used. In this case, the INC mechanism obtains improvements in the simulation time up to 150% versus No-Lim for a uniform distribution of message destinations, and up to a 22.5% for the complement traffic pattern1 . These improvements are far from the ones achieved by the Self-Tuned mechanism. Moreover, INC reaches this better performance without transmitting information through the network. Hence, for deadlock avoidance, INC is clearly a better option than Self-tuned.
1
This result can be found in [3]
A Robust Mechanism for Congestion Control: INC
967
Table 3. Improvement (%) in the time required to deliver all the injected messages for the INC and Self-Tuned mechanisms with different topologies, fully adaptive routing algorithm with deadlock avoidance and 3 virtual channels. Uniform traffic pattern.
5
Nodes
Mechanism
256 (16 × 16) 1024 (32 × 32) 512 (8 × 8 × 8) 4096 (16 × 16 × 16)
INC Self-tuned INC Self-tuned INC Self-tuned INC Self-tuned
Lsat = 1.2 × Ts Lsat = 1.8 × Ts Mlow = 1111 Mlow = 555 Mlow = 1111 Mlow = 555 20.27 64.94 47.93 106.69 18.04 58.16 45.16 82.40 113.43 116.00 109.26 150.15 109.44 93.26 94.59 84.77 0.00 2.33 0.00 24.49 -2.32 -0.22 -1.85 21.45 41.83 83.66 74.36 114.32 38.09 78.15 70.51 81.77
Conclusions
In this paper, we present a mechanism (INC) to avoid network saturation based on message throttling. It can be used with routing algorithms based either on deadlock recovery or avoidance schemes, and works well for different network topologies and network loads. The mechanism tries to estimate network traffic using only local information available to the node, without sending control information through the network. However, it is able to estimate the status of remote areas by measuring the number of flits transmitted by each virtual channel during a fixed interval of time. If this number is lower than a threshold, the mechanism assumes that there is congestion in that virtual channel. In this situation, a flag is set and message injection restrictions are applied. Two kind of flags are used in order to adjust the injection bandwidth, depending on whether the node is already contributing to the congestion or not yet. In the latter case, applied measures are softer (they are only preventive). Moreover, if congestion is repeatedly detected, injection may be completely forbidden during increasing time intervals. Although both the threshold and the forbidden interval time have to be empirically adjusted, we have found that, as the mechanism works in a progressive way, their adjust is not critical. The evaluation results show that the mechanism is able to avoid performance degradation for all the analyzed conditions, without introducing any penalty for low and medium network loads, when none congestion control mechanism is required. Moreover, it outperforms recent previous proposals.
References 1. E. Baydal, P. L´opez and J. Duato, “A Simple and Efficient Mechanism to Prevent Saturation in Wormhole Networks”, Int. Parallel & Distributed Processing Symp., May 2000. 2. E. Baydal, P. L´opez and J. Duato, “Avoiding Network Congestion with Local Information”, Int. Symp. High Performance Computing, May 2002.
968
E. Baydal and P. L´opez
3. E. Baydal, Una contribuci´on al control de la congesti´on en redes de interconexi´on con conmutaci´on wormhole, (in Spanish), Ph. D. Thesis, Universidad Polit´ecnica de Valencia, Nov. 2002. http://www.gap.upv.es. 4. W. J. Dally and H. Aoki, “Deadlock-Free Adaptive Routing in Multicomputer Networks Using Virtual Channels”, IEEE Trans. on Parallel and Distributed Systems, vol. 4, no. 4, pp. 466–475, April 1993. 5. J. Duato, “A new theory of deadlock-free adaptive routing in wormhole networks”, IEEE Trans. on Parallel and Distributed Systems, vol. 4, no. 12, pp. 1320–1331, Dec. 1993. 6. J. Duato and P. L´opez, “Performance evaluation of adaptive routing algorithms for k-ary n-cubes”, Parallel Computer Routing and Communication, K. Bolding and L. Snyder (ed.), Springer-Verlag, pp. 45–49, 1994. 7. J. Duato, S. Yalamanchili and L.M. Ni, Interconnection Networks: An Engineering Approach, 2nd. ed. Morgan Kauffman, 2002. 8. J. H. Kim, Z. Liu and A. A. Chien, “Compressionless routing: A framework for Adaptive and Fault-Tolerant Routing”, IEEE Trans. on Parallel and Distributed Systems, Vol. 8, No. 3, 1997. 9. F. T. Leighton, Introduction to Parallel Algorithms and Architectures: Arrays, Trees, Hypercubes, San Mateo, CA, USA, Morgan Kaufmann Publishers, 1992. 10. P. L´opez, J.M. Mart´ınez and J. Duato, “A Very Efficient Distributed Deadlock Detection Mechanism for Wormhole Networks”, Int. Symp. on High Performance Computer Architecture, Feb. 1998. 11. J.M. Mart´ınez, P. L´opez and J. Duato, “A Cost–Effective Approach to Deadlock Handling in Wormhole Networks”, IEEE Trans. on Paral. and Distrib. Processing, pp. 719–729, July 2001. 12. T.M. Pinkston and S. Warnakulasuriya, “On Deadlocks in Interconnection Networks”, Int. Symp. on Computer Architecture, June 1997. 13. A. Smai and L. Thorelli, “Global Reactive Congestion Control in Multicomputer Networks”, Int. Conf. on High Performance Computing, 1998. 14. M. Thottetodi, A.R. Lebeck, S.S. Mukherjee, “Self-Tuned Congestion Control for Multiprocessor Networks”, Technical Report CS-2000-15, Duke University, Nov. 2000. 15. M. Thottetodi, A.R. Lebeck, S.S. Mukherjee, “Self-Tuned Congestion Control for Multiprocessor Networks”, Int. Symp. on High Performance Computer Architecture, Feb. 2001.
RoCL: A Resource Oriented Communication Library Albano Alves1 , Ant´onio Pina2 , Jos´e Exposto, and Jos´e Rufino 1
Instituto Polit´ecnico de Bragan¸ca, Campus Sta. Apol´onia, 5301-857 Bragan¸ca-Portugal
[email protected] 2 Universidade do Minho
[email protected] Abstract. RoCL is a communication library that aims to exploit the low-level communication facilities of today’s cluster networking hardware and to merge, via the resource oriented paradigm, those facilities and the high-level degree of parallelism achieved on SMP systems through multi-threading. The communication model defines three major entities – contexts, resources and buffers – which permit the design of high-level solutions. A low-level distributed directory is used to support resource registering and discovering. The usefulness and applicability of RoCL is briefly addressed through a basic modelling example – the implementation of TPVM over RoCL. Performance results for Myrinet and Gigabit Ethernet, currently supported in RoCL through GM and MVIA, respectively, are also presented. Keywords: cluster computing, message-passing, directory, multi-threading.
1
Introduction
The appearing of commodity SMP workstations and high-performance SANs aroused the interest of researchers to the topic of cluster computing. Important tools have been developed and have provided an inexpensive vehicle for some classical problems. However, to address an important class of non-scientific large-scale multi-threaded applications and achieve the desired efficiency, in the presence of multiple communication technologies, current approaches and paradigms are not adequate. 1.1
Inter-node Communication
Cluster nodes are typically interconnected by means of high-performance networks like Myrinet or Gigabit Ethernet, but it is also very common to have an alternate low cost communication facility, like Fast Ethernet, to handle node setup and management. To interface high-performance NICs, user level communication libraries, like GM [9] and MVIA [11] (a VIA [4] implementation), became the right choice because it is possible to avoid context switching and memory copies. Several runtime systems and programming environments have been ported to exploit those technologies (MPI over GM, MPI over Infiniband, PVM over VIA, etc). To provide a uniform interface to multiple communication technologies, intermediate-level communication libraries, such as Madeleine [3], had also been developed. Nevertheless, H. Kosch, L. B¨osz¨orm´enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 969–979, 2003. c Springer-Verlag Berlin Heidelberg 2003
970
A. Alves et al.
it is not usual to combine multiple technologies in order to speed up application execution. Moreover, exploiting a secondary network like Fast Ethernet to perform specific communication tasks to alleviate the overhead of the main networking hardware seems to be an interesting topic that no one has properly addressed yet. The point is, low-level communication libraries allow the exploitation of highperformance networking hardware but their interfaces are inadequate for application programming and their communication models, mainly concerned with node-to-node message exchanging, make difficult the integration with higher-level abstractions. For instance, GM supports up to eight communication end-points per node which are not enough to deal with the requirements of highly multi-threaded applications. 1.2
Execution Environment
Traditionally the use of parallelism and high-performance computation has been directed to the development of scientific and engineering applications. Multiple platforms are available to help programmers design and develop their applications but it is important to note that the majority of the runtime environments are practically static, since application modules cannot be started disregarding the running ones; in a PVM application, for example, an initial task launches some other tasks (processes), which in turn may launch other ones, and the identifiers of relevant tasks have to be announced by their creators to other participants; in a PM2 [10] application, for example, the programmer is responsible for defining the routines that will be used to deliver active messages. To deal with the growing complexity of today’s large-scale parallel applications, as is the case of a system that integrates crawling, indexing and querying facilities, it is necessary to have a flexible execution environment where: multiple applications from multiple users collaborate to reach a common goal; I/O operations (access to databases, for example) represent a significant part of the application execution time; system modules may execute intermittently; application requirements may vary unpredictably. Execution environments that fulfil these requirements may be developed using the high-level abstractions provided by MPI, PANDA or others. However, the use of an existing platform imposes some constraints that are incompatible with the innovation needed to deal with the requirements of novel and more complex applications.
2
Resource Oriented Communication
RoCL, the Resource oriented Communication Library we present in this paper, uses existing low-level communication libraries to interface networking hardware. As a new intermediate-level communication library it offers a novel approach to system programmers to facilitate the development of a higher-level programming environment that supports the resource oriented computation paradigm of CoR[12]. 2.1
General Concepts
RoCL communication model defines three major entities – contexts, resources and buffers – and uses the services provided by a specific low-level distributed directory.
RoCL: A Resource Oriented Communication Library
971
A context is defined whenever the library is started up by calling the appropriate init primitive. Every context owns one or more low-level communication ports to send/receive messages, acting as a message store. Resources are a common metaphor used to model both communication and computation entities whose existence is announced by registering them in a global distributed directory service. Every resource is associated to an existing context and possesses a unique identifier. RoCL does not define the properties of resources neither limits their definition. Resources are instances of application level concepts whose idiosyncrasies result from a set of attributes specified at creation. An attribute is a pair name, value where name is a string and value is a byte sequence. A resource R with n attributes is defined by the expression R = {name1 , value1 , ..., namen , valuen }. To register a specific resource, the programmer must enumerate its attributes (see 3.1). To minimize memory allocation and registering operations, RoCL uses a buffer management system. Messages are held on specific registered memory areas to allow zerocopy communication. Prior to sending messages the programmer must acquire adequate size buffers. At reception the library is responsible for providing the communication subsystems with the necessary pre-registered memory blocks. Resource global identifiers are used to determine the origin and destination of messages. The identity of a resource may be previously known by the application or it may be obtained from the directory by querying it. Figure 1 presents the steps required for the operation of a basic client/server interaction, according to the RoCL communication model. Server
Client
1. initialize a RoCL context 2. register resource R1 = {type, server} and store the returned identifier in gid1 3. wait for any message sent to gid1 and store the buffer pointer in msg1 4. process message data through msg1 5. return buffer msg1 to the library
1. initialize a RoCL context 2. register resource R2 = {type, client} and store the returned identifier in gid2 3. discover resource {type, server} and store the returned identifier in gid3 4. obtain a buffer with a specific size and store the returned pointer in msg2 5. prepare message data by using msg2 6. send msg2 from gid2 to gid3 7. return buffer msg2 to the library
Fig. 1. A basic modelling example.
2.2
Basic Interface
The basic set of primitives that programmers may use to exploit RoCL is presented in table 1, organized according to the involved RoCL entities. Resource handling primitives are discussed through section 3. Buffer management and communication primitives have been presented in detail in a previous paper (see [1]). In this context it is important to highlight that buffer and communication handling primitives were designed to keep up zero-copy messaging. The use of low-level communication libraries like GM and VIA does not automatically guarantee zero-copy communication; the higher-level abstraction layer must define an appropriate interface to
972
A. Alves et al.
preserve the low-level features. SOVIA [8] and PVM over VIA [5], for example, use VIA, which permits zero-copy communication, but because users may continue to use the traditional sockets and PVM interfaces, those systems must copy or register user data (memory regions) before sending and can not avoid one copy at reception. Table 1. RoCL basic primitives. Contexts int rocl init() rocl exit() Resources int rocl register(rocl attrl t *attrs) rocl delete(int gid) int rocl query(int *gid, rocl attrl t *attrs) Buffers void * rocl bfget(int len) rocl bfret(void *ptr) rocl bftoret(void *ptr) int rocl bfstat(void *ptr) Communication rocl send(int ogid, int dgid, int tag, void *ptr, int len) int rocl recv(int dgid, int ogid, int tag, void **ptr, int *aogid, int *atag, int *alen, int timeout)
3
Directory Service
RoCL creates a fully dynamic system where communication entities may appear and disappear, at any moment, during application execution. To support these features we use the resource abstraction along with the facilities of a global distributed directory service. The importance of such a service is emphasized in [2] and [7]. RoCL directory service is a global distributed system that provides efficient and scalable access to the information about registered resources. This service enables the development of more flexible distributed computing services and applications. 3.1 Attribute Lists A resource is defined/registered by specifying an attribute list. The primitives used to manipulate resource attribute lists are presented in table 2. Attribute lists are used both for resource registering and querying. To successfully register a resource, all its attributes have to be completely specified, i.e. each attribute has to have a name and a value. For querying purposes, some attributes may be partially defined, i.e. attribute values may be omitted (NULL values) in order to inform the library about the attributes we want to know for a specific resource.
RoCL: A Resource Oriented Communication Library
973
Table 2. RoCL primitives to handle attribute lists. rocl attrl t * rocl new attrl(int max len) int rocl add attr(rocl attrl t *attrs, char *name, void *val, int len) void * rocl get attr(rocl attrl t *attrs, char *name, int *len) rocl kill attrl(rocl attrl t *attrs)
An attribute list is stored in a contiguous memory region to avoid memory copies when sending it to a server (see 3.2). In fact, an attribute list itself is used as a request/reply packet, requiring some space to be reserved at the attribute list head to allow the attachment of control information. 3.2
Local Operation
RoCL resources are registered at each cluster node using a local server that obtains a subset of global identifiers at start-up, thus minimizing inter-node communication (figure 2). The local resource database (DB in figure 2) is maintained in main memory and hashing techniques are used to accelerate query operations. Directory server
Resource entries (id + attribute list) Resources GM
Store defined context
VIA UDP
Communication subsystems
Store
Fig. 2. Resource local registering.
A query received by a local server corresponds to a request packet that may contain: the resource identifier, some completely specified attributes (with valid names and values) and some partially specified attributes (with NULL values). If the resource identifier is present, the search mechanism is trivial; otherwise the completely specified attributes are used to produce hash indexes. After finding the right resource, all partially specified attributes are examined and each one will be completed if an attribute with a matching name was previously registered for that resource. Because the library reserves space in the attribute list to store expected values for incomplete attributes, request packets may be used as reply packets avoiding memory allocation and copying. 3.3
Global Operation
If a particular query can not be satisfied at the local server, a global search is initiated. In a global search, all servers running across the cluster receive the request but only the one where the query succeeds is committed to reply.
974
A. Alves et al.
As a first approach to support global searches, we used UDP broadcast to spread requests through the Fast Ethernet network. This approach benefits from the native broadcasting support at protocol and hardware level. Requests may also be delivered to servers by combining UDP broadcast and GM or VIA spanning trees. The general operation will be as follows: 1) local servers periodically announce their presence using UDP broadcast; 2) each server maintains a list with all active servers; 3) spanning trees are used to reach all active servers. The use of spanning trees results from the fact that Myrinet hardware does not support broadcasting and the connection oriented model, adopted by VIA, is not compatible with multicasting. 3.4
Multiple-Answer Queries
Queries that don’t specify a resource identifier may result in multiple answers returned by one or more servers. This happens because different resources may share some (or even all) attribute names and values. RoCL provides dedicated primitives, (see table 3) to manage multiple answers to a single query. This interface should not be used whenever we just want a single answer or if it is known in advance that there will be a sole answer to a particular query. Table 3. RoCL primitives to handle multiple-answer queries. rocl handler t * rocl query start(rocl attrl t *attrs) int rocl query next(rocl handler t *handler, rocl attrl t *attrs) int rocl query stop(rocl handler t *handler)
When using this interface, results are fetched from the local server, one at a time, as if multiple independent single answer queries were in progress. Each request/reply packet transports the data – an attribute list – corresponding to a single resource. To support multiple answers, the local server maintains some control information for each query that is in progress, in order to be able to decide to: look for an answer in the local database, broadcast the query, store the answers returned by remote servers, search the local database for the next result, return an answer obtained from a remote server or request the next result from a remote server. Each remote server may return only one result as a reply to a specific broadcast. The local server stores the multiple answers received and sends a unicast request to a specific remote server whenever a result previously returned by that server is used up.
4
Inter-resource Message Passing
RoCL applications address messages to resources previously located using the directory service. Inter-resource message passing raises two main problems: message routing, because there is not a direct mapping between resources and communication subsystem addresses, and message dispatching, because resources are animated by threads and multiple low-level communication ports must be multiplexed.
RoCL: A Resource Oriented Communication Library
4.1
975
Message Routing
Contexts are the only valid end-points known by the communication subsystems. Therefore, resources must be mapped into contexts before message sending.
rocl_init()
rocl_register({<x,a>})
B A RoCL
Subsystems startup ADDRs
#
Attributes
A
GM=... VIA=...
B
x=a Context=A
Communication subsystems
A - Context identifier
B - Resource identifier
Fig. 3. Resource-context mapping.
The mapping between resources and contexts is handled as shown in figure 3. Contexts are registered at library initialization; they are managed as system resources. The addresses (or ports) of the communication subsystems are used as context attributes. The RoCL library uses the context global identifier, returned by the directory service, as an automatic attribute for resources, i.e. all resources will be tagged with the identifier assigned to the context where they belong. This approach is quite inefficient because three steps will be required to send a message to a particular resource: 1) the resource context must be obtained by querying the directory system, 2) the context addresses must also be obtained and finally 3) the message will be sent. The two first steps will require some messages (requests and replies) to be exchanged and therefore communication latency will be unacceptable. To overcome this problem, the library uses two dedicated caches to store the most recently required mappings between resource identifiers and context identifiers and between context identifiers and their communication subsystem addresses. 4.2
Message Dispatching
RoCL resources are animated by threads, meaning that RoCL must support concurrent/parallel access to communication facilities. Besides that, message reception is totally asynchronous, meaning that message delivering must take into account that receivers may not be waiting the messages sent to them. RoCL is a fully connectionless communication system that uses a dispatching mechanism based on system threads and message queues. System threads, one per communication subsystem, wait for messages using polling and interrupt handling techniques and store the received messages in a receiving queue. Resources access this queue to retrieve messages according to some selection criteria (message tag, origin identifier, etc). RoCL provides blocking and timed receivings, through the timeout parameter (see table 1): a negative timeout indicates a blocking behavior.
976
A. Alves et al.
Sending primitives may directly access the communication subsystems but, whenever concurrent calls occur, messages are stored in a sending queue and the primitives return immediately. Because receiving and sending queues only handle message descriptors, containing a pointer to the message data, no extra copies are introduced. A detailed explanation of message dispatching and the way it is related to quite distinct user-level communication protocols – GM and VIA – may be found in [1].
5 Applicability and Performance Although RoCL was designed as an intermediate-level message-passing library to the development of a new programming platform that is still under construction, it constitutes per se a basic programming tool. So, it is already possible to present particular applicability examples along with raw performance results. 5.1 TPVM over RoCL Assuming a simplified view of TPVM [6] we will examine the design/implementation of some of its basic functionality using RoCL primitives. PVM tasks (processes under UNIX) will create a RoCL context at start-up when they start running. They also register themselves as task resources, using an attribute type, task, and their global resource identifiers are used as PVM task identifiers (TIDs). A simplified PVM task spawning mechanism may be achieved as follows: – the parent task uses rsh to execute a special process launcher passing to it the conventional arguments required to start the spawned program along with its TID; – the launcher registers a temporary resource using as attributes its process identifier (PID) and the received parent task identifier ({type, tmp, pid, ..., ptid, ...}); – the launcher starts the target program using the exec primitive; – the spawned task, at start-up, queries the directory service, using its process identifier, to find the temporary resource and to obtain the parent task identifier; – the spawned task sends its global resource identifier to the parent/spawner process. To remotely activate TPVM threads we will need a launcher thread, automatically created when the PVM process/task starts, that will block waiting for messages that request the activation of a specific TPVM thread. The launcher thread – the pod controller – registers itself as a system thread resource using an attribute type, systhread. The tpvm export primitive will correspond to a simple RoCL register operation. A TPVM thread will be defined by using a name and the global resource identifier of its pod controller – {type, threaddef , name, ..., systhread, ...}. The tpvm spawn primitive will use the thread name to query the directory service and find the global identifier of the pod controller. This global identifier is used to send a request message to the launcher thread, which will create the desired thread. Instantiated threads also register themselves using an attribute type, thread and send their global identifiers to the spawners. A spawned thread obtains its parent identifier directly; the request sent to the launcher thread carries the spawner global identifier.
RoCL: A Resource Oriented Communication Library
977
A host database may also be provided as a collection of RoCL resources. However, since no PVM daemons are used, this database will only be useful to find out available machines and select a particular target node for spawning operations. 5.2
Inter-resource Message-Passing Performance
RoCL message-passing performance is influenced by the communication subsystems we support, the way we interface them and the capabilities of LINUX threads 1 .
Fig. 4. RoCL performance.
Figure 4:left presents round-trip times relating to three networking technologies – Myrinet (LANai 9), Gigabit Ethernet (SysKonnect 9821) and Fast Ethernet (Intel EtherExpress PRO/100) – exploited through GM, MVIA and UDP 2 . Performance tests were performed on dual Pentium III 733MHz workstations running RedHat 9.0. Myrinet and Gigabit adapters were attached to 64bits/66MHz PCI slots. RoCL over GM allows to achieve a 30µs round-trip time for 1 byte messages, which corresponds to an overhead of 10µs when compared to GM node-to-node performance. MVIA for SysKonnect, as expected, outperforms all communication subsystem alternatives but GM. Surprisingly, for small messages, SysKonnect hardware produces better results than Myrinet, when using UDP. In order to evaluate message-passing alternatives, it is mandatory to also analyze the impact of communication on computation and vice-versa. To evaluate this interdependence we had calculated the execution rate of a particular computation cycle, using one thread per processor, without performing any communication task, and then we run both the original round-trip and the computation benchmarks concurrently. The impact on computation may be expressed by the ratio ExCS /Ex0 , where ExCS stands for the execution rate obtained when we run, concurrently, the roundtrip benchmark using one of the communication subsystem alternatives and Ex0 stands 1 2
Currently RoCL is only supported under LINUX. GM runs on Myrinet, MVIA runs on SysKonnect and Intel and UDP runs on each of them.
978
A. Alves et al.
for the execution rate obtained without background communication. Similarly, the ratio RtCS0 /RtCS , where RtCS and RtCS0 stand for the round-trip times obtained for a given communication subsystem, respectively, with or without computations taking place concurrently, express the impact on communication. To easily compare round-trip ratios from different communication subsystems, we use the ratio RtGM0 /RtCS , where RtGM0 stands for the round-trip times obtained using GM (the best round-trip times we may achieve with RoCL). These two ratios express the performance sustainability of computation and communication, when they use the same CPU(s). As we consider that both communication and computation performances are equally fundamental to determine the success of high performance computing, we use an overall ratio, for each communication subsystem, calculated as a geometric average. Figure 4:right presents the overall performance sustainability we can expect to achieve in RoCL applications. It is important to note that the RoCL impact on computation is in accordance to the selected hardware and communication subsystem: FastEhternet adapters perform badly and require higher CPU intervention as message size increases; UDP, a complex protocol, requires more CPU cycles than MVIA and GM and so the overall performance drops.
6
Conclusions
RoCL introduces a new communication paradigm to facilitate the design and implementation of high-level execution environments. In this paper, the key concepts related to inter-resource message-passing and the operation of a low-level distributed directory service – a very important component of RoCL – were presented. The case study “TPVM over RoCL" shows that the abstractions provided by RoCL allow the rapid design and implementation of a great variety of high-level applications. Performance values indicate that RoCL exploits efficiently the low-level communication subsystems and that multi-threaded dispatching mechanisms are now feasible. Scalability evaluation of the directory service is still undergoing. Due to the limited number of cluster nodes available for testing, we intend to use simulation techniques.
References 1. A. Alves, A. Pina, J. Exposto, and J. Rufino. ToCL: a thread oriented communication library to interface VIA and GM low-level protocols. to appear ICCS ’03, 2003. 2. M. Beck, J. Dongarra, G. Fagg, G. A. Geist, P. Gray, J. Kohl, M. Migliardi, K. Moore, T. Moore, P. Papadopoulous, S. Scott, and V. Sunderam. HARNESS: A next generation distributed virtual machine. Future Generation Computer Systems, 15(5–6):571–582, 1999. 3. L. Boug´e, J.-F. M´ehaut, and R. Namyst. Madeleine: An Efficient and Portable Communication Interface for RPC-Based Multithreaded Environments. In PACT ’98, 1998. 4. Compaq Computer Corp., Intel Corporation & Microsoft Corporation. Virtual Interface Architecture Specification. http://www.vidf.org/info/04standards.html, 1997. 5. R. Espenica and P. Medeiros. Porting PVM to the VIA architecture using a fast communication library. In PVM/MPI ’02, 2002.
RoCL: A Resource Oriented Communication Library
979
6. J. Ferrari and V. Sunderam. TPVM: Distributed Concurrent Computing with Lightweight Processes. In HPDC ’95, 1995. 7. S. Fitzgerald, I. Foster, C. Kesselman, G. von Laszewski, W. Smith, and S. Tuecke. A Directory Service for Configuring High-Perf. Distributed Computations. In HPDC ’97, 1997. 8. J.-S. Kim, K. Kim, and S.-I. Jung. SOVIA: A User-level Sockets Layer Over Virtual Interface Architecture. In CLUSTER ’01, 2001. 9. Myricom. The GM Message Passing System. http://www.myricom.com, 2000. 10. R. Namyst and J. M´ehaut. PM2 : Parallel Multithreaded Machine. A computing environment for distributed architectures. In ParCo’95, 1995. 11. National Energy Research Scientific Computing Center. M-VIA: A High Performance Modular VIA for Linux. http://www.nersc.gov/research/FTG/via, 2002. 12. A. Pina, V. Oliveira, C. Moreira, and A. Alves. pCoR - a Prototype for Resource Oriented Computing. In HPC ’02, 2002.
A QoS Multicast Routing Protocol for Dynamic Group Topology Li Layuan and Li Chunlin Department of Computer Science, Wuhan University of Technology, Wuhan 430063, P. R.China
[email protected] Abstract. This paper discusses the multicast routing problem with multiple QoS constraints, which may deal with the delay, delay jitter, bandwidth and packet loss metrics, and describes a network model for researching the routing problem. It presents a multicast routing protocol with multiple QoS constraints (MRPMQ). The MRPMQ attempts to significantly reduce the overhead of constructing a multicast tree with multiple QoS constraints. In MPRMQ, a multicast group member can join or leave a multicast session dynamically, which should not disrupt the multicast tree. It also attempts to minimize overall cost of the tree, and satisfy the multiple QoS constraints and least cost (or lower cost) requirements. In this paper, the proof of correctness and a complexity analysis of the MRPMQ are also given. Simulation results show that MRPMQ is a feasible approach to multicast routing with multiple QoS constraints. Keywords: Multicast routing; protocol; multiple QoS constraints; QoS routing; NP-complete.
1
Introduction
The traditional multicast routing protocols, e.g., CBT and PIM [1-4], were designed for best-effort data traffic. They construct multicast trees primarily based on connectivity. Such trees may be unsatisfactory when QoS is considered due to the lack of resources. Several QoS multicast routing algorithms have been proposed recently. Some algorithms [5-7] provide heuristic solutions to the NP-complete constrained Steiner tree problem, which is to find the delay-constrained least-cost multicast trees. These algorithms however are most practical in the Internet environment because they have excessive computation overhead, require knowledge about the global network state, and do not handle dynamic group membership. Jia’s distributed algorithm [2] does not compute any path or assume the unicast routing table can provide it. However, this algorithm requires excessive message processing overhead. The spanning join protocol by Carlberg and Crowcroft [3] handles dynamic membership and does not require any global network state. However, it has excessive communication and message processing overhead because it relies on flooding to find a feasible tree branch to connect a new member. QoSMIC [4], proposed by Faloutsos et al., alleviates but does not eliminate the flooding behavior. In addition, an extra control element, called Manager router, is introduced to handle the join requests of new members. H. Kosch, L. Böszörményi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 980–988, 2003. © Springer-Verlag Berlin Heidelberg 2003
A QoS Multicast Routing Protocol for Dynamic Group Topology
981
Multicast routing and its QoS-driven extension are indispensable components in a QoS-centric network architecture [5,7,9–10]. Its main objective is to construct a multicast tree that optimizes a certain objective function (e.g., making effective use of network resources) with respect to performance-related constraints (e.g., end-to-end delay bound, inter-receiver delay jitter bound, minimum bandwidth available, and maximum packet loss probability).
2 Network Model As far as multicast routing is concerned, a network is usually represented as a weighted digraph G = (V, E), where V denotes the set of nodes and E denotes the set of communication links connecting the nodes. |V| and |E| denote the number of nodes and links in the network, respectively, Without loss of generality, only digraphs are considered in which there exists at most one link between a pair of ordered nodes [8]. Associated with each link are parameters that describe the current status of the link. Let s V be a source node of a multicast tree, and M ⊆ {V{s}} be a set of end nodes of the multicast tree. Let R be the set of positive weights and R+ be the set of nonnegative weights. For any link e E, we can define the following QoS metrics: delay function delay (e): E R, cost function cost (e): E R, bandwidth function bandwidth (e); E R, and delay jitter function delay-jitter (e): E R+. Similarly, for any node n V, one can also define some metrics: delay function delay (n): V R, cost function cost (n): V R, delay jitter function delay-jitter (n): V R+ and packet loss function packet-loss (n): V R+. We also use T (s,M) to denote a multicast tree, which has the following relations: 1) delay (p (s,t)) = 2) cost (T(s,M))=
∑ delay (e ) + ∑ delay ( n ) .
e∈ p ( s,t )
n∈ p ( s ,t )
e∈T ( s,M )
n ∈T ( s,M )
∑ cost ( e ) + ∑ cost ( n ) .
3) bandwidth (p(s,t)) = mine³p(s,t){bandwidth (e)}. 4) dealy-jitter (p (s,t)) =
∑ delay − jitter ( e ) + ∑ delay − jitter ( n ) .
e∈ p ( s,t )
5) packet-loss (p (s,t)) = 1 −
n∈ p ( s,t )
∏ (1 − packet - loss (n))
n∈ p ( s,t )
where p (s,t) denotes the path from source s to end node t of T (s, M). Definition 1. The QoS-based multicast routing problem deals mainly with some elements: network G=(V,E), multicast source s V, the set of end nodes M ⊆ {V{s}}, delay(·) R, delay-jitter(·) R+, cost(·) R, bandwidth(·) R, and packet-loss(·) R+. This routing problem is to find the T (s, M) which satisfies some QoS constraints for all t³M: 1 Delay constraint: delay (p (s,t)) Dt 2 Bandwidth constraint: bandwidth (p (s,t)) B
982
L. Layuan and L. Chunlin
3 Delay jitter constraint: delay-jitter (p (s,t)) J 4 Packet loss constraint: packet-loss (p (s,t)) L Simultaneously, the cost (T (s, M)) should be minimum. In the above QoS constraints, the bandwidth is a concave metric, the delay and delay jitter are additive metrics, and the packet loss is multiplicative metric. In these metrics, the multiplicative metric can be converted to an additive metric. For simplicity, we assume that all nodes have enough resources, i.e., they can satisfy the above QoS constraints. Therefore, we only consider the links’ or edges’ QoS constraints, because the links and the nodes are equivalent to the routing issue in question.
3 MRPMQ The Join procedure of MRPMQ can be formally described as follows. 1) if a new member (ti) wishes to join a T (s,M) the new member sends JOINreq to some neighbor tj 2) if (d(s,*)+d(j,i) D) (dj(s,*)+dj(j,i) J) (bw(tu,tv) B) {where d(s,*) and dj(s,*) are the delay sum and the delay jitter sum from the source s to all downstream nodes of a path, respectively, but except for the last pair of nodes. u and v are the sequence numbers between two adjacent nodes on the path from source to the new member} tj transfers JOINack to ti fi if bw (tu,tv)D) (dj (s,*)+dj (j,i)>J) if the next hop is the immediate upstream node tj of tj tj transfers JOINreq to tj tj adds JOINpend for ti to the forwarding entry tj transfers JOINack (or JOINnak) to tj fi if the next hop is not the immediate upstream node if (d (s,*) Dd (j,i)) (dj (s,*) Jdj (j,i)) tj transfers JOINreq to tj* tj adds the routing entry marks tj* as upstream node tj* transfers JOINack (or JOINnak) to tj if tj receives JOINack tj forwards a pruning msg to tj fi fi fi fi 4) if (d(s,*)+d(j,i) D) (dj(s,*)+dj(j,i) J) tj computes a new path if (d (p (s,i))=min[d (s,*), Dd (j,i)])
A QoS Multicast Routing Protocol for Dynamic Group Topology
983
(dj(p (s,j))=min[dj (s,*), Jdj (j,i)]) tj receives JOINack fi tj receives JOINnak fi We can use the following example to show how the MRPMQ works and how the multicast tree is constructed in a distributed fashion. Fig. 1 is a network graph. In this example, node t0 is the multicast source. t4, t9, t14, t19 and t24 are the joining nodes. Recall that the characteristics of a network’s edge can be described by a fourtuple (D,J,B,C). In this example shown in Fig. 1, suppose delay constraint is D=20, delay jitter constraint J=30 and bandwidth constraint B=40. t4 wishes to join the group, it computes the paths according to the multiple QoS constraints. The path
0
(4,6,45,12)
5
(5,7,50,12) 10 (4,6,45,12) 15 (5,7,50,11)
20
(4,7,50,11)
(5,6,45,11)
1 (3,6,45,10) (5,6,50,10)
(6,5,50,10)
2
(5,6,45,12) 8
(6,5,45,12) 4
16
(5,6,45,12)
22 (1,4,45,11)
(1,2,50,10)
(2,2,45,10) 18
23 (2,2,45,10) (1,2,50,10)
(2,4,45,10)
(5,6,45,11) 9
(3,5,50,11)
(3,4,45,10)
(3,5,50,12)
(5,7,45,12)
(5,6,45,10)
17
13 (3,5,50,10)
21
(3,3,45,9) 12
(4,5,50,11)
3
(4,6,50,10)
(4,6,50,10)
(3,4,50,11) 7
(4,6,45,12)
(4,6,45,12) 11
6
(5,7,50,12) 14
(5,6,50,10)
(4,6,45,11)
(5,6,50,12)
(4,6,45,11)
(5,6,45,12)
(5,5,50,10)
(8,12,50,25)
(5,6,45,11) 19
(5,5,50,11)
(4,7,50,12)
(4,5,45,11)
24
Fig. 1. An example network graph
(t0 t1 t6 t7 t8 t4) can satisfy tthe delay constraint, the delay jitter constraint and the bandwidth constraint, and has minimum cost. Therefore, the join path should be the path (t0 t1 t6 t7 t8 t4). The bold lines of Fig. 2(a) show the tree when t4 has joined the group. When t9 joins the group, it computes a path (t0 t1 t6 t7 t8 t9) which should satisfy the delay, delay jitter and bandwidth constraints, and also have minimum cost. The JOINreq is accepted at t8. The bold lines of Fig. 2(b) show the tree when t9 has joined the group. When t14 joins the group, it computes the paths with multiple QoS constraints. The path (t0 t1 t6 t7 t12 t13 t14) does not satisfy the delay jitter constraint. The path (t0 t5 t6 t7 t8 t13 t14) does not satisfy delay and delay jitter constraints. The path (t0 t6 t7 t12 t13 t14) and path
984
L. Layuan and L. Chunlin
(t0 t5 t6 t7 t12 t13 t14) satisfy the delay, delay jitter and bandwidth constraints. The latter has the lower cost. Therefore, the join path should be the path (t0 t5 t6 t7 t12 t13 t14). Meanwhile, t6 should prune off from the original parent t1, the resulting tree is shown in Fig. 2(c) (see the bold lines of Fig. 2(c)). The tree after t19 joins the group is also shown in Fig. 2(d). When t24 joins the group, it computes the join paths. If t18 receives JOINreq from t24, it finds out that the existing path (t0 t5 t6 t7 t12 t13 t18) does not satisfy the delay constraint for the new member t24, while the new path (t0 t5 t6 t7 t12 t17 t18) does not satisfy the delay jitter constraint for t24. t18 computes a new feasible path with delay constraint, which is given by d (p (s,j)) = min[d (s,*), Dd (j,i)] = min[(d (0,5)+d (5,6)+d (6,7)+d (7,12)+ d (12,13)+d (13,18)), Dd (18,24)] = min[19,18]=18 and delay jitter constraint, which can be given by dj (p (s,j)) = min[dj (s,*), Jdj (j,i)] = min[(dj (0,5)+dj (5,6)+dj (6,7)+dj (7,12)+ dj (12,13)+dj (13,18)), Jdj (18,24)] = min[28,28]=28 Thus, this new feasible path should be path (t0 t6 t7 t12 t13 t18). t6 should prune off from the old parent t5, and the final tree can be shown in Fig. 2(e) (see the bold lines of Fig. 2(e)). The loop-free routing for the above protocol can be achieved by maintaining a searching tree at any time.
Fig. 2. Constructing multicast tree
4 Correctness and Complexity Analysis Theorem 1. If a path from a new member to T(s,M) has sufficient resources to satisfy the QoS constraints and has minimum cost, the algorithm searches only one path. Proof. Note that a necessary condition for multiple paths to be searched is that a single path does not satisfy the QoS constraints, such as (d(p(s,j)) min[d(s,*), D d(j,i)]) (dj(p(s,j)) min[dj(s,*), Jdj(j,i)]). However, if sufficient resources are
A QoS Multicast Routing Protocol for Dynamic Group Topology
985
available on every link and node of the path, no node forwarding JOINreg will ever enter the multiple paths search state. Thus, the above theorem holds. Lemma 1. Whenever during the routing process, all paths being searched form a T(s,M) structure. Proof. The paths being searched will be marked by the routing entries at the nodes. In MRPMQ, any routing entry has a single out interface and one or multiple in interfaces. Hence, the nodes will form a searching tree structure. This tree is just a T(s,M). Theorem 2. An available and feasible path found by MRPMQ is loop-free. Proof. This Theorem follows directly from the above Lemma 1. Theorem 3. MRPMQ can find an available and feasible path if one exists. Proof. This theorem can be proved by contradiction. Suppose MRPMQ fails while an available and feasible path does exist. Let e(i,j) be the first link in the path that the protocol did not explore. Since e(i,j) is the first unexplored link of the path, ti must have received a request message from the previous link or ti is the new member issuing the request message. In either case, ti is not in the initial state. Therefore, ti is in the failure state, which requires ti to explore all outgoing links including e(i,j). It contradicts the assumption that e(i,j) is not explored. In MRPMQ, route computation can generally be made by the end node. If the join path is computed on-demand, the complexity depends on the unicast protocol. If QoS metrics are delay and bandwidth, there exist QoS routing heuristics which are O (|V| x |E|), where |V| is the number of nodes and |E| is the number of edges in a network. For most networks, |E|=O(|V|), hence the complexity is O(|V|2). For a multicast group with |M| members, the computation overhead is O(|V|2|M|). The study shows that computation complexities of CSPT and BSMA [5] are O (|E| log|V|) and O(|V|3 log|V|), respectively. The study shows that the average message processing overheads to construct the multicast tree of MRPMQ, Jia’s algorithm, and QoSMIC (centralized or distributed) are K.2|M|, K.2|M|, |M| (wg(w-1)(y-1)+c-k)gx (centralized QoSMIC) and |M| (wg(w-1)(y-1)+|T|)gx (distributed QoSMIC), respectively, where the x factor is added to reflect the fact that messages have to be processed at more than one node, w is the average degree of a node, y is the maximum TTL used for a search, |T| is the tree size, c is the number of candidates for a BID-ORDER session and x depends on the topology and y, while 2 x 1+K.
5 Simulations In the simulations, we compare the quality of routing trees by their network cost for constructing a multicast tree (cost (T (s, M))) [10]. The network cost is obtained as the mean value of the total number of simulation runs. At each simulation point, the simulation runs 80 times. Each time the nodes in the group G are randomly picked out from the network graph. The network cost is simulated against two parameters: delay bound D and group size. In order to simulate real situations, the group size is always made less than 20% of the total nodes, because multicast applications running in a wide area network usually involve only a small number of nodes in the network, such
986
L. Layuan and L. Chunlin
as video conference systems, distance learning, co-operative editing systems, etc. [6– 7]. Fig. 3 shows the network cost versus group size. In this round of simulations, the network size is set to 300 and D is dmax+3/8dmax. From Fig. 3, we can see when group size grows, the network cost produced by MRPMQ, BSMA and KMB increases at a rate much lower than CSPT. MRPMQ performs between BSMA and KMB. BSMA, KMB and the proposed MRPMQ can produce trees of comparable costs. Fig. 4 is the network cost versus D. During this round of simulations, the network size is fixed at 300 nodes, group size is 20. From Fig. 4, it can be seen that the network cost of the CSPT algorithm is on the top and almost does not change as D increases. This is because the generation of the shortest path tree does not depend on D. Of the remaining three algorithms, the proposed MRPMQ has the lowest cost. From Fig. 4, we can also see that tree costs decrease for the MRPMQ, BSMA and KMB algorithms as the delay bound is relaxed. This shows all three schemes indeed can reduce the cost when the delay bound is relaxed. From Fig. 3 and Fig. 4, one can see that the MRPMQ, BSMA and KMB algorithms can produce trees of comparable costs. However, compared with the BSMA and KMB algorithms, the proposed MRPMQ has the advantage of being fully distributed and allowing incremental tree build-up to accommodate dynamic joining of new members. Furthermore, the MRPMQ is much less costly in terms of computation cost and in terms of cooperation needed from other network nodes compared with other schemes.
Fig. 3. Network cost vs. group size
A QoS Multicast Routing Protocol for Dynamic Group Topology
987
Fig. 4. Network cost vs. delay bound
6 Conclusion In this paper, we discuss the multicast routing problem with multiple QoS constraints, which may deal with the delay, delay jitter, bandwidth and packet loss metrics, and describe a network model for researching the routing problem. We have presented a multicast routing protocol with multiple QoS constraints (MRPMQ). The MRPMQ can significantly reduce the overhead of establishing a multicast tree. In MRPMQ, a multicast group member can join or leave a multicast session dynamically, which should not disrupt the multicast tree. The MRPMQ also attempts to minimize the overall cost of the tree. This protocol may search multiple feasible tree branches in distributed fashion, and can select the best branch connecting the new member to the tree. The join of a new member can have minimum overhead to on-tree or non-tree nodes. The correctness proof and complexity analysis have been made. Some simulation results are also given. The study shows that MRPMQ is a feasible approach to multicast routing with multiple QoS constraints. Further work will investigate the protocol’s suitability for inter-domain multicast and hierarchical network environments.
Acknowledgment. The work is supported by National Natural Science Foundation of China and NSF of Hubei Province.
988
L. Layuan and L. Chunlin
References [1] Li Layuan and Li Chunlin, “The QoS routing algorithm for ATM networks”, Computer Communications, Vol. 24, No. 3-4, 2001, pp. 416–421. [2] X. Jia. “A distributed algorithm of delay-bounded multicast routing for multimedia applications in wide area networks”, IEEE/ACM Trans. on Networking, Vol. 6, No. 6, Dec. 1998, pp. 828–837. [3] K. Carlberg and J. Crowcroft, “Building shared trees using a one-to-many joining mechanism”, ACM Computer Communication Review, Vol. 27, No. 1 , Jan. 1997, pp. 5– 11. [4] M. Faloutsos, A. Banerjea, and R. Pankaj, “QoSMIC: Quality of Service sensitive multicast Internet protocol”, SIGCOMM’98, Vol. 28, September 1998. [5] Q. Zhu, M. Parsa, and J. J. Garcia-Luna-Aceves, “A source-based algorithm for delayconstrained minimum-cost multicasting,” Proc. IEEE INFOCOM 95, Boston, MA, April 1995. [6] Y. Xiong and L.G. Mason, “Restoration strategies and spare capacity requirements in selfhealing ATM networks”, IEEE/ACM Trans. on Networking, Vol. 7, No. 1, Feb. 1999, pp. 98–110. [7] B. M. Waxman, “Routing of multipoint connections.” IEEE Journal on Selected Area in Communications, Dec. 1998, pp. 1617–1622. [8] R. G. Busacker and T. L. Saaty, Finite Graphs and Networks: An introduction with applications, McGraw-Hill, 1965. [9] R. A. Guerin and A. Orda, QoS routing in networks with inaccurate information: Theory and algorithms, IEEE/ACM Trans. on Networking, Vol. 7, No. 3, June 1999, pp. 350– 363. [10] Li Layuan and Li Chunlin, Computer Networking, National Defense Industry Press, Beijing, 2001.
A Study of Network Capacity under Deflection Routing Schemes Josep F`abrega and Xavier Mu˜ noz Departament de Matem` atica Aplicada IV, Universitat Polit`ecnica de Catalunya Campus Nord C3, Jordi Girona 1–3, 08034 Barcelona, Spain {matjfc,xml}@mat.upc.es
Abstract. Routing in bufferless networks can be performed without packet loss by deflecting packets when links are not available. The efficiency of this kind of protocol (deflection routing) is highly determined by the decision rule used to choose which packets have to be deflected when a conflict arises (i.e. when multiple packets contend fo a single outgoing link). As the load offered to the network increases the probability of collision becomes higher and it is to be expected that at a certain maximum offered load the network gets saturated. We present an analytical method to compute this maximum load that nodes can offer to the network under different deflection criteria.
1
Introduction
Deflection routing [1] is a routing scheme for bufferless networks based on the idea that if a packet cannot be sent through a certain link due to congestion, it is deflected through any other available one (instead of being buffered in a node queue) and rerouted to destination from the node at which the packet arrives. In this way, congestion causes packets admitted to the network to be misrouted temporarily, in contrast with traditional schemes where such packets might be buffered or dropped. This kind of protocol has been proposed, for instance, to route packets in all-optical networks because optical storage is not possible with nowadays technology [2,4,7] (Messages can only be shortly delayed by a fiber loop in order to wait for a quick processing of their headers, but cannot be buffered in queues without optical to electrical conversion.) Many approximations have been proposed in the literature for implementing deflection routing [8]. The efficiency of the protocol is highly determined by the decision criteria used to deflect packets when collisions arise (i.e. when two packets should use the same outgoing link and one of them must be deflected). The strategies used to solve these conflicts can be divided into two categories. On one hand, those that give priority to the most disadvantaged packets in order to avoid deadlock or timeouts (and thus trying to guarantee that all packets arrive
Research supported by the Ministry of Science and Technology, Spain, and the European Regional Development Fund (ERDF) under projects TIC-2001-2171 and TIC2002-00155.
H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 989–994, 2003. c Springer-Verlag Berlin Heidelberg 2003
990
J. F` abrega and X. Mu˜ noz
to destination): MAXDIST (packets that are further away to destination have a higher priority), MAXTIME (older packets are given a higher priority), and MAXDEFL (the larger the number of times a packet has been deflected, the higher the priority). On the other hand, strategies that give preference to those packets that might arrive to destination as soon as possible. The decision criteria within this group are analogous to the preceding case: MINDIST, MINTIME and MINDEFL. As the load offered to the network increases, deflection routing becomes less efficient (the probability of collision increases) and at a certain maximum offered load R the network becomes saturated. Figure 1 shows the results of simulations under different decision criteria for a certain network topology. As the reader can check, the maximum allowed traffic highly depends on the decision rule used to deflect packets. This paper shows an analytical method to compute this maximum load R. Some specific theoretical results for the hypercube and shufflenet networks can be found in [11,10].
Fig. 1. Delivered vs. offered traffic
2
The Model
Network topology is another important aspect that determines the efficiency of deflection routing. We suppose here that the network is modeled by a directed graph (digraph) in which all nodes offer the same constant traffic load to the network by means of an input queue. We also assume that the communication model have the following properties: packets have fixed length and are transmitted synchronously, processing time at the intermediate nodes is zero, only one packet can travel through a given link at a time, and a packet in an input queue enters in the network as soon as a link is available and competes with other packets in the network for the same links (transmit no hold acceptance (txnh). (Other queue management policies can be found in [5].) With respect to the topology and routing table the following restrictions are also supposed: all
A Study of Network Capacity under Deflection Routing Schemes
991
nodes have the same number of outgoing and incoming links (i.e. the digraph is δ-regular), packets will always try to follow the shortest path to destination, and there is only one shortest path between any pair of nodes, which means that a packet remains at the same distance or further away to its destination after being deflected. Besides the previous assumptions we consider two more suppositions in order to simplify our problem. Firstly, a packet entering to a node exits through any outgoing arc with probability 1/δ (uniform traffic distribution). This approximation is reasonable if the routing table is such that the edge-forwarding index [12] is close to the minimum. Secondly, since the purpose of our study is to deal under heavy traffic conditions, it will be assumed that each node has a packet ready to be transmitted as soon as a link is free. This supposition holds only for deterministic distribution of packet arrivals but it is not true for other traffic distributions such as Poisson or sporadic. Nevertheless the supposition is acceptable if the arrival ratio is high enough and if there are large input buffers which are overflown. With these assumptions, in steady-state the number of incoming packets to a node equals the number of incoming links to that node, and equals also the number of outgoing packets and outgoing links. In other words, the probability of a link being free during a time unit is zero. Hence the number of packets in the network equals de number of arcs in the digraph, and then, by Little’s Law we have nRt = m where R stands for the maximum admissible throughput per node (i.e. the maximum load rate nodes can offer to the network) and t is the average time for a packet to arrive to destination. In the case of δ–regular directed graphs, we have R = δ/t. The following two sections will be devoted to give methods to compute R under different decision criteria.
3
Random Policy
To compute the average time t that (in steady-state) a packet is in the network hopping through its nodes, let us define an absorbing Markov chain with states corresponding to the possible distances that the packet could be to its destination: 0, 1, . . . , D, where D is the diameter of the digraph and state zero stands for a packet that has arrived at destination (see Figure 2). The transition
Fig. 2. Markov chain used to compute the average time to arrive to destination
992
J. F` abrega and X. Mu˜ noz
probability matrix M = (mij ) of this chain is given by mij = Pa (i) = 1 − Pd (i) if j = i − 1 and i > 0, mij = Pd (i)Pt (i, j) if j ≥ i and i > 0, and 0 otherwise, where Pd (i) is the probability that a packet being at distance i is deflected and Pt (i, j) is the conditional probability that a deflected packet which is at distance i to its destination z goes to a node which is at distance j to z. The deflection probability Pd (i) depends on the number of packets that should use the same link at a time. Given a packet that wants to use certain link, the probability that N other packets the same outgoing want to use δ−1−N N (1 − 1/δ) (1/δ) . Therelink (N collisions) is given by Pc (N ) = δ−1 N fore, the probability of being deflected conditioned to being at distance i to δ−1 destination is Pd (i) = N =1 (1 − Pa (i | N )) Pc (N ), where Pa (i | N ) stands for the probability Pa (i) conditioned to N collisions. In the case of random policy, this probability does not depend on i, and according to assumption of uniform traffic, is given by Pa (i | N ) = 1/(N + 1). Consequently, in the case of random is, for any distance i, given by of deflection δ−1 policy, the probability δ−1−N N δ (1 − 1/δ) (1/δ) = (1 − 1/δ) . Pd = N =1 (1 − 1/(N + 1)) δ−1 N On the other hand, the probabilities Pt (i, j) depend only on the network topology as well as the initial probability Pin (i) of each state (i.e. the probability that a new packet entering the network is assigned a destination node at distance i from the source). A detailed analysis of these probabilities can be found in [9] for the case of Kautz networks [3]. Once defined the transition and initial probabilities, the Markov chain makes it possible to compute the average time that a packet is in the network by computing the mean time t to absorption to the zero state. This time will give us the maximum admissible throughput R = δ/t.
4
Distance Priority Criteria
In case of distance criteria, the probability that a packet advances towards destination depends on its distance to that destination, but also on the probability that competitors are closer to or further to its own destinations. In other words, to compute the probability of advancing Pa (i) = 1 − Pd (i) we need to know the probability of being at each state in the Markov chain. Hence, a more complex analysis must be performed. Since packets fighting for a certain outgoing link have not arrived yet to destination (i.e. they are not in the zero state) we can consider the probabilities of being at each state conditioned to not being in state 0. Let us call these new conditioned state probabilities P (i). In order to compute P (i) and the new transition probabilities it is convenient to state the problem in a different way: there is a fixed number of packets (equal to the number of links) hopping through the nodes in such a way that they will never exit the network, but once a packet reaches its destination, a new random destination is immediately assigned to it. The new ergodic Markov chain associated to this problem is shown in Figure 3. Because of the assumption that each node has a packet ready to be transmitted as soon as a link is free, the traffic in the network will be exactly the same as in
A Study of Network Capacity under Deflection Routing Schemes
993
Fig. 3. Conditioned Markov chain
the original problem and, moreover, the transition probabilities will also be the same for i ≥ 2. More precisely, the entries mij of the new transition probability matrix M = (mij ) are 1 − Pd (i) if j = i − 1 and i = 1, Pd (i)Pt (i, j) if j ≥ i and i = 1, Pd (i)Pt (i, j) + (1 − Pd (i))Pin (j) if i = 1, and 0 otherwise, where Pin (j) stands for the probability that the new assigned destination is at distance j from source, as in the preceding section. Each deflection probability Pd (i) is now a function of the state probabilities P (1), . . . , P (D). For instance, if MINDIST criteria is used, Pd (i) can be computed as in previous section, but now with the advancing probabilthe N N −k P (i)k )/(k + 1) if i < D and Pa (i|N ) = ity Pa (i|N ) = k=0 N k (P (d > i) N P (D) /(N + 1) if i = D, where P (d > i) stands for the probability that a competitor packet is at distance greater than i to its destination. Table 1. Results from theory and simulation
d 2 2 3 3 4 4
MINDIST D Theor. Simul. 3 0,52 0,54 4 0,37 0,38 3 0,62 0,69 4 0,43 0,47 3 0,75 > 0, 80 4 0,53 0,58
MAXDIST Theor. Simul. 0,37 0,39 0,23 0,22 0,39 0,41 0,21 0,20 0,45 0,45 0,22 0,20
Random policy Theor. Simul. 0,46 0,46 0,30 0,30 0,54 0,55 0,33 0,33 0,64 0,65 0,39 0,38
Even if the transition probabilities are unknown, M is the probability matrix of an ergodic Markov chain (finite, irreducible and aperiodic). Let V = (V (1), . . . , V (D)) be the stationary distribution of M (i.e. a probability left eigenvector associated to the eigenvalue 1), each V (i) being a function of P = (P (1), . . . , P (D)). The probabilities Pd (i) can be computed then by solving the equation V = P . Finally, we consider again the absorbing Markov chain M and compute the expected time to absorption in state 0, as in Section 3.
994
5
J. F` abrega and X. Mu˜ noz
Validation through Simulation
Table 1 gives, for the case of the d-regular Kautz network K(d, D) with diameter D, a comparison of theoretical results obtained by applying our analytical model with results from simulation. In the computation of the theoretical results, the values of Pin (i) and Pt (i, j) are those obtained in [9]. The comparison is a validation of the hypothesis and assumptions used in paper. Further details and description of the simulation can be found in [14].
References 1. Baran, P.: On distributed communications networks. IEEE Trans. Comm. Sys. 12 (1964) 1–9 2. Baransel, C., Dobosiewicz, W., Gburzynski, P.: Routing in multihop packet switching: Gb/s challenge. IEEE Network 9 (1995) 38–61 3. Bermond, J.C.; Peyrat, C.: De Bruijn and Kautz networks: A competitor for the hypercube? In: Andre, F., Verjus, J.P. (eds): Hypercube and Distributed Computers, North-Holland, Amsterdam (1989) 279–294. 4. Borgonovo, F., Fratta, L.: Deflection networks: Architectures for metropolitan and wide area networks. Comput. Networks ISDN Syst. 2 (1992) 171–183 5. Bononi, A., Pruncal, P.R.: Analytical evaluation of improved access techniques in deflection routing networks. IEEE/ACM Trans. on Networking 4(5) (1996) 726– 730 6. Chich, T.: Optimisation du routage par d´eflexion dans les r´eseaux de t´elecommunications m´etropolitains. Ph.D. Thesis. LRI Paris (1997) 7. Chich, T.; Cohen, J.; Fraigniaud, P.: Unslotted deflection routing: a practical and efficient protocol for multihop optical networks. IEEE/ACM Trans. on Networking 9 (2001) 47–59 8. Cl´erot., F.: R´eseaux fonctionnant par d´eflexion: deux ans d´ej` a. . . Internal Research Rapport. CNET (1996) 9. F` abrega, J.; Mu˜ noz,X.: How to find the maximum admisible throughput in deflection routing networks. Internal Research Rapport. MA4. UPC (2002) 10. Gary, S.-H., Kobayashi, H.: Performance analysis of Shufflenet with deflection routing. Proceedings of IEEE Globecom 93 2 (1993) 854–859 11. Greenberg, A.G.; Hajek, B.: Deflection routing in hypercube networks. IEEE Trans. Comm. 40 (1992) 1070–1081 12. Heydemann, M.-C., Meyer,J. C., Sotteau, D.: On forwarding indices of networks. Disc. Appl. Math. 23 (1989) 103–123 13. Homobono, N.: Resisstance aux pannes des grands r´eseaux d’interconnexion. Ph.D. Thesis. LRI Paris (1987) 14. Ortega. D.: Simulaci´ on de encaminamientos por deflexi´ on sobre algunas familias de digrafos. Master Thesis. UPC. Barcelona (2000)
Implementation and Performance Evaluation of M-VIA on AceNIC Gigabit Ethernet Card In-Su Yoon1 , Sang-Hwa Chung1 , Ben Lee2 , and Hyuk-Chul Kwon1 1
Pusan National University School of Electrical and Computer Engineering Pusan, 609-735, Korea {isyoon, shchung, hckwon}@pusan.ac.kr 2 Oregon State University Electrical and Computer Engineering Department Owen Hall 302, Corvallis, OR 97331
[email protected] Abstract. This paper describes the implementation and performance of M-VIA on the AceNIC Gigabit Ethernet card. The AceNIC adapter has several notable hardware features for high-speed communication, such as jumbo frames and interrupt coalescing. The M-VIA performance characteristics were measured and evaluated based on these hardware features. Our results show that latency and bandwidth improvement can be obtained when the M-VIA data segmentation size is properly adjusted to utilize the AceNIC’s jumbo frame feature. The M-VIA data segmentation size of 4,096 bytes with MTU size of 4,138 bytes showed the best performance. However, larger MTU sizes did not necessarily result in better performance due to extra segmentation and DMA setup overhead. In addition, the cost of M-VIA interrupt handling can be reduced with AceNIC’s hardware interrupt coalescing. When the parameters for the hardware interrupt coalescing were properly adjusted, the latency of interrupt handling was reduced by up to 170 µs.
1
Introduction
Gigabit Ethernet based clusters are considered as scalable, cost-effective platforms for high performance computing. However, the performance of Gigabit Ethernet has not been fully delivered to the application layer because of the TCP/IP protocol stack overhead. In order to circumvent these problems, a group of user-level communication protocols has been proposed. Examples of user-level communication protocol are U-Net [1], Fast Message [2], Active Message [3] and GAMMA [4]. The Virtual Interface Architecture (VIA) [5] has emerged to standardize these different user-level communication protocols. Since the introduction of VIA, there have been several software and hardware implementations of
This work has been supported by Korean Science and Engineering Foundation (Contract Number: R05-2003-000-10726-0) and National Research Laboratory Program (Contract Number: M10203000028-02J0000-01510).
H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 995–1000, 2003. c Springer-Verlag Berlin Heidelberg 2003
996
I.-S. Yoon et al.
VIA. M-VIA (Modular VIA) [6] is a software implementation that employs Fast or Gigabit Ethernet as the underlying platform. This paper discusses the implementation of M-VIA on the AceNIC Gigabit Ethernet card. The AceNIC Gigabit Ethernet card has several notable hardware features which are jumbo frames and interrupt coalescing. Therefore, this paper presents a study of what effects jumbo frames and interrupt coalescing features have on the performance of M-VIA.
2
M-VIA Overview
M-VIA is implemented as a user-level library and at least two loadable kernel modules for Linux. The core module (via ka module) is device-independent and provides the majority of functionality needed by VIA. M-VIA device drivers implement device-specific functionality. In M-VIA device drivers, the via ering module includes operations such as construction and interpretation of mediaspecific VIA headers and mechanisms for enabling VIA to co-exist with traditional networking protocols, i.e., TCP/IP. In this paper, we present our implementation of M-VIA on the AceNIC by developing a new AceNIC driver module (via acenic module) for the M-VIA. Also, the via ering module was modified to support different M-VIA segmentation sizes.
3 3.1
AceNIC Hardware Features Jumbo Frames
Although jumbo frames are available to transfer large data, the original MVIA segmentation size was designed to support the standard Ethernet MTU size of 1,514 bytes. When M-VIA transfers data, the via ering module organizes data into pages and then each page is divided into the M-VIA segmentation size. Then, the via acenic module writes the physical address, length, and other information of each data segment on the AceNIC’s descriptor. Finally, each data segment is transferred to the AceNIC’s buffer via DMA. Since segmentation and DMA setup require substantial amount of processing time, it is important to reduce the number of data segments. In our implementation, the M-VIA segmentation size was adjusted to utilize AceNIC’s jumbo frame feature. M-VIA segmentation size of 8,958 bytes is obtained by subtracting M-VIA data header of 42 bytes from an MTU of 9,000 bytes. With large MTU and segmentation size, the number of M-VIA packets is significantly reduced. When M-VIA segmentation size is made equal to the page size, the via ering module needs to only generate one segment for each page. In this case, we can use an MTU of 4,138 bytes, which is obtained by adding a data header to M-VIA segment size of 4,096 bytes.
Implementation and Performance Evaluation of M-VIA
3.2
997
Interrupt Coalescing
Interrupt coalescing delays the generation of an interrupt until a number of packets arrive. The waiting time before generating interrupt is controlled by setting the AceNIC’s internal timer. The internal timer starts counting clock ticks from the time when the first packet arrives. The number of coalescing clock ticks can be changed by modifying module parameters. With the hardware interrupt coalescing, the host can amortize the interrupt handling cost over a number of packets and thus save host processing cycles. Because many Ethernet cards do not support hardware interrupt coalescing, M-VIA implements this feature in software. When M-VIA sends intermediate data segments, it does not mark the completion flags of the descriptors, which are ignored by the interrupt handler. When the completion flag of the final descriptor is marked, it indicates the completion of the interrupt coalescing. MVIA’s software interrupt coalescing conflicts with AceNIC’s because an interrupt can be generated by the expired timer before M-VIA marks the completion flag on the final descriptor. Therefore, we maximized the number of coalescing clock ticks to prevent interrupts from being generated during send operations. However M-VIA does not implement the interrupt coalescing when it receives data, and instead depends entirely on the receive interrupt handler of NIC. The cost of M-VIA’s receive interrupt handling is reduced using the AceNIC’s interrupt coalescing.
4
Experimental Results
The performance of M-VIA was measured using 800 MHz Pentium III PCs with 256 MB SDRAM and 33 MHz/32-bit PCI bus. The AceNIC Gigabit Ethernet card used was a 3Com 3C985B-SX. The PCs were running with Linux kernel 2.4. M-VIA latency and bandwidth were measured using vnettest program. 4.1
M-VIA Data Segmentation Size and Hardware MTU
Figure 1 shows the performance of M-VIA with various segmentation sizes. One interesting observation is that M-VIA segmentation size of 4,096 bytes shows a sawtooth shape of the curve. This is because fewer packets are generated when the data size is multiples of the page size. For M-VIA segmentation size of 8,192 bytes, the performance is worse than that of 4,096-byte case until the data size reaches approximately 50 KB. Although AceNIC was configured to carry 8,192-byte frames, the via ering module segments data by page size. Therefore, an 8,192-byte frame requires two segmentations and DMA initiation processes resulting in extra overhead. However, the receiving side can benefit from larger MTUs for bulk data due to reduced number of interrupts. The segmentation size of 8,958 bytes shows even worse performance because of extra segmentation and DMA initiation costs. For data size from 8 KB to approximately 36 KB, 8,958-byte segment size resulted in even worse performance compared to the
998
I.-S. Yoon et al.
1,472-byte case. This is due to the fact that small frames allow packets to be sent faster to the receiving side, while large frames have to wait until they are filled up. However, for data size larger than 36 KB, 8,958-byte case performs better because small frames generate more frequent interrupts on the receiving side.
Fig. 1. Performance with various M-VIA segmentation sizes
4.2
Hardware Interrupt Coalescing Feature
The interrupt coalescing of AceNIC is controlled by a pair of parameters (tx coal, rx coal ), which specify the number of coalescing clock ticks for transmission and reception, respectively. These parameters indicate the duration of packet send/receive before interrupting the host. Figure 2 shows the M-VIA latency for AceNIC using 1,514-byte MTU. When the parameters are set to (1, 1), the interrupt coalescing is disabled for both transmission and reception. Thus, AceNIC invokes the interrupt handler routine as soon as a packet arrives. This leads to a minimum latency of 67 µs for a 32-byte data, but results in significantly longer latencies for larger messages. The maximum latency difference between (1, 1) and (2000, 90) is approximately 400 µs for 64 KB data. To evaluate the latency of the M-VIA’s software interrupt coalescing, the parameters were set to (2000, 1). The tx coal value of 2,000 is sufficient for M-VIA to complete its transmit operations before the expired timer generates an interrupt. This results in significantly lower latency than disabling the interrupt coalescing. The approximate minimum and maximum latency differences are 2 µs and 212 µs, respectively. To confirm that the hardware interrupt coalescing for receiving data improves performance, experiments with parameters set to (2000, 90) were performed. The value 90 was determined experimentally to give the best performance. For data sizes larger than 17 KB, lower latencies were observed compared to when the parameters were set to
Implementation and Performance Evaluation of M-VIA
999
(2000,1). The maximum latency difference between (2000, 90) and (2000, 1) is approximately 170 µs for sending a 64 KB data. For data sizes smaller than 17 KB, slightly higher latencies were observed because of the increased waiting time on the receiving side, but the difference was negligible.
Fig. 2. M-VIA latency with hardware interrupt coalescing feature
Fig. 3. M-VIA vs. TCP/IP
4.3
Comparison of M-VIA and TCP/IP
Figure 3 shows a comparison between M-VIA and TCP/IP in terms of latency and bandwidth. In this experiment, the 8,192-byte and 8,958-byte segment sizes were excluded because 4,096-byte segment size resulted in better performance. When both M-VIA and TCP/IP use the same MTU size of 1,514 bytes, M-VIA has lower latency than TCP/IP. M-VIA and TCP/IP have minimum latencies
1000
I.-S. Yoon et al.
of 89 µs and 123 µs respectively. The minimum latency difference is 15 µs with 4 KB data. For data sizes larger than 16 KB, the latency difference is approximately 76 µs. M-VIA and TCP/IP have maximum bandwidths of 60.9 MB/s and 56.9 MB/s, respectively. Comparing M-VIA using MTU size of 4,138 bytes with TCP/IP, the minimum latency difference is 57 µs with 4 KB data and the maximum latency difference is 246 µs with 64 KB data. M-VIA has a maximum bandwidth of 72.5 MB/s with segmentation size of 4096 bytes.
5
Conclusion
We presented our implementation and performance study of M-VIA on the AceNIC Gigabit Ethernet card by developing a new AceNIC driver for M-VIA. In particular, we focused on AceNIC’s jumbo frame and interrupt coalescing features for M-VIA. We experimented with the various M-VIA data segmentation sizes and MTUs. The M-VIA data segmentation size of 4,096 bytes with MTU size of 4,138 bytes showed the best performance. Comparing M-VIA using MTU size of 4,138 bytes with TCP/IP, M-VIA latency improves by approximately 57∼246 µs and results in maximum bandwidth of 72.5 MB/s. Also the latency time of M-VIA’s interrupt handling was reduced by up to 170 µs with the AceNIC’s hardware interrupt coalescing.
References 1. T. Von Eicken, A. Basu, V. Buch, and W. Vogels: “U-NET: A User Level Network Interface for Parallel and Distributed Computing”, Proc. of the 15th ACM Symposium on Operating Systems Principles (SOSP), Colorado, December 1995 2. S. Pakin, M. Lauria, and A. Chien: “High Performance Messaging on Workstation: Illinois Fast Message (FM) for Myrinet”, Proc. of Supercomputing ’95, December 3. T. von Eicken, D. E. Culler, S. C. Goldstein, and K. E. Schauser: “Active Messages: a Mechanism for Integrated Communication and Computation”, 19th International Symposium on Computer Architecture, May 1992 4. G. Chiola and G. Ciaccio: “GAMMA: a Low-cost Network of Workstations Based on Active Messages”, Proc. of 5th EUROMICRO workshop on Parallel and Distributed Processing, London, UK, January 1997 5. Intel, Compaq and Microsoft Corporations: Virtual Interface Architecture specification version 1.0, December 1997, http://developer.intel.com/design/servers/vi/ 6. P. Bozeman and B. Saphir: “A Modular High Performance Implementation of the Virtual Interface Architecture”, Proc. of the 2nd Extreme Linux Workshop, June 1999
Topic 15 Mobile and Ubiquitous Computing Max M¨ uhlh¨ auser, Karin A. Hummel, Azzedine Boukerche, and Alois Ferscha Topic Chairs
The growing availability of mobile and wireless network infrastructures and, at the same time, the development of powerful and smart mobile computing devices opens up new possibilities and challenges, but also poses new issues for distributed systems and applications. One of primary issues of ubiquitous computing appears to be the integration of the potentials of mobile computing, context-awareness and seamless connectivity of services and devices — one of its primary aims is the provision of an omnipresent but calm computing landscape, empowering people with smart assistance and intelligent guidance. We formed this workshop to reflect on the evident changes of research interests from parallel and distributed computing towards mobile computing and ubiquitous computing issues. We framed the call for papers text to solicit papers contributing to one or more of following topics: mobility, ubiquity, awareness, intelligence, and natural interaction. Mobility addresses solutions that help to make time, geographic, media and service boundaries less and less important. Ubiquity refers to a situation in which we are surrounded by a multitude of interconnected embedded systems, which are (mostly) invisible and moved into the background of our surrounding (workplace, building, home, outdoor). Awareness refers to the ability of the system to recognise and localise objects as well as people and their intentions. Intelligence refers to the abality of an environment to adapt to the people that live in it, learn from their behaviour, and possibly recognise as well as show emotion. Natural interaction finally refers to advanced modalities like natural speech- and gesture recognition, as well as speech-synthesis, which will allow a much more human-like communication with the digital environment than is possible today. The papers selected — after a rigorous reviewing process — for publication in Topic 15 all address problems ultimately related to the design and development of ubiquitous computing systems: Related to networking, a few contributions study enhanced mobility management and QoS strategies, like improved routing protocols and mobile IP optimization and utilization. Contributions addressing embedded and context aware services introduce new models, frameworks and software architectures which incorporate sensor interfaces. Furthermore, models for context-aware behavior are presented as well as middleware adaptations due to mobile coordination patterns. Location and position tracking methods are discussed based on outdoor technologies, like mobile communication infrastructures and triangulation protocols, and indoor radio frequency technologies. Rewarding results are presented in terms of accuracy for different outdoor locaH. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 1001–1002, 2003. c Springer-Verlag Berlin Heidelberg 2003
1002
M. M¨ uhlh¨ auser et al.
tion estimation methods. In order to argue the new possibilities of ubiquitous computing, various contributions present attracting real-world demonstrations and applications. This is the first year that “mobile and ubiquitous computing” has been solicited from within Euro-Par, and the number of submissions indicate an overwhelming appreciation of the topic. In numbers, 39 papers have been submitted to the topic. Out of these contributions, 14 have been accepted as regular papers [36%] and 4 as short papers [10%] based on the results of three referees. We owe special thanks to the Euro-Par Organizing Committee for their efforts and support, as well as to the referees providing detailed and helpful comments. We would also like to thank all the authors who have considered Euro-Par as the outlet for their valuable research work, and would at the same time like to encourage potential authors for this consideration for then next Euro-Par.
A Comparative Study of Protocols for Efficient Data Propagation in Smart Dust Networks I. Chatzigiannakis1,2 , T. Dimitriou3 , M. Mavronicolas4 , S. Nikoletseas1,2 , and P. Spirakis1,2 1
Computer Technology Institute, P.O. Box 1122, 26110 Patras, Greece {ichatz,nikole,spirakis}@cti.gr 2 Department of Computer Engineering and Informatics, University of Patras, 26500 Patras, Greece 3 Athens Information Technology (AIT)
[email protected] 4 Department of Computer Science, University of Cyprus, CY-1678 Nicosia, Greece
[email protected] Abstract. Smart Dust is comprised of a vast number of ultra-small fully autonomous computing and communication devices, with very restricted energy and computing capabilities, that co-operate to accomplish a large sensing task. Smart Dust can be very useful in practice i.e. in the local detection of a remote crucial event and the propagation of data reporting its realization to a control center. In this work, we have implemented and experimentally evaluated four protocols (PFR, LTP and two variations of LTP which we here introduce) for local detection and propagation in smart dust networks, under new, more general and realistic modelling assumptions. We comparatively study, by using extensive experiments, their behavior highlighting their relative advantages and disadvantages. All protocols are very successful. In the setting we considered here, PFR seems to be faster while the LTP based protocols are more energy efficient.
1
Introduction, Our Results, and Related Work
Recent dramatic developments in micro-electro-mechanical (MEMS) systems, wireless communications and digital electronics have already led to the development of small in size, low-power, low-cost sensor devices. Such extremely small devices integrate sensing, data processing and communication capabilities. Examining each such device individually might appear to have small utility, however the effective distributed co-ordination of large numbers of such devices may lead to the efficient accomplishment of large sensing tasks.
This work has been partially supported by the IST Programme of the European Union under contract numbers IST-1999-14186 (ALCOM-FT) and IST-2001-33116 (FLAGS).
H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 1003–1016, 2003. c Springer-Verlag Berlin Heidelberg 2003
1004
I. Chatzigiannakis et al.
Large numbers of sensor nodes can be deployed in areas of interest (such as inaccessible terrains or disaster places) and use self-organization and collaborative methods to form a sensor network. Their wide range of applications is based on the possible use of various sensor types (i.e. thermal, visual, seismic, acoustic, radar, magnetic, etc.) in order to monitor a wide variety of conditions (e.g. temperature, object presence and movement, humidity, pressure, noise levels etc.). Thus, sensor networks can be used for continuous sensing, event detection, location sensing as well as micro-sensing. Hence, sensor networks have important applications, including (a) military (like forces and equipment monitoring, battlefield surveillance, targeting, nuclear, biological and chemical attack detection), (b) environmental applications (such as fire detection, flood detection, precision agriculture), (c) health applications (like telemonitoring of human physiological data) and (d) home applications (e.g. smart environments and home automation). For an excellent survey of wireless sensor networks see [1] and also [6, 12]. Note however that the efficient and robust realization of such large, highlydynamic, complex, non-conventional networking environments is a challenging algorithmic and technological task. Features including the huge number of sensor devices involved, the severe power, computational and memory limitations, their dense deployment and frequent failures, pose new design and implementation aspects which are essentially different not only with respect to distributed computing and systems approaches but also to ad-hoc networking techniques. Contribution: We focus on an important problem under a particular model of sensor networks that we present. More specifically, continuing the research of our team (see [2], [4]), we study the problem of local detection and propagation, i.e. the local sensing of a crucial event and the energy and time efficient propagation of data reporting its realization to a (fixed or mobile) control center. This center may be some human authorities responsible of taking action upon the realization of the crucial event. We use the term “sink” for this control center. We note that the protocols we present here can also be used for the more general problem of data propagation in sensor networks (see [10]). As opposed to [5] (where a 2-dimensional lattice deployment of particles is used) we extend the network model to the general case of particle deployment according to a random, uniform distribution. We study here the more realistic case when the control center is not a line in the plane (i.e. as in [2]) but a single point. Under these more general and realistic (in terms of motivation by applications) modelling assumptions, we implemented and experimentally evaluated four information propagation protocols: (a) The Probabilistic Forwarding Protocol (PFR) that avoids flooding by favoring in a probabilistic way certain “close to optimal” transmissions, (b) the Local Target Protocol (LTP), where data is propagated by each time selecting the best (with respect to progress towards the sink) particle for passing information and (c) we propose two variations of LTP according to different next particle selection criteria. We note that we had to carefully design the protocols to work under the new network models.
A Comparative Study of Protocols for Efficient Data Propagation
1005
The extensive simulations we have performed show that all protocols are very successful. In the setting we considered here, PFR seems to achieve high success rates in terms of time and hops efficiency, while the LTP based protocols manage to reduce the energy spent in the process by activating less particles. Discussion of Selected Related Work: A family of negotiation-based information dissemination protocols suitable for wireless sensor networks is presented in [9]. In contrast to classic flooding, in SPIN sensors negotiate with each other about the data they possess using meta-data names. These negotiations ensure that nodes only transmit data when necessary, reducing the energy consumption for useless transmissions. A data dissemination paradigm called directed diffusion for sensor networks is presented in [10]. An observer requests data by sending interests for named data; data matching the interest is then “drawn” down towards that node by selecting a single path or through multiple paths by using a low-latency tree. [11] presents an alternative approach that constructs a greedy incremental tree that is more energy-efficient and improves path sharing. In [8] a clustering-based protocol is given that utilizes randomized rotation of local cluster heads to evenly distribute the energy load among the sensors in the network. In [15] a new energy efficient routing protocol is introduced that does not provide periodic data monitoring (as in [8]), but instead nodes transmit data only when sudden and drastic changes are sensed by the nodes. As such, this protocol is well suited for time critical applications and compared to [8] achieves less energy consumption and response time. We note that, as opposed to the work presented in this paper, the above research focuses on energy consumption without examining the time efficiency of their protocols. Furthermore, note that our protocols are quite general in the sense that (a) they do not assume global network topology information, (b) do not assume geolocation information (such as GPS information) and (c) use very limited control message exchanges, thus having low communication overhead.
2
The Model
Sensor networks are comprised of a vast number of ultra-small homogenous sensors, which we call “grain” particles. Each grain particle is a fully-autonomous computing and communication device, characterized mainly by its available power supply (battery) and the energy cost of computation and transmission of data. Such particles (in our model here) cannot move. Each particle is equipped with a set of monitors (sensors) for light, pressure, humidity, temperature etc. Each particle has a broadcast (digital radio) beacon mode which can be also a directed transmission of angle α around a certain line (possibly using some special kind of antenna, see Fig. 2). We adopt here (as a starting point) a two-dimensional (plane) framework: A smart dust cloud (a set of particles) is spread in an area (see Fig. 1). Note that a two-dimensional setting is also used in [8,9,10,11,15].
1006
I. Chatzigiannakis et al.
Definition 1. Let n be the number of smart dust particles and let d (usually measured in numbers of particles/m2 ) be the density of particles in the area. Let R be the maximum (beacon/laser) transmission range of each grain particle. There is a single point in the network area, which we call the sink S, and represents a control center where data should be propagated to. Note that, although in the basic case we assume the sink to be static, in a variation it may be allowed to move around its initial base position, to possibly get data that failed to reach it but made it close enough to it. Furthermore, we assume that there is a set-up phase of the smart dust network, during which the smart cloud is dropped in the terrain of interest, when using special control messages (which are very short, cheap and transmitted only once) each smart dust particle is provided with the direction of S. By assuming that each smart-dust particle has individually a sense of direction (e.g. through its magnetometer sensor), and using these control messages, each particle is aware of the general location of S. We feel that our model, although simple, depicts accurately enough the technological specifications of real smart dust systems. Similar models are being used by other researchers in order to study sensor networks (see [8,15]). In contrast to [10,13], our model is weaker in the sense that no geolocation abilities are assumed (e.g. a GPS device) for the smart dust particles leading to more generic and thus stronger results. In [7] a thorough comparative study and description of smart dust systems is given, from the technological point of view.
Fig. 1. A Smart Dust Cloud
Fig. 2. Example of the Search Phase
A Comparative Study of Protocols for Efficient Data Propagation
3
1007
The Problem
Assume that a single particle, p, senses the realization of a local crucial event E. Then the propagation problem P is the following: “How can particle p, via cooperation with the rest of the grain particles, efficiently propagate information inf o(E) reporting realization of event E to the sink S?” Because of the dense deployment of particles close to each other, multi-hop communication consumes less power than a traditional single hop propagation. Also, multi-hop communication can effectively overcome some of the signal propagation effects in long-distance wireless transmissions. Furthermore, short-range hop-by-hop transmissions may help to smoothly adjust propagation around obstacles. Finally, the low energy transmission in hop-by-hop may enhance security, protecting from undesired discovery of the data propagation operation. Because of the above reasons, data propagation should be done in a multihop way. To minimize the energy consumption in the sensor network we wish to minimize the number of hops (directed transmissions) performed in the data propagation process. Note that the number of hops also characterizes propagation time, assuming an appropriate MAC protocol [18]. Furthermore, an interesting aspect is how close to the sink data is propagated (in the case where data does not exactly reach the sink). Note that proximity to the sink might be useful in the case where the sink is mobile or it performs a “limited flooding” to get the information from the final position it arrived.
4
The Probabilistic Forwarding Protocol (PFR)
The basic idea of the protocol lies in probabilistically favoring transmissions towards the sink within a thin zone of particles around the line connecting the particle sensing the event E and the sink (see Fig. 3). Note that transmission along this line is optimal w.r.t. energy and time. However it is not always possible to achieve this optimality, basically because, even if initially this direct line was appropriately occupied by sensors, certain sensors on this line might become inactive, either permanently (because their energy has been exhausted) or temporarily (because they might enter a sleeping mode to save energy). Further reasons include (a) physical damage of sensors, (b) deliberate removal of some of them (possibly by an adversary), (c) changes in the position of the sensors due to a variety of reasons (weather conditions, human interaction etc). and (d) physical obstacles blocking communication. The protocol evolves in two phases: Phase 1: The “Front” Creation Phase. Because of the probabilistic decision on whether to forward messages or not, initially we build (by using flooding) a sufficiently large “front” of particles, in order to guarantee the survivability of the propagation process. During this phase,
1008
I. Chatzigiannakis et al.
Fig. 3. Thin zone of particles around the line connecting the particle sensing the event E and the sink S.
Fig. 4. Angle φ and closeness to optimal line
each particle having received the data, deterministically forwards them towards the sink. In particular, and for a sufficiently large number of steps, each particle broadcasts the information to all its neighbors, using a directed (towards the sink) angle transmission. Phase 2: The Probabilistic Forwarding Phase. During this phase, each particle possessing the information under propagation, probabilistically favors its transmission within a thin zone of sensors lying close to the (optimal) line between the particle that sensed E and S. In other words, data is propagated with a suitably chosen probability p, while it is not propagated with probability 1 − p, based on a random choice. This probability is calculated as follows: Let φ the angle defined by the line connecting E and the sensor performing the random choice, and the line defined by the position of this particle and S (see Fig. 4). To limit the propagation zone we choose the forwarding probability IPf wd to be φ IPf wd = (1) π Remark that indeed a bigger angle φ suggests a sensor position closer to the direct line between E and S. Clearly, when φ = π, then the sensor lies on this line. Also note that calculations of φ needs only local information. Figure 4 displays this graphically.
A Comparative Study of Protocols for Efficient Data Propagation
1009
Thus, we get that φ1 > φ2 implies that for the corresponding particles p1 , p2 , p1 is closer to the E-S line than p2 , thus φ2 φ1 > = IPf wd (p2 ) (2) π π Remark: Certainly, there might exist other probability choices for favoring certain transmissions. We are currently working on such alternative choices. IPf wd (p1 ) =
5
The Local Target Protocol (LTP) and Its Variations
We now present a protocol for smart dust networks which we call the “Local Target” protocol and two variations that use different next particle selection criteria. In this protocol, each particle p that has received info(E) from p (via, possibly, other particles) does the following: Phase 1: The Search Phase. It uses a periodic low energy broadcast of a beacon in order to discover a particle nearer to S than itself. Among the particles returned, p selects a unique particle p that is “best” with respect to progress towards the sink. We here consider two criteria for measuring the quality of this progress and also a randomized version to get a good average mix: (a) Euclidean Distance. In this case, the particle pE that among all particles found achieves the bigger progress on the p S line, should be selected. We call this variation of the our protocol LTPE . This is considered as the “basis” LTP. (b) Angle Optimality. In this case, the particle pA such that the angle pA p S is minimum, should be selected. We call this variation of the our protocol LTPA . (c) Randomization. Towards a good average case performance of the protocol, we use randomization to avoid bad behavior due to the worst case input distributions for each selection (i.e. particles forming big angles with the optimal line in LTPE and particles resulting to small Euclidean progress in LTPA ). Thus, we find pE , pA and randomly select one of them, with probability 12 . We call this protocol LTPR . Phase 2: The Direct Transmission Phase. Then, p sends info(E) to p and sends a success message to p (i.e. to the particle that it originally received the information from). Phase 3: The Backtrack Phase. If the search phase fails to discover a particle nearer to S, then p sends a fail message to p. In the above procedure, propagation of info(E) is done in two steps; (i) particle p locates the next particle (p ) and transmits the information and (ii) particle p waits until the next particle (p ) succeeds in propagating the message further towards S. This is done to speed up the backtrack phase in case p does not succeed in discovering a particle nearer to S.
1010
6
I. Chatzigiannakis et al.
Efficiency Measures
Definition 2. Let hA (for “active”) be the number of “active” sensor particles participating in the data propagation and let ET R be the total number of data transmissions during propagation. Let T be the total time for the propagation process to reach its final position and H the total number of “hops” required. Clearly, by minimizing hA we succeed in avoiding flooding and thus we minimize energy consumption. Remark that in LTP we count as active those particles that transmit info(E) at least once. Note that hA , ET R , T and H are random variables. Furthermore, we define the success probability of our algorithm where we call success the eventual data propagation to the sink. Definition 3. Let IPs be the success probability of our protocol. We also focus on the study of the following parameter. Suppose that the data propagation fails to reach the sink. In this case, it is very important to know “how close” to the sink it managed to get. Propagation reaching close to the sink might be very useful, since the sink (which can be assumed to be mobile) could itself move (possibly by performing a random walk) to the final point of propagation and get the desired data from there. Even assuming a fixed sink, closeness to it is important, since the sink might in this case begin some “limited” flooding to get to where data propagation stopped. Clearly, the closer to the sink we get, the cheaper this flooding becomes. Definition 4. Let F be the final position of the data propagation process. Let D be F’s (Euclidean) distance from the sink S. Clearly in the case of total success F coincides with the sink and D = 0.
7
Experimental Evaluation
We evaluate the performance of the four protocols by a comparative experimental study. The protocols have been implemented as C++ classes using the data types for two-dimensional geometry of LEDA [16] based on the environment developed in [2], [4]. Each class is installed in an environment that generates sensor fields given some parameters (such as the area of the field, the distribution function used to drop the particles), and performs a network simulation for a given number of repetitions, a fixed number of particles and certain protocol parameters. After the execution, the environment stores the results in files so that the measurements can be represented in a graphical way. In the full paper ([3]), we provide for all protocols a more detailed description of the implementation, including message structures, data structures at a particle, initialization issues and pseudo-code for the protocol. In our experiments, we generate a variety of sensor fields in a 100m by 100m square. In these fields, we drop n ∈ [100, 3000] particles randomly uniformly
A Comparative Study of Protocols for Efficient Data Propagation
1011
distributed on the smart-dust plane, i.e. for densities 0.01 ≤ d ≤ 0.3. Each smart dust particle has a fixed radio range of R = 5m and α = 90o . The particle p that initially senses the crucial event is always explicitly positioned at (x, y) = (0, 0) and the sink is located at (x, y) = (100, 100). Note that this experimental setup is based on and extends that used in [8,11,15]. We repeated each experiment for more than 5,000 times in order to achieve good average results.
1 0.9
Success Rate (Psuccess)
0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0.01
0.03
0.05
0.07
0.09
0.11
0.13
0.15
0.17
0.19
0.21
0.23
0.25
0.27
0.29
Particle Density (d) PFR
LTPe
LTPa
LTPr
Fig. 5. Success Probability (IPs ) over particle density d = [0.01, 0.3].
We start by examining the success rate of the four protocols (see Fig. 5), for different particle densities. Initially, when the density is low (i.e. d ≤ 0.06), the protocols fail to propagate the data to the sink. However as the density increases, the success rate increases quite fast and for high densities, all four protocols almost always succeed in propagating the data to the sink. Thus, all protocols are very successful. We remark a similar shape of the success rate function in terms of density. This is due to the fact that all protocols use local information to decide how to proceed by basically selecting (all protocols) the next particle with respect to a similar criterion (best progress towards the sink). In the case when the protocols fail to propagate the data to the sink, we examine “how close” to the sink they managed to get. Figure 6 depicts the distance of the final point of propagation to the position of the sink. Note that this figure should be considered in conjunction with Fig. 5 on the success rate. Indeed, failures to reach the sink are very rare and seem to appear in very extreme network topologies due to bad particle distribution on the network area. Figure 7 depicts the ratio of active particles over the total number of particles (r = hnA ) that make up the sensor network. In this figure we clearly see that PFR, for low densities (i.e. d ≤ 0.07), it indeed activates a small number of particles (i.e. r ≤ 0.3) while the ratio (r) increases as the density of the particles increases.
1012
I. Chatzigiannakis et al.
140
120
Distance to Sink (D)
100
80
60
40
20
0 0.01
0.03
0.05
0.07
0.09
0.11
0.13
0.15
0.17
0.19
0.21
0.23
0.25
0.27
0.29
Particle Density (d)
PFR
LTPe
LTPa
LTPr
Fig. 6. Average Distance from the Sink (D) over particle density d = [0.01, 0.3].
1
Ratio of Active Particles over Total Particles (r)
0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0.01
0.03
0.05
0.07
0.09
0.11
0.13
0.15
0.17
0.19
0.21
0.23
0.25
0.27
0.29
Particle Density (d) PFR
LTPe
LTPa
LTPr
Fig. 7. Ratio of Active Particles over Total Particles (r) over particle density d = [0.01, 0.3].
The LTP based protocols do not behave in the same way, since the ratio (r) of active particles seems to be independent to the total number of particles. Remark: Because of the way PFR attempts to avoid flooding (by using angle φ to capture “distance” from optimality) its merits are not sufficiently shown in the setting considered here. We expect PFR to behave significantly better with respect to energy in much larger networks and in cases where the event is
A Comparative Study of Protocols for Efficient Data Propagation
1013
sensed in an average α place of the network. Also, stronger probabilistic choices φ (i.e. IPf wd = π , where a > 1 a constant) may further limit propagation time. Furthermore, examining the total number of transmissions performed by the particles (see Fig. 8), it is evident that because the LTP based protocols activate a small number of particles, the overall transmissions are kept low. This is a surprising result, since the PFR protocol was originally designed to work without the need of any control messages so that the energy consumption is low. However, the comparative study clearly shows that avoiding the use of control messages does not achieve the expected results. So, even though all four protocols succeed in propagating the data, it is evident that the LTP based protocols are more energy efficient in the sense that less particles are involved in the process. We continue with the following two parameters: (a) the “hops” efficiency and (b) the time efficiency, measured in terms of rounds needed to reach the sink. As can be seen in Fig. 9, all protocols are very efficient, in the sense that the number of hops required to get to the sink tends below 40 even for densities d ≥ 0.17. The value 40 in our setting is close √ to optimality since in an ideal placement, the diagonal line is of length 100 2 and since the transmission range R = 5 the optimal number of hops (in an ideal case) is roughly 29. In particular PFR achieves this for very low densities (d ≥ 0.07). On the other hand, the LTP based protocols exhibit a certain pathological behavior for low densities (i.e. d ≤ 0.12) due to a high number of executions of the backtrack mechanism in the attempt to find a particle closer to the sink (see also Fig. 10). As far as time is concerned, in the full paper [3] we demonstrate (see Fig. 10 there) that time and number of hops are of the same order and exhibit a very similar behavior.
3000
Average Number of Transmissions (Etr)
2500
2000
1500
1000
500
0 0.01
0.03
0.05
0.07
0.09
0.11
0.13
0.15
0.17
0.19
0.21
0.23
0.25
0.27
0.29
Particle Density (d) PFR
LTPe
LTPa
LTPr
Fig. 8. Average Number of Transmissions (ET R ) over particle density d = [0.01, 0.3].
1014
I. Chatzigiannakis et al.
160
Number of Hops to reach Sink
140
120
100
80
60
40
20
0 0.01
0.03
0.05
0.07
0.09
0.11
0.13
0.15
0.17
0.19
0.21
0.23
0.25
0.27
0.29
Particle Density (d)
PFR
LTPe
LTPa
LTPr
Fig. 9. Average Number of Hops to reach the sink (H) over particle density d = [0.01, 0.3].
60
Number of Backtracks
50
40
30
20
10
0 0.01
0.03
0.05
0.07
0.09
0.11
0.13
0.15
0.17
0.19
0.21
0.23
0.25
0.27
0.29
Particle Density (d) LTPe
LTPa
LTPr
Fig. 10. Average Number of Backtracks over particle density d = [0.01, 0.3].
Finally, in Fig. 10 we compare the three LTP based protocols and the number of backtracks invoked in the the data propagation. It is evident that for very low particle densities (i.e. d ≤ 0.12), all three protocols perform a large number of backtracks in order to find a valid path towards the sink. As the particle density increases, the number of backtrack reduces fast enough and almost reaches zero.
A Comparative Study of Protocols for Efficient Data Propagation
8
1015
Future Work
We plan to also investigate alternative probabilistic choices for favoring certain data transmissions for the PFR protocol and consider alternative backtrack mechanisms for the LTP protocol. Also, we wish to study different network shapes, various distributions used to drop the sensors in the area of interest and the fault-tolerance of the protocols. Finally, we plan to provide performance comparisons with other protocols mentioned in the related work section, as well as implement and evaluate hybrid approaches that combine the PFR and LTP protocols in a parameterized way.
References 1. I.F. Akyildiz, W. Su, Y. Sankarasubramaniam and E. Cayirci: Wireless sensor networks: a survey. In the Journal of Computer Networks, Volume 38, pp. 393– 422, 2002. 2. I. Chatzigiannakis, S. Nikoletseas and P. Spirakis: Smart Dust Protocols for Local Detection and Propagation. In Proc. 2nd ACM Workshop on Principles of Mobile Computing – POMC’2002. 3. I. Chatzigiannakis, T. Dimitriou, M. Mavronicolas S. Nikoletseas and P. Spirakis: A Comparative Study of Protocols for Efficient Data Propagation in Smart Dust Networks. FLAGS, TR, http://ru1.cti.gr/FLAGS. 4. I. Chatzigiannakis and S. Nikoletseas: A Sleep-Awake Protocol for Information Propagation in Smart Dust Networks. In Proc. 3rd Workshop on Mobile and AdHoc Networks (WMAN)–IPDPS Workshops, IEEE Press, p. 225, 2003. 5. I. Chatzigiannakis, T. Dimitriou, S. Nikoletseas and P. Spirakis: A Probabilistic Forwarding Protocol for Efficient Data Propagation in Sensor Networks. FLAGS Technical Report, FLAGS-TR-14, 2003. 6. D. Estrin, R. Govindan, J. Heidemann and S. Kumar: Next Century Challenges: Scalable Coordination in Sensor Networks. In Proc. 5th ACM/IEEE International Conference on Mobile Computing – MOBICOM’1999. 7. S.E.A. Hollar: COTS Dust. Msc. Thesis in Engineering-Mechanical Engineering, University of California, Berkeley, USA, 2000. 8. W. R. Heinzelman, A. Chandrakasan and H. Balakrishnan: Energy-Efficient Communication Protocol for Wireless Microsensor Networks. In Proc. 33rd Hawaii International Conference on System Sciences – HICSS’2000. 9. W. R. Heinzelman, J. Kulik and H. Balakrishnan: Adaptive Protocols for Information Dissemination in Wireless Sensor Networks. In Proc. 5th ACM/IEEE International Conference on Mobile Computing – MOBICOM’1999. 10. C. Intanagonwiwat, R. Govindan and D. Estrin: Directed Diffusion: A Scalable and Robust Communication Paradigm for Sensor Networks. In Proc. 6th ACM/IEEE International Conference on Mobile Computing – MOBICOM’2000. 11. C. Intanagonwiwat, D. Estrin, R. Govindan and J. Heidemann: Impact of Network Density on Data Aggregation in Wireless Sensor Networks. Technical Report 01750, University of Southern California Computer Science Department, November, 2001. 12. J.M. Kahn, R.H. Katz and K.S.J. Pister: Next Century Challenges: Mobile Networking for Smart Dust. In Proc. 5th ACM/IEEE International Conference on Mobile Computing, pp. 271–278, September 1999.
1016
I. Chatzigiannakis et al.
13. B. Karp: Geographic Routing for Wireless Networks. Ph.D. Dissertation, Harvard University, Cambridge, USA, 2000. 14. µ-Adaptive Multi-domain Power aware Sensors: http://www-mtl.mit.edu/research/icsystems/uamps, April, 2001. 15. A. Manjeshwar and D.P. Agrawal: TEEN: A Routing Protocol for Enhanced Efficiency in Wireless Sensor Networks. In Proc. 2nd International Workshop on Parallel and Distributed Computing Issues in Wireless Networks and Mobile Computing, satellite workshop of 16th Annual International Parallel & Distributed Processing Symposium – IPDPS’02. 16. K. Mehlhorn and S. N¨ aher: LEDA: A Platform for Combinatorial and Geometric Computing. Cambridge University Press, 1999. 17. TinyOS: A Component-based OS for the Network Sensor Regime. http://webs.cs.berkeley.edu/tos/, October, 2002. 18. W. Ye, J. Heidemann and D. Estrin: An Energy-Efficient MAC Protocol for Wireless Sensor Networks. In Proc. 12th IEEE International Conference on Computer Networks – INFOCOM’2002. 19. Wireless Integrated Sensor Networks: http:/www.janet.ucla.edu/WINS/, April, 2001.
Network Based Mobile Station Positioning in Metropolitan Area Karl R.P.H. Leung1 , Joseph Kee-Yin Ng2 , Tim K.T. Chan2 , Kenneth M.K. Chu2 , and Chun Hung Li2 1
Department of Information & Communications Technology, Hong Kong Institute of Vocational Education (Tsing Yi), Hong Kong
[email protected] 2 Department of Computer Science, Hong Kong Baptist University, Hong Kong {jng, timchan, mkchu, chli}@comp.hkbu.edu.hk
Abstract. Recently, mobile station positioning is drawing considerable attention in the field of wireless communications. Many location estimation techniques have been proposed. Although location estimation algorithms based on received signal strength technique may not be the most promising approach for providing location services, signal strength is the only common attribute available among various kind of mobile network. In this paper, we report our study of a Crude Estimation Method (CEM) which estimate the location of a mobile station based on the ratio of the signal strengths received from different base transceiver stations. We conducted series of field tests with the networks of two major mobile phone operators of Hong Kong. Among 6120 real world readings obtained from these field tests, the average errors of CEM is 49.03 meter with a variance of 538.14. When comparing the results of CEM with the Center of Gravity Method (CGM), another mobile location estimation also based on signal strength ratio, CEM has an improvement of 33.85% with about the same variance. These results are very encouraging in the study of mobile network based mobile station positioning in metropolitan areas.
1
Introduction
Positioning is the base technology for most location-based applications such as fleet management. Although Global Positioning System (GPS) is a common method used for estimating locations, using GPS in metropolitan areas, like Hong Kong, is not an effective method. It is because satellite signals are often reflected, deflected and blocked by skyscrapers and cause many blind spots when using GPS in metropolitan areas. The U.S. Federal Communication Commission (FCC) requires all cellular operators must be able to estimate the mobile device locations with an accuracy of
The paper is partially supported by the Hong Kong SAR Government Innovation Technology Fund – ITSP under ITS/22/02.
H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 1017–1026, 2003. c Springer-Verlag Berlin Heidelberg 2003
1018
K.R.P.H. Leung et al.
100 meters in 67 percent of the time for the network-based solution when making an emergency call [1]. CDMA mobile phone network architecture is being used in the United States. Hence many methods which make use of the features of the existing CDMA network are studied to perform location estimation. These methods have been utilizing the Time-Of-Arrival (TOA), Angle of Arrival (AOA), Time- Different of Arrival (TDOA) and Signal Strength, for estimating the cellular phone location. However, in European countries and also regions like Hong Kong, most of mobile phone service providers are using Global System of Mobile Network Architecture (GSM), instead of CDMA. Most of the features being used for location estimation in CDMA are either not available or with much poor precision. Using the GSM network for positioning seems to be impractical in many countries. Because of limited information about the TOA, AOA or TDOA, signal strengths received from the serving base station and its neighboring base stations seem to be the only information for location estimation. However, this method has to deal with the large and random deviations of the receiving signals [2]. Second, due to the high penetration power of GSM, in many sparsely populated European cities, only 2 to 3 towers of macro cell sites is sufficient for providing mobile phone services for a huge area. This highly centralized network topology causes more difficulties for location estimation because of too few distinct references. However, cities like Hong Kong and similar metropolitan areas are densely populated with skyscrapers. Hence more cell sites are needed to provide a good telecommunication coverage than in sparsely populated areas. Willassen and Andresen proposed a method for estimating MS location based on GSM network [3]. By mapping signal strength received by the MS to the actual distance, triangulation method was applied to estimate the location of the MS. This method was studied intensively by simulation. However, the method had not been tested in a real GSM network. The HATA formula [4] proposed by Okumura for mapping the signal strength to the actual distance is one the most widely used propagation-path models. The model is based on the empirical measurements of radii 1-10 km in Tokyo. However, variations of terrain factors, transmission frequencies, and cell types of the mobile stations are necessary factors to be considered in this formula. These factors make the calculation complicated, are location dependent, and impose extra maintenance cost to the network operators. Although location estimation algorithms based on signal attenuation may not be the most promising approach for providing location services, signal strength is the only common attribute available among various kind of mobile network. Furthermore, the cell layout in metropolitan areas like Hong Kong is very different. It is our hypothesis that in city areas like Hong Kong, there are methods which can estimate MS locations by signal strengths as accurate as those methods used in CDMA network. A Center of Gravity Method (CGM) has proposed for estimating GSM mobile station locations [5] which come up with encouraging results. In the paper, we report our Crude Estimation Method (CEM). This is a mobile network based mobile station positioning method which makes use of
Network Based Mobile Station Positioning in Metropolitan Area
1019
the received signal strengths received by the mobile station. We also give a brief summary of the Weighted Center of Gravity Method in Section 2. This is the method which we are going to compare with extensively. Then the Crude Estimation Method is described in Section 3. The design of field tests is discussed in Section 4 which is followed by the experiment and discussion of experimental results in Section 5. Finally, we conclude our work in Section 6.
2
Weighted Center of Gravity Method
A Center of Gravity Method (CGM) is proposed for estimating locations of GSM handsets [5]. Theoretically, the relationship between the distance and the transmitted signal strength is based on the inverse square law. Thus, the weighting assignment function should be 1/x2 . However, the terrain and the interference distort the law with high density of buildings in the area like Hong Kong. So the weighting function becomes 1/xα . With the concept of Center of Gravity and the weighting function, the algorithm of Weighted CG is constructed for the location estimation. By using the reciprocal function of the signal strengths as a weighting, the location estimation formula can be defined as follows: Let x, y be the xy-coordinate of the mobile phone, and (x1 , y1 ), (x2 , y2 ) , ... , (xn , yn ) the actual position(s) of the BTSs1 . With n being the number of reference points, which should be less than the number of received cell sites. dB1 , dB2 ,... dBn represent the corresponding signal strength for each BTS. Thus, the location estimation formula for x, and y are: x=
x1 /dB1α + x2 /dB2α + x3 /dB3α + ... + xn /dBnα 1/dB1α + 1/dB2α + 1/dB3α + ... + 1/dBnα
(1)
y=
y1 /dB1α + y2 /dB2α + y3 /dB3α + ... + yn /dBnα 1/dB1α + 1/dB2α + 1/dB3α + ... + 1/dBnα
(2)
where 3 ≤ n ≤ 9 While 3 is the minimum number of channels for the algorithm to work, and 9 is the maximum number of channels a Nokia handset2 can provide. The major disadvantage of CGM is, due to the property of center of gravity, the estimated MS location can only be inside the convex hull of the BTSs. This is a strong restriction in location estimation. The Crude Estimation Method (CEM) that we are going to propose, is an investigation on a location estimation method based on signal strength and is not subject to this restriction.
3
Crude Estimation Method
The Crude Estimation Method (CEM) proposed in this paper, similar to the CGM, makes use of the signal strengths received by all the Base Transceivers 1 2
The BTS can be uniquely identified by the Broadcast Channel (BCCH) and the BSIC from the operator. Although all our proposed methods are handset independent, all our experiments and measurements are based on the Nokia handset (Model 6150).
1020
K.R.P.H. Leung et al.
signal strengths
Crude Estimation
set of possible marker locations
Location Recommendation
an estimated marker location
Fig. 1. Locus Method
Stations (BTSs) when a call is made. This method consists of two phases, Crude Estimation phase and Location Recommendation phase. In the Crude Estimation phase, the Crude Estimation Algorithm (CEA) is applied to estimate a location from every combination of two signal strengths received by the MS. This will generate a bag of possible locations. Then in the Location Recommendation phase, by manipulating the bag of possible locations obtained in Crude Estimation, a point is recommended as the estimated location. These processes are visualized in Figure 1. 3.1
Crude Estimation Algorithm
Let d be the distance between a MS and a BTS. Let s be the signal loss in the communication between the BTS and MS. Our CEA assumes that the relation s ∝ 1/de , for some e, holds. The intuition of CEM is assuming that the location of a MS lies on the locus which satisfy the ratio of the signal strength in the communication with the two BTSs. With the geographical location of the two BTSs and the relation between signal strength and distance, a locus of possible locations of the MS can be constructed. We choose the turning points of the locus as the crude estimated locations. This crude estimation approach caters for the possibilities that the estimated locations are not lying on the straight line between the two BTSs. Hence the estimated locations can be outside the convex hub of the BTSs. The derivation of the CEM is shown as follows.
Signal Strength 1
Possible Location (x, y) Signal Strength 2
Base Station 1
Base Station 2
Fig. 2. Locus formed by the ratio of signal strengths from two Base Stations
Let the location of two BTSs be (α1 , β1 ) and (α2 , β2 ). Let the signal strengths from the two BTSs to the MS be s1 and s2 . Then we have
Network Based Mobile Station Positioning in Metropolitan Area
1021
1 1 (3) s2 ∝ e (4) de1 d2 Let (x, y) be the location of MS. The relations of the distances between MS’s location and the two BTSs are, s1 ∝
2
2
2
2
(x − α1 ) + (y − β1 ) = d21 (5) (x − α2 ) + (y − β2 ) = d22 (6) By dividing Equation 5 by Equation 6, we have Equation 7, which is the ratio of the distances between the MS and the two BTSs. By substituting Equation 3, and Equation 4 into Equation 7, for some k we have Equation 8. 2
2
(x − α1 ) + (y − β1 )
2
2
d21 (x − α1 ) + (y − β1 ) s2 2 e (8) 2 2 2 2 = k{ s } 2 (7) d 1 (x − α2 ) + (y − β2 ) (x − α2 ) + (y − β2 ) 2 2 Let Q = k{ ss21 } e then Equation 8 becomes =
(x − α1 )2 + (y − β1 )2 = Q((x − α2 )2 + (y − β2 )2 )
(9)
⇒ x2 − 2α1 x + α12 + y 2 − 2β1 y + β12 = Q(x2 − 2α2 x + α22 + y 2 − 2β2 y + β22 ) (10) ⇒ (1 − Q)x2 − 2(α1 − Qα2 )x + (1 − Q)y 2 − 2(β1 − Qβ2 )y + α12 + β12 − Q(α22 + β22 ) = 0
(11)
Let A = (1 − Q), B = α1 − Qα2 , C = β1 − Qβ2 , D = α12 + β12 − Q(α22 + β22 ). Then Equation 11 becomes Equation 12. By differentiating Equation 12 once, we have Equation 13 Ax2 − 2Bx + Ay 2 − 2Cy + D = 0 (12)
B − Ax dy = (13) dx Ay − C
By putting Equation 13 equal to zero, we have √ B C ± C 2 − B 2 − AD x= and y = A A
Then when C 2 − B 2 − AD ≥ 0, we can obtain one or two estimated locations. These locations are the crude estimated locations. 3.2
Location Recommendation
We collect all the estimated locations by CEA for every combination of two received signal strengths. This form a bag of estimated locations. In this location recommendation phase, one recommended location is generated from this bag of estimated locations. We assumed that the set of result points is in a normal distribution. Thus, statistical approach is used in the recommendation process. We first remove the outlier estimated locations which are not within the standard deviations of the bag of estimated locations. Then we pick the center of gravity of the remained estimated locations as the recommended estimated location. Hence the recommended estimated location is calculated as follows.
1022
K.R.P.H. Leung et al.
Let N be the number of estimated locations obtained by CEA for every combination of two received signal strengths, and let the i-th location be (xi , yi ). So, the mean and variance of the location (xµ , yµ ) and (xσ , yσ ) are, N xµ =
i=1
N
xi
N , yµ =
i=1
N
yi
N
2
(xi − xµ ) , yσ = N −1
i=1
, xσ =
N
2
(yi − yµ ) N −1
i=1
Thus, the set of clustered result points S is S ::= {i : 1..N | (xi ≤ (xµ ± xσ ) ∧ yi ≤ (yµ ± yσ )) • (xi , yi )} The recommended point (xr , yr ) is D (xr =
i=1
D
xi
D , yr =
i=1
D
yi
)
where, D is the cardinality of S and (xi , yi ) ∈ S.
4
Field Test
We obtained supports from two major mobile operators of Hong Kong in conducting this research. We call them Operator A and Operator B3 . We conducted three field tests with two different mobile phone networks in two different areas in Hong Kong. With Operator A, we conducted two field tests. One of these field tests was conducted in the Prince Edward district and the other was conducted in the Mong Kok district. Both districts are urban areas where Mong Kok is more crowded and consists of building taller than Price Edward on the average. We could only conduct field tests in Mong Kok area with Operator B. In our field tests, we first planned some check points in the districts for our study. We call these points Marker Location. Then we get the geo-location of the active BTSs in the districts on the days of field tests. The networks of Operator A and Operator B are very different from each other. For Operator A, in the Prince Edward district, there were six (6) macro cells and forty-four (44) micro cells. The average distance among all these fifty BTSs was 804.87 meter. In the Mong Kok district, Operator A had fifty-four (54) micro cells and no macro cell. The average distance among all these fifty-four BTSs was 510.18 meter. On the other hand, Operator B had eighteen (18) macro cells and only four (4) micro cells in the Mong Kok district. The average distance among all these twenty-two BTSs was 889.59 meter. In our study, we collect the signal strengths from the MS. Nokia Model 6150 handsets were used in the field tests. A maximum of nine signal strengths together with their corresponding BTSs can be collected from the Net Monitor of 3
The mobile phone operators find the network information are sensitive to their business. Hence we have to hide their names and network information according to our Non-disclosure agreements.
Network Based Mobile Station Positioning in Metropolitan Area
1023
the handset. We collected 120 readings at each marker location. As the hardware restriction that the interval between each successive shots was 1 second, it took two minutes to collect all 120 readings. We had chosen fifteen markers in the Prince Edward district and eighteen markers in the Mong Kok district. The marker locations of these two districts are shown in Figure 3. Hence we collected 1800 readings at the Prince Edward district and 2160 readings from each of the operators in the Mong Kok district. In total, we have 6120 sets of real world data for our experiment.
Fig. 3. Marker locations in Prince Edward (Left) & Mongkok (Right)
5 5.1
Experiment Determining Value of e
In our experiment, we first validate our assumption that s ∝ 1/de for some e and also determine the value of e. We validate the relation and determine the value of e by plotting graphs of signal strength against distance with the data we collected. Since the networks of the two operators were different, we plot graphs for each of the operator. Furthermore, we eliminate some noises by averaging the signal strengths of each of the marker locations. The graphs of the signal strength from operator A and operator B are shown in Figure 4 and Figure 4, respectively. The curve obtained from Figure 4 is y = 125.02x−0.2122 and the curve obtained from Figure 4 is y = 129.8x−0.2056 . Hence, we can draw the following conclusions: our assumption that s ∝ 1/de is valid; and the value of e is around 0.2. 5.2
Experiment Results
Assuming that k in Equation 8 is related to the signal strengths in the communication, k is a function of the signal strengths received by the two BTSs. CEM assumes that the MS lies on the locus which is the ratio of the two received signal strengths. We experimented with different relations between the two received
1024
K.R.P.H. Leung et al.
Fig. 4. Signal Strength vs Distance of Operator A (Left) & Operator B (Right)
signal strengths and find out that k yields good results when it is expressed as follows: √ s1 k = √ where s1 > s2 . √ s1 + s2 CEM is applied to the data collected from the field tests with the e and k. For each reading, the MS could receive signals from up to nine BTSs. For the signals from every combination of two BTSs received, we applied the CEM once. Hence we had at most 9 C2 × 2 estimated locations for each reading. Since we took 120 readings at each marker location, we then have at most 9 C2 × 2 × 120 estimated locations for each marker location. We studied the average error of all these estimated locations, their variance and their best and worst cases as well. Afterwards, we looked at the overall average, variance, best and worst cases. We also compare CEM results of those obtained from CGM by using the same set of data. These results and comparisons are shown in Table 3 and Table 2. 5.3
Discussion of Experimental Results
A summary of the overall results of Table 3, Table 2, is shown in Table 1. The overall average errors of CEM in all tests are better than CGM by 33.85% which varies from 12.45% to 67.5%. The overall variance of CEM is slightly better than CGM by 5.05%. In analyzing the variance of CEM and CGM in the three tests, there are much less fluctuation of CEM than CGM in the Prince Edward district. However, the fluctuation of CEM is higher than CGM in the Mong Kok district. With these analysis, we can conclude that CEM can provide better location estimation than CGM on the average. However, the fluctuation of accuracy of CEM may not be better than CGM.
6
Conclusions and Future Work
We proposed a Crude Estimation Method for mobile station location estimation. This method is based on the signal strengths received by the mobile station and the geo-location of the base transceiver stations. We conducted field tests with two different real world GSM mobile phone network in Hong Kong. From the
Network Based Mobile Station Positioning in Metropolitan Area
1025
Table 1. Comparison of CEM & CGM Average Variance Best Worst Prince Edward (average BTSs distance: 804.87m) CEM 51.51827944 791.0404695 7.532442214 126.1238434 CGM 86.2916469 991.52 5.73326 464.98298 diff -67.5 -25.344 23.886 -268.672 Mong Kok - Operator A (average BTSs distance: 510.18m) CEM 42.26 365.21 0.66 282.52 CGM 50.64 297.88 0.57 193.85 diff -19.83 18.44 13.57 31.38 Mong Kok - Operator B (average BTSs distance: 889.59m) CEM 53.32 458.18 1.76 208 CGM 59.96 406.61 1.3 282.65 diff -12.45 11.26 26.26 -35.89 Overall results CEM 49.03 538.14 0.66 282.52 CGM 65.63 565.34 0.57 464.98 diff -33.85% -5.05% 13.57% -64.59%
6120 sets of readings, the average error of CEM is 49.03 meters with a variance of 538.14. In comparing these results with other similar methods including the Weighted Center of Gravity Method (CGM), another method proposed by our research team, CEM is a promising method. Furthermore, CEM does not have the disadvantage that the MS must lies inside the convex hull of the base transceiver stations. We have experimented the CEM method with two real world networks in two metropolitan areas in Hong Kong. Our next step is to study the theoretical aspects of this method. These include finding the intrinsic contribute to the value of e and the function k, and investigating the theoretical aspects of this method.
References 1. Federal Communications Commission: Revision to the commission’s rules to ensure compatibility with enhanced 911 emergency calling system. Technical Report CC Docket No. 94-102 (1996) 2. Geoffery G. Messier, Michel Fattouche and Brent R. Petersen: Locating an IS95 mobile using its signal. In: The Tenth International Conference on Wireless Communications (Wireless 98). Volume II., Calgary, AB, Canada (1998) 562–574 sponsored by TRLabs, the Communications Research Centre and IEEE Canada. 3. Willassen, S.Y.: A method of implementing mobile station location in gsm (1998) URL:http://www.willassen.no/msl/bakgrunn.html. 4. Y. Okumura, et.al.: Field strength and its variability in vhf and uhf and mobile service. Technical report, Rev. Elec. Comm. Lab. (1968) 5. Joseph K. Ng, Stephen K. Chan, and Kenny K. Kan: Location Estimation Algorithms for Providing Location Services within a Metropolitan Area based on a Mobile Phone Network. In: Proceedings of the 5th International Workshop on Mobility Databases and Distributed Systems (MDDS 2002) , Aix-en-Provence, France (2002) 710–715
1026
A
K.R.P.H. Leung et al.
Test Results Table 2. Results of CEM and CGM in Mong Kok Marker 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 OverAll
CEM (Op. A) CGM (Op. A) CEM (Op. B) CGM (Op. B) Average Variance Average Variance Average Variance Average Variance 17.26 10.13 36.58 46.86 34.32 66.95 14.39 52.81 29.79 341.25 15.5 85.93 40.87 38.01 31.44 74.18 42.37 347.42 35.05 236.43 64.08 270.38 52.37 179.74 46.99 256.09 31.89 84.3 84.53 27.85 42.91 210.57 16.27 100.99 15.81 95.24 23.11 6.93 14.14 97.8 26.51 89.51 26.14 142.99 60.67 565.82 31.13 365.03 19.76 11.86 25.26 102.57 71.31 919.58 50.57 193.87 42.12 191.95 42.09 84.9 37.41 164.75 49.66 212.95 83.31 634.0 48.49 152.55 82.68 1046.13 44.15 473.96 42.36 661.31 155.39 124.59 77.83 3146.26 68.23 702.45 66.78 339.72 93.06 1068.56 58.52 361.19 59.24 152.24 23.23 174.03 84.84 946.42 31.86 11.27 50.63 549.46 46.76 110.01 53.37 232.45 23.96 268.76 63.76 1314.5 29.85 76.78 83.06 62.33 54.84 232.18 112.6 485.28 134.2 2651.7 50.61 1488.21 98.14 529.16 173.05 1633.21 44.14 311.29 29.83 258.22 69.01 199.59 101.44 260.23 34.38 126.31 64.09 127.73 25.65 229.37 75.12 284.25 14.05 139.36 18.51 21.56 21.07 163.1 44.62 76.45 42.26 365.21 50.64 297.88 53.32 458.18 59.96 406.61
Table 3. Results of CEM and CGM in Prince Edward (Operator A)
Marker 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Overall
Average 43.96 57.75 33.79 18.3 9.39 99.39 65.46 45.46 92.54 27.1 70.78 71.5 28.7 74.65 33.99 51.52
CEM Variance Best 94.54 8.31 351.63 15.3 7.30 28.31 2.42 15.36 18.02 7.53 19.06 67.19 81.08 15.76 101.61 27.91 43.32 82.15 80.7 12.0 571.76 21.23 198.45 36.54 5.74 23.88 73.02 24.06 0.38 33.54 791.04 7.53
Worst 55.39 76.70 41.09 22.25 42.88 100.69 82.2 79.69 110.06 49.67 126.12 95.79 33.37 106.23 37.48 126.12
Average 78.94 48.03 66.91 31.88 76.3 101.51 86.68 117.41 108.44 64.27 222.36 72.75 45.28 134.16 39.45 86.29
CGM Variance Best 285.31 20.7 295.42 10.12 2799.16 9.36 29.54 22.63 254.52 47.49 18.12 92.92 5001.58 21.54 221.35 66.14 2062.54 13.37 726.86 14.76 35.46 207.95 770.26 5.73 99.23 15.63 1920.08 35.84 353.31 12.3 991.52 5.73326
Worst 130.15 72.53 207.39 50.71 114.8 113.89 464.98 152.56 203.33 172.91 240.06 127.51 83.26 275.31 112.18 464.98
Programming Coordinated Motion Patterns with the TOTA Middleware 1,2
2
1
Marco Mamei , Franco Zambonelli , and Letizia Leonardi 1
Dipartimento di Ingegneria dell’Informazione – Università di Modena e Reggio Emilia Via Vignolese 905 – Modena – ITALY 2 Dipartimento di Ingegneria dell’Informazione – Università di Modena e Reggio Emilia Via Allegri 13 – Reggio Emilia – ITALY {mamei.marco, franco.zambonelli, letizia.leonardi}@unimo.it
Abstract. In this paper, we present TOTA (“Tuples On The Air”), a novel middleware to coordinate the movements of a large number of autonomous components (i.e. agents) in a ubiquitous computing scenario. The key idea in TOTA is to rely on spatially distributed tuples for both representing contextual information and supporting uncoupled and adaptive interactions between application components. The TOTA middleware takes care both of propagating tuples across a network on the basis of application-specific rules and of adaptively re-shaping the resulting distributed structures accordingly to changes in the network structures. Application agents – via a simple API – can locally sense such distributed structures to achieve context-awareness and to effectively coordinate their movements.
1
Introduction
Computing is becoming intrinsically ubiquitous and mobile [6]. Computer-based systems are going to be embedded in all our everyday objects and in our everyday environments. These systems will be typically communication enabled, and capable of interacting with each other in the context of complex distributed applications, e.g., to support our cooperative activities [4], to monitor and control our environments [2], and to improve our interactions with the physical world [9]. Also, since most of the embeddings will be intrinsically mobile, distributed software processes and components (from now on, we adopt the term “agents” to generically indicate the active components of a distributed application) will have to effectively interact with each other and effectively orchestrate their motion coordination activities despite the network and environmental dynamics induced by mobility. The above scenario introduces peculiar challenging requirements in the development of distributed software systems: (i) since new agents can leave and arrive at any time, and can roam across different environments, applications have to be adaptive, and capable of dealing with such changes in a flexible and unsupervised way; (ii) the activities of the software systems are often contextual, i.e., strictly related to the environment in which the systems execute (e.g., a room or a street), whose H. Kosch, L. Böszörményi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 1027–1037, 2003. © Springer-Verlag Berlin Heidelberg 2003
1028
M. Mamei, F. Zambonelli, and L. Leonardi
characteristics are typically a priori unknown, thus requiring to dynamically enforce context-awareness; (iii) the adherence to the above requirements must not clashes with the need of promoting a simple programming model requiring light supporting infrastructures, possibly suited for resource-constrained and power-limited devices [15]. Unfortunately, current practice in distributed software development are unlikely to effectively address the above requirement: (i) application agents are typically strictly coupled in their interactions (e.g., as in message-passing models), thus making it difficult to promote and support spontaneous interoperations; (ii) agents are provided with either no contextual information at all or with only low-expressive information (e.g., raw local data or simple events), difficult to be exploited for complex coordination activities; (iii) due to the above, the results is usually in an increase of both application and supporting environment complexity. Here we focus on the problem of motion coordination in ubiquitous computing environments, because we think that this problem resembles perfectly the peculiar challenges introduced before and highlights the ineffectiveness of current practices. The approach we propose in this paper builds on the lessons of uncoupled coordination models like event-based [5] and tuple space programming [4] and aims at providing agents with effective contextual information that – while preserving the lightness of the supporting environment and promoting simplicity of programming – can facilitate both the contextual activities of application agents and the definition of complex distributed motion coordination patterns. In TOTA (“Tuples On The Air”), all interactions between application agents take place in a fully uncoupled way via tuple exchanges. However, unlike in traditional tuple-based model, there is not any notion like a centralized shared tuple space. Rather, tuples are some how injected into the network and, in the network, can propagate and diffuse accordingly to a tuple-specific propagation pattern, automatically re-adapting such patterns accordingly to the dynamics changes that can occur in the network. All interactions are mediated by these distributed tuples, which can express both information explicitly provided by other agents or a local view of some global property of the network. As we will show later in this paper, this facilitate the ease definition of very complex coordination activities. The contribution of this paper is to motivate and present the key concepts underlying TOTA, and to show how it can be effectively exploited to develop in a simple way adaptive and context-aware distributed applications, suitable for the dynamics of pervasive computing scenarios. To this end, the following of this paper is organized as follows. Section 2 overviews the TOTA approach. Section 3 details how to program distributed motion coordination in TOTA. Section 4 discusses related works. Section 5 concludes and outlines future works.
2
The Tuples on the Air Approach
TOTA proposes relying on distributed tuples for both representing contextual information and enabling uncoupled interactions among distributed application agents. Unlike traditional shared data space models [4], tuples are not associated to a specific
Programming Coordinated Motion Patterns with the TOTA Middleware
1029
node (or to a specific data space) of the network. Instead, tuples are injected in the network and can autonomously propagate and diffuse in the network accordingly to a specified pattern. Thus, TOTA tuples form a sort of spatially distributed data structure able to express not only data to be transmitted between application agents but, more generally, some property of the distributed environment. To support this idea, TOTA is composed by a peer-to-peer network of possibly mobile nodes, each running a local version of the TOTA middleware. Each TOTA node holds references to a limited set of neighboring nodes. The structure of the network, as determined by the neighborhood relations, is automatically maintained and updated by the nodes to support dynamic changes, whether due to nodes’ mobility or to nodes’ failures. The specific nature of the network scenario determines how each node can found its neighbors: e.g., in a MANET scenario, TOTA peers are found within the range of their wireless connection; in the Internet they can be found via an expanding ring search. More details on the architecture of the TOTA middleware can be found in [9]. Upon the distributed space identified by the dynamic network of TOTA peers, each agent is capable of locally storing tuples and letting them diffuse through the network. Tuples are injected in the system from a particular node, then they spread hop-by-hop accordingly to their propagation rule, generally being stored in the nodes they visit. Accordingly, a tuple in TOTA is defined in terms of a "content", and a "propagation rule". T=(C,P) The content C of a tuple is basically an ordered set of typed fields representing the information carried on by the tuple. The propagation rule P determines how a tuple should be distributed and propagated in the TOTA network. This includes determining the "scope" of the tuple (i.e. the distance at which such tuple should be propagated and possibly the spatial direction of propagation) and how such propagation can be affected by the presence or the absence of other tuples in the system. In addition, the propagation rules can determine how tuple’s content should change while it is propagated. Tuples do not necessarily have to be distributed replicas: by assuming different values in different nodes, tuples can be effectively used to build a distributed data structure expressing some kind of spatial/contextual information. We emphasize that the TOTA middleware supports tuples propagation actively and adaptively: by constantly monitoring the network local topology and the income of new tuples, the middleware automatically re-propagates tuples as soon as appropriate conditions occur. For instance, when new peers get in touch with a network, TOTA automatically checks the propagation rules of the already stored tuples and eventually propagates the tuples to the new peers. Similarly, when the topology changes due to peers’ movements, the distributed tuple structure automatically changes to reflect the new topology. From the application agents’ point of view, executing and interacting basically reduces to inject tuples, perceive local tuples, and act accordingly to some applicationspecific policies. Software agents on a TOTA node can inject new tuples in the network, defining their content and their propagation rule. They have full access to the
1030
M. Mamei, F. Zambonelli, and L. Leonardi
local content of the middleware (i.e., of the local tuple space), and can query the local tuple space – via a pattern-matching mechanism – to check for the local presence of specific tuples. Moreover, the TOTA middleware offers a read-only, view of the tuple spaces of the one-hop TOTA neighbors. This feature is fundamental since the main TOTA algorithms require the knowledge of tuples present in at least a one-hop neighborhood (e.g. to evaluate the tuples’ gradients as described in 2.1). In addition, agents can be notified of locally occurring events (i.e., changes in tuple space content and in the structure of the network neighborhood). 2.1
Motion Coordination in TOTA
To fix ideas on a case study, we focus on applications in which a group of autonomous and mobile agents have to globally coordinate their respective movements. The goals of their coordination can be various: letting agents meet somewhere [4], move avoiding traffic jams [1], distribute themselves accordingly to a specific geometry [11, 13], etc. Our specific case study, consists in how to program a group of agents, each running on the TOTA middleware, moving maintaining a suitable formation. For example, we can imagine security guards in a museum to move and monitor the museum in a coordinated way, i.e., according to a specific formation in which they have to preserve a specified distance from each other. Security guards can be provided with wireless enabled palm computer connected each other in an ad-hoc network and running the TOTA middleware. TOTA agents could direct guards movements to help them maintaining the formation.
Fig 1. Distribution of a single flocking tuple (left); Regular formation of flocking peers (right)
To this end, we can take inspiration from the work done in the swarm intelligence research [1]: flocks of birds stay together, coordinate turns, and avoid each other, by following a very simple swarm algorithm. Their coordinated behavior can be explained by assuming that each bird tries to maintain a specified separation from the nearest birds and to match nearby birds’ velocity. To implement such a coordinated behavior in TOTA and apply it in our case study, we can have that each security guard generates a tuple T=(C,P) with following characteristics:
Programming Coordinated Motion Patterns with the TOTA Middleware
1031
C = (peerName,val) P = (“val” is initialized at 2, propagate to all the peers decreasing by one in the first two hops, then increasing “val” by one for all the further hops) This tuple creates a distributed data structure in which the val field assumes the minimal value at specific distance from the source (e.g., 2 hops), distance expressing the intended spatial separation between security guards. For a tuple, the val field assumes a distribution approaching the one showed in Figure 1-left. The TOTA middleware ensures dynamic updating of this distribution to reflect peers’ movements. To coordinate movements, peers have simply to locally perceive the generated tuples, and to follow downhill the gradient of the val fields. The result is a globally coordinated movement in which peers maintain an almost regular grid formation (see Figure 1-right). 2.2
Implementation
From an implementation point of view, we developed a first prototype of TOTA running on laptops and on Compaq IPAQs equipped with 802.11b and Personal Java. IPAQ connects locally in the MANET mode (i.e. without requiring access points) creating the skeleton of the TOTA network. Tuples are being propagated through multicast sockets to all the nodes in the one-hop neighbor. Actually we own only a dozen of IPAQs and laptops on which to run the system. Since the effective testing of TOTA would require a larger number of devices, we have implemented an emulator to analyze TOTA behavior in presence of hundreds of nodes. The emulator, developed in Java, enables examining TOTA behavior in a MANET scenario, in which nodes topology can be rearranged dynamically either by a drag and drop user interface or by autonomous nodes’ movements. The strength of our emulator is that, by adopting well-defined interfaces between the emulator and the application layers, the same code “installed” on the emulated devices can be installed on Java real devices (e.g. Compaq IPAQs) enabled with wireless connectivity. This allow to test application first in the emulator, then to transfer them directly in a network of real devices. More details on the implementation can be found in [10].
3
TOTA Programming
When developing applications upon TOTA, one has basically to know: what are the primitive operations to interact with the middleware; how to specify tuples and their propagation rule; how to exploit the above to code agent coordination. 3.1
TOTA Primitives
TOTA is provided with a simple set of primitive operations to interact with the middleware.
1032
M. Mamei, F. Zambonelli, and L. Leonardi
public public public public rct); public
void inject (TotaTuple tuple); Vector read (Tuple template); Vector delete (Tuple template); void subscribe (Tuple template, ReactiveComponent comp, String void unsubscribe (Tuple template, ReactiveComponent comp);
Inject is used to inject the tuple passed as an argument in the TOTA network. Once injected the tuple starts propagating accordingly to its propagation rule (embedded in the tuple definition). The read primitive accesses the local TOTA tuple space and returns a collection of the tuples locally present in the tuple space and matching the template tuple passed as parameter. The delete primitive extracts from the local middleware all the tuples matching the template and returns them to the invoking agent. In addition, subscribe and unsubscribe primitives are defined to handle events. These primitives rely on the fact that any event occurring in TOTA (including: arrivals of new tuples, connections and disconnections of peers) can be represented as a tuple. Thus: the subscribe primitive associates the execution of a reaction method in the agent in response to the occurrence of events matching the template tuple passed as first parameter. The unsubscribe primitives removes matching subscriptions. 3.2
Specifying Tuples – The Flocking Tuple
TOTA relies on an object oriented tuple representation. The system is preset with an object oriented hierarchy of tuples. A programmer can inherit from this hierarchy to create his own application specific tuple. We are still developing a full hierarchy, but we already completed the definition of the branch leading to tuple whose content is a function of the hop distance from the source. The root of the full hierarchy is the abstract class TotaTuple, that provides a general framework for tuples programming. The class at the basis of the hop-based tuples is the abstract class HopBasedTuple, it inherits from TotaTuple, implements a general purpose breadth first, expanding ring propagation with a method called propagate. In this class three abstract methods, controlling the tuple propagation, its content update and its behavior have been defined: public abstract boolean decidePropagation(); public abstract Tuple changeTupleContent(); public abstract boolean decideStore ();
These methods must be implemented when subclassing from the abstract class HopBasedTuple to create and instantiate actual tuples. This approach is very handy when programming new tuples, because there is no need to re-implement every time the propagation and maintenance mechanism from scratch, but it allows to customize the same breadth first, expanding ring propagation for different purposes. In particular the core structure of the method propagate is the following:
Programming Coordinated Motion Patterns with the TOTA Middleware
1033
if(this.decidePropagation()) { updatedTuple = this.changeTupleContent(); if(this.decideStore()) { tota.store(updatedTuple); } }
All these methods acts on the tuple itself (the “this” in OO terminology) and can access the local TOTA middleware in which they execute (i.e., in which a propagating tuple has arrived). The first method (decidePropagation) is executed upon arrival of a tuple on a new node, and returns true if the tuple has to be propagated in that middleware, false otherwise. So for example an implementation of this method that returns always true, realize a tuple’s propagation that floods the network because the tuple is propagated in a breadth first, expanding ring way to all the peers in the network. Vice-versa an always false implementation creates a tuple that does not propagate, because the expanding ring propagation is blocked in all directions. In general more sophisticate implementations of this method are used to narrow down tuple spreading to avoid network flooding. The second method (changeTupleContent) creates a new tuple that updates the previous one. This method is used because a tuple can change its content during the propagation process. Basically before propagating a tuple to a new node, the tuple is passed to this method that returns an eventually modified version of it, to be propagated. So for example, if the tuple content is an integer value and this method returns a tuple having that value increased by one, we are creating a tuple whose integer content increases by one at each propagation hop. The third method (decideStore) returns true if the tuple has to be stored in the local middleware, false otherwise. This method can be used to create transient tuples, i.e., tuples that roam across the network like messages (or events) without being stored. Eventually, once the above method determined that a tuple has to be locally stored, a method of the middleware with restricted access (the store method) is invoked on the local TOTA middleware (tota) to store a tuple in the local tuple space. To clarify these points we consider the flocking motion coordination problem introduced in 2.1. Let’s create the tuple FlockingTuple: the content C of the tuple is encoded by the object state, the method propagate P is inherited from the abstract class HopBasedTuple, the three abstract methods, controlling the tuple propagation, its content update and its behavior must be implemented. public class FlockingTuple extends HopBasedTuple { public String peerName; public int val = 2; /* always propagate */ public boolean decidePropagation() { return true; } public HopBasedTuple changeTupleContent() { /* here it must be a function of the hop value. The correct hop distance is maintained in the super-class HopBasedTuple */ // flocking shape if(hop 2) val++; return this; } /* always store */ public boolean decideStore() { return true; } }
The decidePropagation and decideStore methods return always true, because the flocking tuple should be spread to the whole network and being stored in every node. This is in order to create just one large flock, rather than some separate flocks. The changeTupleContent method changes the tuple being propagated by decreasing its val field for the first two hops then letting it increase. In this way the val field has a minimum located two-hops away from its source. 3.3
The Flocking Agent
The algorithm followed by flocking agents is very simple: they simply have to determine the closest peer, and then move by following downhill that peer’s flocking tuple. To achieve this goal each agent will evaluate the gradient of the flocking tuple it is going to follow, by comparing tuple’s values in its neighborhood. public class FlockingAgent extends Thread implements AgentInterface { private TotaMiddleware tota; … public void run() { // inject the meeting tuple to participate the meeting FlockingTuple ft = new FockingTuple (); ft.setContent(peer.toString()); tota.inject(ft); while(true) { // read other agents’ flocking tuples FlockingTuple query = new FlockingTuple(); Vector v = tota.read(query); /* evaluate the gradients and select the peer to which the gradient goes downhill */ GenPoint destination = getDestination(v); // move downhill following the meeting tuple peer.move(destination); } } … }
4
Related Works
Several proposals in the last few years are challenging the traditional ideas and methodologies of distributed software engineering by proposing frameworks and models inspired by physical and biological systems to facilitate, among the others, distributed motion coordination.
Programming Coordinated Motion Patterns with the TOTA Middleware
1035
In robotics, the idea of potential fields driving robots movement is not new [8]. For instance, one of the most recent re-issue of this idea, the Electric Field Approach (EFA) [7], has been exploited in the control of a team of Sony Aibo legged robots in the RoboCup domain. Following this approach, each Aibo robot builds a field-based representation of the environment from the images captured by its head mounted camera, and decides its movements by examining the fields’ gradients of this representation. Although very close in spirit, EFA and TOTA are very different from the implementation point of view: in TOTA, fields are distributed data structure actually spread in the environment; in EFA, fields are just an agent internal representation of the environment and they do not actually exists. TOTA requires a supporting infrastructure to host fields’ data structures, but it completely avoids the complex algorithms involved in field representation and construction. Smart Messages [2], rooted in the area of active messages, is an architecture for computation and communication in large networks of embedded systems. Communication is realized by sending “smart messages” in the network, i.e., messages which include code to be executed at each hop in the network path. The execution of the message at each hop determines the next hop in the path, making messages responsible for their own routing. Smart Messages share with TOTA, the idea of putting intelligence in the network by letting messages (or tuples) execute hop-by-hop small chunk of code to determine their propagation. However, in Smart Messages code is used mainly for routing or packets mobility purposes. In TOTA instead, the tuples are not simply routed though the network, but can be persistent and create a distributed data structure that remains stored in the network, and TOTA propagation code can also be used to change the message/tuple content, in order to realize distributed data structure and not just replicas. As a final note, we emphasize that (i) recent approaches in the area of modular robots [13] exploit the idea of propagating “coordination field” across the robot agents so as to achieve a globally coherent behavior in robot’s re-shaping activities; (ii) in the popular simulation game “The Sims” [14], characters move and act accordingly to specific fields that are assumed to be spread in the simulated environment and sensed by characters depending on situations (e.g., they sense the food field when hungry); (iii) ant-based optimization systems [1, 12] exploit a virtual environment in which ants can spread pheromones, diffusing and evaporating in the environment according to specific rules. (iv) amorphous computers [3, 11] exploit propagation of fields to let particles self-organize their activities. Although serving different purposes, these approaches definitely share with TOTA the same physical inspiration.
5
Conclusions and Future Works
Tuples On The Air (TOTA) promotes programming motion coordination activities by relying on distributed data structures, spread over a network as sorts of electromagnetic fields, and to be used by application agents both to extract contextual information and to coordinate with each other in an effective way. As we have tried to show in this paper, TOTA suits the needs of pervasive computing environments (spontaneous interoperation, effective context-awareness, lightness of supporting
1036
M. Mamei, F. Zambonelli, and L. Leonardi
environment) and facilitates both the access to distributed information and the expression of distributed coordination patterns. Several issues are still to be investigated to make TOTA a practically useful framework for the development of pervasive applications. First, proper access control models must be defined to rule accesses to distributed tuples and their updates (and this is indeed a challenging issue for the whole area of pervasive and mobile computing). Second, because the maintenance of all the distributed tuples may create scalability issues, we think it would be necessary to enrich TOTA with mechanisms to compose tuples with each other, so as to enable the expression of unified distributed data structures from the emanation of multiple sources. Finally, deployment of TOTA applications in real-world scenarios will definitely help identify current shortcomings and directions of improvement, possibly suggesting application even beyond the motion coordination area. Acknowledgements. Work supported by the Italian MIUR and CNR in the “Progetto Strategico IS-MANET, Infrastructures for Mobile ad-hoc Networks”.
References [1]
E. Bonabeau, M. Dorigo, G. Theraulaz, “Swarm Intelligence”, Oxford University Press, 1999. [2] C. Borcea, et al., “Cooperative Computing for Distributed Embedded Systems”, 22th International Conference on Distributed Computing Systems, Vienna (A), IEEE CS Press July 2002. [3] W. Butera, “Programming a Paintable Computer”, PhD Thesis, MIT Media Lab, Feb. 2002. [4] G. Cabri, L. Leonardi, F. Zambonelli, “Engineering Mobile Agent Applications via Context-Dependent Coordination”, IEEE Transactions on Software Engineering, 28(11), Nov. 2002. [5] G. Cugola, A. Fuggetta, E. De Nitto, “The JEDI Event-based Infrastructure and its Application to the Development of the OPSS WFMS”, IEEE Transactions on Software Engineering, 27(9): 827–850, Sept. 2001. [6] D. Estrin, D. Culler, K. Pister, G. Sukjatme, “Connecting the Physical World with Pervasive Networks”, IEEE Pervasive Computing, 1(1):59–69, Jan. 2002. [7] S. Johansson, A. Saffiotti, “Using the Electric Field Approach in the RoboCup Domain”, RoboCup 2001, LNAI, Springer Verlag, pp. 399–404, Seattle, WA, USA, 2001. [8] O. Khatib, “Real-time Obstacle Avoidance for Manipulators and Mobile Robots”, The International Journal of Robotics Research, 5(1):90–98, 1986. [9] M. Mamei, L. Leonardi, M. Mahan, F. Zambonelli, “Coordinating Mobility in a Ubiquitous Computing Scenario with Co-Fields”, Workshop on Ubiquitous Agents on Embedded, Wearable, and Mobile Devices, AAMAS 2002, Bologna, Italy, July 2002. [10] M. Mamei, F. Zambonelli, L. Leonardi, “Tuples On The Air: a Middleware for ContextAware Computing in Dynamic Networks”, 1st International ICDCS Workshop on Mobile Computing Middleware (MCM03) Providence, Rhode Island. May 2003. [11] R. Nagpal, “Programmable Self-Assembly Using Biologically-Inspired Multiagent Control”, 1st International Conference on Autonomous Agents and Multiagent Systems, Bologna (I), ACM Press, July 2002.
Programming Coordinated Motion Patterns with the TOTA Middleware
1037
[12] V. Parunak, S. Bruekner, J. Sauter, "ERIM’s Approach to Fine-Grained Agents", NASA/JPL Workshop on Radical Agent Concepts, Greenbelt (MD), Jan. 2002. [13] W. Shen, B. Salemi, P. Will, “Hormone-Inspired Adaptive Communication and Distributed Control for CONRO Self-Reconfigurable Robots”, IEEE Transactions on Robotics and Automation, Oct. 2002. [14] The Sims, www.thesims.com [15] F. Zambonelli, V. Parunak, “From Design to Intention: Signs of a Revolution”, 1st Intl. ACM Conference on Autonomous Agents and Multiagent Systems”, Bologna (I), July 2002.
iClouds – Peer-to-Peer Information Sharing in Mobile Environments Andreas Heinemann , Jussi Kangasharju, Fernando Lyardet, and Max M¨ uhlh¨ auser Telecooperation Group, Department of Computer Science Darmstadt University of Technology Alexanderstr. 6, 64283 Darmstadt, Germany
[email protected] {jussi, fernando, max}@tk.informatik.tu-darmstadt.de
Abstract. The future mobile and ubiquitous computing world will need new forms of information sharing and collaboration between people. In this paper we present iClouds, an architecture for spontaneous mobile user interaction, collaboration, and transparent data exchange. iClouds relies on wireless ad hoc peer-to-peer communications. We present the iClouds architecture and different communication models, which closely resemble familiar communication forms in the real world. We also design a hierarchical information structure for storing the information in iClouds. We present our prototype implementation of iClouds which runs on wireless-enabled PDAs.
1
Introduction
People living in the future ubiquitous computing world will need new ways to share information and interests as well as collaborate with each other. Given the success of wireless communications, such as mobile telephones, 802.11b, or Bluetooth, many of these activities will benefit from or rely on wireless ad hoc communications. This paper presents the iClouds project in which we study the architectures and mechanisms required to support mobile user interaction, collaboration, and transparent data exchange. The iClouds project is part of the Mundo research activity, which we will present in the next section. Our motivation behind iClouds can be expressed as follows: “Whenever there is a group of people, they may share a common goal or have a related motivation. Information of interest may be in possession of only a few of them.” The goal of iClouds is to make this information available to the whole group, based on individual user contribution, through peer-to-peer communications and data exchange.
The author’s work was supported by the German National Science Foundation (DFG) as part of the PhD program “Enabling Technologies for Electronic Commerce” at Darmstadt University of Technology.
H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 1038–1045, 2003. c Springer-Verlag Berlin Heidelberg 2003
iClouds – Peer-to-Peer Information Sharing in Mobile Environments
(a) Communication horizon
1039
(b) iClouds with 3 peers
Fig. 1. Information clouds
Consider a person walking with a wireless-enabled PDA. The communication range of the wireless device defines a sphere around that node. We call this sphere or communication horizon an information cloud or iCloud (see Fig. 1(a)). In practice this will not be an ideal sphere due to radio signal interference with buildings and other structures. The limited communication range (a few hundred meters at most) is a desired property, because it allows for easy ad hoc meetings and collaboration. When several nodes come close together, as shown in Fig. 1(b), the devices can communicate with each other and exchange information depending on what information the users provide and need. This exchange happens automatically, without any need for direct user intervention. We have identified several application scenarios in which iClouds is beneficial: – Local Information Acquisition. Residents of a city publish information about their city which tourists are interested in. This could be information about sights, restaurants, or useful telephone numbers, such as taxi number, etc. – Common Goal Pursuit. iClouds can bring people with common interests together to help them collaborate. For example, consider students in a classroom. Some students may have formed study groups and others might be interested in joining those groups. Students already in the groups could publish the groups and interested students could directly contact them and join the group. – Advertisement and mCommerce. A store can publish ads which are picked up by interested customers. These customers further pass the ads along to other interested users when they are away from the store, thus increasing the reach of the ads. If any of the users who have received ads in this way actually make a purchase, the store could give a bonus to the person who passed the ad along. This bonus could be, for example, points or a discount on the next purchase. This paper is organized as follows. Section 2 describes the underlying Mundo project. Section 3 presents the iClouds architecture and communication mechanisms. In Section 4 we describe our prototype implementation of iClouds. Sec-
1040
A. Heinemann et al.
tion 5 discusses related work. Finally, Section 6 concludes the paper and presents directions for future work.
2
Mundo Overview
In this section, we provide a brief overview of the Mundo project. A more complete description can be found in [1]; below we will briefly present the different entities in Mundo. ME (Minimal Entity) We consider ME devices as the representation of their users in the digital world. The personal ME is the only entity always involved in the user’s activities. Our decision for one individually owned device rather than a publicly available infrastructure comes from trust establishment. As an increasing amount of computer based activities have (potential) legal impact and involve privacy issues, it is vital that users can trust the device carrying out such transactions. US (Ubiquitous aSsociable object) Minimalization pressure will not permit feature-rich MEs in the near future. Hence, they must be able to connect to local devices such as memory, processors, displays, and network devices in an ad hoc manner. We call this process association and we call such devices ubiquitous associable objects (US). Through association, the ME can personalize the US to suit the user’s preferences and needs. For privacy reasons, any personalization of an US must become unavailable if it is out of range of the user’s ME device. IT (smart ITem) There are also numerous smart items that do not support association that would turn them into an US. Vending machines, goods equipped with radio frequency IDs, and landmarks with “what is” functionality are just a few examples. We define such smart items as ITs. An IT is any digital or real entity that has an identity and can communicate with the ME. Communication may be active or passive; memory and computation capabilities are optional. WE (Wireless group Environment) We expect ad hoc networking to be restricted to an area near to the user of a ME device, as connections with remote services will involve a non-ad hoc network infrastructure. The functionality of a WE is to bring together two or more personal environments consisting of a ME and arbitrary US entities each. A WE makes connections between the devices possible and also allows for sharing and transferring hardware (e.g., US devices) and software or data between WE users. iClouds is our most advanced project (in terms of realization) within the WE sub-domain of Mundo. While iClouds focuses on pure information sharing, other WE-related activities consider hardware sharing and corresponding security aspects. THEY (Telecooperative Hierarchical ovErlaY) We regard overlay cells as the backbone infrastructure of Mundo. These telecooperative hierarchical overlay cells (THEY) connect users to the (non-local)
iClouds – Peer-to-Peer Information Sharing in Mobile Environments
1041
world, deliver services, and data to the user. THEY support transparent data access, for example, by caching frequently used data on US devices, and offer transparent cooperation between different physical networks. Because iClouds is based on ad hoc networking, it does not require the services of THEY.
3
iClouds System Description
iClouds devices are small mobile devices (like PDAs) with mobile communication support for a maximum of a few 100 meters; one example is a PDA with 802.11b support. There is no need for any central servers in the iClouds architecture; instead, each device is completely independent. The diameter of the iClouds communication horizon (Fig. 1(a)) should not exceed a few hundred meters. We want to give iCloud users the option for spontaneous collaboration and when two iCloud users “see” each other, they should be within a short walking distance from each other (a couple of minutes at maximum). To allow for this easy collaboration, we specifically exclude multi-hop communication. Therefore, iClouds does not require any routing protocols; all communications happen directly between the concerned parties. 3.1
Data Structures, Communication Pattern, and Information Exchange
The two most important data structures found on the iClouds device are two information lists (iLists for short): – iHave-list (information have list or information goods) The iHave list holds all the information the user wants to contribute to the iCloud. The items could be, for example, simple strings, or more complex entities expressed in XML. We discuss these issues in Section 3.2. – iWish-list (information wish list or information needs) In the iWish list, the user specifies what kind of information he is interested in. The exact format of the items on this list depend on the format of the entries on the iHave list. Typically they would be search patterns which are matched against the entries on the iHave lists of other users. Note that the items on the iWish list are more private, since they reflect the needs of the user, which the user may not want to disclose to others. Each iClouds device periodically scans its vicinity to see if known nodes are still active and in communication range and also to see if any new nodes have appeared. Information about active nodes is stored in a data structure called neighborhood. In the second stage, the iClouds devices align their information goods and needs. This is achieved by exchanging iLists. Items on the iWish-lists are matched against items on the iHave-lists. On a match, information items move from one iHave-list to the other.
1042
A. Heinemann et al. Table 1. Information Flow Semantics (from Alice’s point of view) pull (from Bob) push (to Bob) iHave-List Standard search Advertise iWish-List Active service inquiry Active search
For example, consider two iClouds users, Alice and Bob, who meet on the street. When their iClouds devices discover each other, they will exchange their iHave lists and match them locally against their iWish lists. If an item on Bob’s iHave list matches an item on Alice’s iWish list, her iClouds device will transfer that item onto her iHave list. We have two main communication methods for transferring the iLists. Peers can either pull the iLists from other peers or they can push their own iLists to peers they encounter. In addition, either of these two operations is applicable to both lists, which gives us four distinct possibilities of communication. We summarize these possibilities, along with their real-world equivalents, in Table 1. In each of the four cases shown in Table 1, the matching operation is always performed on the peer who receives the list (Alice’s in pull and Bob’s in push). Each of the four possible combinations corresponds to some interaction in the real world: – Standard search. Alice pulls iHave-List from Bob. This is the most natural communication pattern. Alice asks for the information stored on Bob’s device and performs a match against her information needs (specified in her iWish-List) on her device. We can also see the user as just passively “browsing” what is available. – Advertise. Alice pushes her iHave-List to Bob. This is a more direct approach. Alice gives her information goods straight to Bob and it’s up to Bob to match this against the things he is interested in. As an example, consider iClouds devices mounted on shopping mall doorways pushing advertisements onto customer devices when they enter the building. – Active service inquiry. Alice pulls iWish-List from Bob. This is best suited for shopping clerks. They learn at a very early stage, what their customers are interested. An example of this query could be: “Can I help you, please show me what are you looking for?”. In general, especially for privacy reasons and user acceptance, we believe it is a good design choice to leave the iWish-list on the iClouds device. Hence, this model of communication would likely be extremely rare in the real world. – Active search. Alice pushes her iWish-List to Bob. With active search, we model the natural “I’m looking for X. Can you help me?”. This is similar to the standard search mechanism, except that the user is actively searching for a particular item, whereas in the standard search the user is more passive. 3.2
Information Modeling and List Matching
A key issue in iClouds is a set of rules or constraints for the information items on the iHave-list, as well as for the search patterns stored on the iWish-list.
iClouds – Peer-to-Peer Information Sharing in Mobile Environments
1043
Restaurants
Mediterranean
Indian
Italian
Vegetarian
Buca Giovanni
Star of India
Buca Giovanni
Bombay Restaurant
Savoia Ristorante
Bombay Restaurant
Savoia Ristorante
Cafe Spice
Fig. 2. Hierarchical data organization
For example, consider free text for both lists. A hypothetical item Best Italian Restaurant on Market Street will not match an information wish Interested in Mediterranean Food. Therefore the chances are close to zero, that any information is passed through iClouds, although one node holds information the other node is interested in. To overcome this, we propose a hierarchical system, as shown in Fig. 2, similar to the organization of the Usenet, the product catalog of eBay, or the online catalog of Yahoo. Items are divided into categories (e.g., restaurants, hotels, etc.) which are further divided into subcategories (e.g., Italian, Indian, etc. restaurants). Note that entries may belong to several categories. For example, the Bombay Restaurant in Fig. 2 is listed under Indian and Vegetarian, because it offers both Indian and vegetarian food. These hierarchies can be stored at a central server and can be downloaded to an iClouds device while it is in its cradle during power recharge or a normal sync operation. This system should be open for extension and categories could be moderated by different people to improve quality. Hierarchies are well understood by users and, in addition, technologies, such as XML and XPATH, support the construction of hierarchically organized information and search very well. The names of the most common tags, such as address or telephone, will need to be standardized. We will also need a standardized way for user extensions to the hierarchy. These are topics of our on-going work. 3.3
User Notification and Privacy
Collaboration regarding a common goal or a related motivation requires that users are notified by the iClouds device on a successful match. This can be a simple beep or a vibrating alarm as found in mobile phones today. For each item (search pattern) on the iWish-list, the user can specify, whether a notification should be sent or not. This enables a user to differentiate between pure collect pattern, e.g. Good Restaurants?, and patterns that require some kind of action, e.g. talk to the other person. iClouds devices are linked to their owners, broadcast information, and are traceable, hence they raise the question of user privacy. To protect user privacy, iWish-lists never leave the device, unless explicitly allowed by the user. Therefore, to construct a user profile is not possible. A user can mark each item on the
1044
A. Heinemann et al.
iHave-list as private. Private items will then be unavailable to other parties. In addition, the comparatively short communication range constitutes a natural protection for user privacy.
4
Prototype
To gain more practical experiences with iClouds, we have built a first prototype and set up a testbed. The prototype runs on Toshiba Pocket PC e740 (Windows CE) with integrated 802.11b cards. For the underlying link layer network connectivity, we run the PDAs in 802.11b ad hoc mode with statically assigned IP addresses The prototype was developed using PersonalJava from the Java2 Micro Edition (J2ME). Our information list data structures consist of strings. Currently, we have not yet implemented hierarchies. iList comparison is based on a simple substring matching function. A successful match will copy iHave-list items to new devices. The user is notified of this event by a beep from the device. This allows the user to check her updated information goods and plan further action. We use a UDP based ping/pong mechanism for scanning the vicinity for new nodes. New node discovery is done by periodically pinging every IP address within a given set. A node sends a ping message to other nodes in and waits for a pong message. Upon receiving a pong, the new node is added to the active neighborhood. Otherwise, after a certain timeout, the node is removed from the neighborhood. When a node encounters a new node, it pulls the iHave list from the new node using an HTTP GET request. Because a PDA is not a good device for data input, we believe that the iLists should be managed in a desktop application and the lists should be synced to the PDA during the normal sync operation.
5
Related Work
The Proem Platform [3] targets very similar goals. The main difference to iClouds is that they focus on Personal Area Networks (PAN) for collaboration. We believe that it is fruitful to focus on a wider area (mobile networks that cover several 100 meters in diameter) and that it is not necessary to encounter communication partners physically for information exchange. Sharing information among mobile, wireless users is also subject of the 7DS Architecture [5,6]. In contrast to iClouds, in 7DS the users are intermittently connected to the Internet and cache information, i.e., HTML pages, which is accessed during that time frame. Later these caches are used to fulfill requests among the nodes. The Usenet-on-the-fly system [2] makes use of channels to share information in a mobile environment. But the information spreading is limited by a hop count in the message. This has the disadvantage, that an unlucky user might be one hop to far away from the information source, although she might be interested in receiving the information.
iClouds – Peer-to-Peer Information Sharing in Mobile Environments
1045
Mascolo et al. describe XMIDDLE [4], a peer-to-peer middleware that allows the synchronization of XML documents across several mobile devices of one user. The data modeling and exchange in iClouds has to fulfill similar tasks, it has to work between different users though. Basic information services require contributions from users. This is true for many current systems. The Usenet news is certainly one of the most prominent and successful systems. Tveit [7] proposes a peer-to-peer based network of agents, that support product and service recommendations for mobile users. Recommendations are provided by aggregating and filtering individual user input. Tveit focuses on infrastructured wireless networks, e.g. mobile phone networks.
6
Conclusion
In this paper we have presented iClouds, an architecture for supporting spontaneous user interaction, collaboration, and transparent data exchange in a mobile ubiquitous computing context. iClouds achieves these goals through ad hoc peer-to-peer communications between wireless-enabled devices. iClouds supports many natural forms of interaction, such as browsing for information, searching, and advertising. We have devised a hierarchical information structure for storing and matching information in iClouds. We have also implemented a prototype of iClouds which runs on PDAs with wireless LAN cards. As part of our future work, we will develop a model for specifying the iClouds information hierarchies and ways to extend them. In addition, we plan on investigating ontologies as a possible way to improve the matching between items on the iLists.
References 1. Hartl, A. et al., Engineering Multimedia-Aware Personalized Ubiquitous Services. In IEEE MSE 2002, Newport Beach, CA, December 2002. 2. Becker, C., Bauer, M., H¨ ahner, J., Usenet-on-the-fly - Supporting Locality of Information in Spontaneous Networking Environments, Workshop on Ad Hoc Communications and Collaboration in Ubiquitous Computing Environments (at CSCW), New Orleans, LA, November 2002. 3. Kortuem, G. et al., When Peer-to-Peer comes Face-to-Face: Collaborative Peer-toPeer Computing in Mobile Ad-hoc Networks, International Conference on Peer-toPeer Computing, Link¨ oping, Sweden, August 2001. 4. Mascolo, C., Capra, L., Emmerich, W.: An XML-based Middleware for Peer-toPeer Computing. International Conference on Peer-to-Peer Computing, Link¨ oping. Sweden, August 2001. 5. Papadopouli, M., Schulzrinne, H., Seven degrees of separation in mobile ad hoc networks, IEEE GLOBECOM, 2000. 6. Papadopouli, M., Schulzrinne, H., Design and implementation of a peer-to-peer data dissemination and prefetching tool for mobile users, First NY Metro Area Networking Workshop, 2001. 7. Tveit, A., Peer-to-peer based Recommendations for Mobile Commerce, First International Workshop on Mobile Commerce, Rome, Italy, 2001.
Support for Personal and Service Mobility in Ubiquitous Computing Environments* K. El-Khatib**, N. Hadibi, and Gregor v. Bochmann School of Information Technology & Engineering, University of Ottawa 161 Louis Pasteur St., Ottawa, Ont., K1N 6N5, Canada {elkhatib, nhadibi, bochmann}@site.uottawa.ca
Abstract. This paper describes an agent-based architecture that extends personal mobility to ubiquitous environment. A software agent, running on a portable device, leverages the existing service discovery protocols to learn about all services available in the vicinity of the user. Short-range wireless technology such as Bluetooth can be used to build a personal area network connecting only devices that are close enough to the user. Acting on behalf of the user, the software agent runs a QoS negotiation and selection algorithm to select the most appropriate available service(s) to be used for a given communication session, as well as the configuration parameters for each service, based on session requirements, the user preferences and the constraints of the devices that provide the service(s). The proposed architecture supports also service hand-off to take account of service volatility as a result of user mobility.
1 Introduction Our work here is motivated by the growing need to provide personal mobility for persons roaming in ubiquitous computing environments. Personal mobility [1] is defined as the ability of a user to get access to telecommunication services from any terminal (e.g. workstations, notebooks, Personal Digital Assistants (PDA), cellular phones) at any time and from any place based on a unique identifier of the user, and the capability of the network to provide services in accordance with the user’s service profile. Closely related to the subject of personal mobility is service or session mobility [6], which refers to the possibility of suspending a service on a device and picking it up on another device at the same point where it was suspended. An example of service mobility is a call transfer from the mobile phone of the user to his office phone. Ubiquitous computing is a new trend in computation and communication; it is at the intersection of several technologies, including embedded systems, service discovery, wireless networking and personal computing technologies. It is best described by Mark Weiser, father of ubiquitous computing, as the world with “invisible” machines; a world such that “its highest ideal is to make a computer so imbedded, so ___________________________________ * This work was partially supported by the Communications and Information Technology Ontario (CITO) ** Currently working at the National Research Council of Canada.
H. Kosch, L. Böszörményi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 1046–1055, 2003. © Springer-Verlag Berlin Heidelberg 2003
Support for Personal and Service Mobility in Ubiquitous Computing Environments
1047
fitting, so natural, that we use it without even thinking about it”[2]. In such a ubiquitous environment, devices are only visible through the services they provide; specific information about the devices such as location, address or configuration parameter is transparent to the user. A user roaming in a ubiquitous environment might be surrounded by several services, depending on his/her current location. One of the major contributing factors to the big interest in ubiquitous computing is the advance in short-range radio frequency communication that created the notion of personal level communication infrastructure, referred to as Wireless Personal Area Networking (WPAN). Bluetooth [3] is an example of WPAN technology. Devices connected to the WPAN have the capability to locate, communicate, and provide services for each other. This capability additionally allows these devices to collaboratively provide an ad hoc distributed computing environment and to deliver services that cannot be possible delivered with only one device. For instance, a video display service can use an audio playing service in its vicinity to play an audio and video recording. Also, an audio play-out service can use the service of a PC connected to the Internet to download and play music from the Internet. Given these trends in personal communication, personal mobility comes as a challenge to ubiquitous environments. The architecture proposed in this paper is an improvement on our personal mobility architecture [8] for classical non-ubiquitous environments. It leverages technologies in short-range wireless communication, such as Bluetooth, to construct the WPAN; it also leverages service discovery protocols, such as Jini, SDP, and Salutation, to discover services available just in the WPAN of the roaming user. Running a service discovery protocol over a WPAN restricts the services available for the use of the user to devices that are within the WPAN and hence close enough to the user. The architecture supports also optional service mobility, which allows, when possible, the service to follow the user as he moves. A major component of our architecture is a Personal Agent (PA) process running on a personal device carried by the user; the PA triggers the service discovery, service selection and service mobility. It also enforces the user’s policies and preferences stated in the user profile. Throughout the paper, we will show how the presented architecture is used during a communication session between Alice, a team manager on a business trip, and her team-members. The elaboration of the scenario is presented in Section 3. Before the meeting, Alice would like to have small chat with Bob, who is a team leader in her group. Using the multimedia workstation in the business office of the hotel, Alice sends an invitation to Bob ten minutes before the meeting time. Bob, sitting in the lounge area, receives the call on his PDA for a multimedia conversation with Alice. Since Bob has indicated in his profile that he is always willing to accept calls from his manager Alice, Bob’s PDA tries to find a microphone, a speaker, a video display service and a camera to make for a full multimedia session. Assuming that such services exist in the lounge area, the PDA can discover and reserve them for the communication session with Alice. The PDA sends back the addresses of the play-out services, and the videoconference is started between Alice and Bob. When the time for the meeting comes, Bob moves with his PDA into the conference room where all the team members are waiting. Bob’s PDA detects that the services that Bob was using are not available anymore, and since he has already set the FOLLOW-ME option of the session to ON, his PDA tries to discover similar services to continue the session in the conference room. The PDA then detects and
1048
K. El-Khatib, N. Hadibi, and G. v. Bochmann
selects the big screen, the camera, the speaker as well as the microphone of the conference room; Alice’s picture appears on the big screen and she is now ready to participate in the meeting. The rest of the paper is organized as follows: Section 2 starts by presenting a literature review of a number of architectures for personal mobility with highlights to their limitations in ubiquitous environments. We then propose our architecture for supporting personal mobility in ubiquitous environment and its main components. After that, we present the algorithm used for service and Quality of Service (QoS) parameter values selection and explain how our architecture supports service mobility. Section 3 continues with more details about the usage scenario introduced in Section 1. We finally conclude in Section 4.
2 Architecture for Personal and Service Mobility 2.1 Related Architectures A number of architectures [4, 5, 6, 7, 8] were proposed to solve the personal mobility problem in the context of the Internet. All these architectures share the same concept of a Directory Service (DS) that provides a user-customized mapping from a unique user identifier to a device that is best suitable for the user to use. The basic idea of these architectures is that the user keeps a profile in the DS including static information about all the devices he/she uses, and the logic or policy for when to use these devices (user preferences). During session initiation, the DS executes the logic in the user profile and handles the communication request according to the user’s specified preferences. While these architectures provide personal mobility for users with multiple telecommunication devices (telephone, PDA, laptop, cellular phone, pager), they all fail short to extend personal mobility to ubiquitous environments because: - the information in the directory service is static, - there is no support for service or device discovery, - they lack support for service mobility, and finally - they lack support for complex (combined) services. A number of researchers have addressed the problem of service selection in ubiquitous computing environments. The work in [9] presented two approaches for selecting services based on the physical proximity and line of sight of the handheld device relative to the service. The authors in [10] used a centralized approach where a central gateway is responsible for delegating rendering tasks to devices in the environment of the user. A PDA carried by the user is responsible for detecting the devices, and sending the list of available devices to the central gateway. Both these architectures suffer from the drawback of using infrared communication for finding and selecting services. Because infrared communication requires the knowledge of the location of the existing devices due to the line-of-sight restriction, these architectures cannot be used in ubiquitous environments because they require user’s awareness of the available services. Service mobility and QoS issues are also not discussed in these works. In another work [11], the authors investigated the use of a browser running on a PDA to enable ubiquitous access to local resources as well as resources on the
Support for Personal and Service Mobility in Ubiquitous Computing Environments
1049
World Wide Web. The browser, called the Ubicompbrowser, detects devices and resources in the environment of the user, and delegates the rendering of the requested resources to these devices in order to overcome the limitation of the PDA. A major drawback of the Ubicompbrowser is that it requires the user to know its current location to restrict the set of available services. Additionally, the Ubicompbrowser does not deal with the issue of QoS negotiation, neither with the issue of service mobility. 2.2 Proposed Architecture Our work presented here is inspired by the Ubicompbrowser project, and is intended to support personal mobility in ubiquitous environments. Our architecture builds on our previous architecture for personal mobility [8] and includes additional functionalities to overcome its shortcomings in ubiquitous environments. The modified architecture uses the short-range Bluetooth wireless communication to construct the user’s WPAN, and to restrict the domain of services available to the user just to those running on devices that are within this WPAN. Our architecture differs also from the architecture in [9,10] in that service selection is done automatically on behalf of the user without requiring the user to point and select each service individually using infrared, since the user might not (and should not) be aware of the services and their locations. We also address the problem of service mobility by using periodical search for services similar1 to the services currently used by the user, in order to provide smooth hand-off for these services. Our previous architecture for personal mobility [8] is based on the concept of a Home Directory. The Home Directory (HD) has two functions: (a) storage of the user’s profile and (b) forwarding of incoming communication requests. As a storage facility, the HD is a personalized database of users profiles, with each profile holding the user’s contact information, current access information, preferences for call handling, presence service, authentication, authorization, and accounting information. With this logic stored with the data in the profile, end users can control the access permission to their devices according to their own preferences. To forward communication requests, a Home Directory Agent (HDA) executes the logic in the user profile every time the user receives an incoming call through the HD. The user can also invoke the HDA in order to find (and establish) the best way to place an outgoing communication request. In a ubiquitous computing environment, the set of available devices for the user may change continuously as the user changes his/her location. Updating manually the information about currently available services is not a feasible. Additionally, discovering the services and sending update messages to the HDA is not a practical solution since the set of available services might change very often, which results in many update messages. Moreover, an update message might incur a certain delay so that by the time the message is received by the HDA, the information included in the message might be outdated. To overcome these limitations, we propose to run a modified version of the HDA on a hand-held device, such as a PDA, that is carried by the user. We call this 1
We say that two services are similar if they serve the same purpose, for instance a TV and a wall projector, or a PC speakers and a mini-stereo.
1050
K. El-Khatib, N. Hadibi, and G. v. Bochmann
modified version of the HDA the Personal Agent (PA), and it is responsible for detecting devices in the vicinity of the user as well as managing the user’s communication sessions. We require that the hand-held device, on which the Personal Agent runs, have access to the Internet (through a wireless modem or IEEE 802.11[12] connection) in order to retrieve the user profile and send/receive communication requests through the HDA. The PDA is also supposed to be able to construct a Wireless Personal Area Network (such as Bluetooth WPAN) in order to be able to detect and communicate with other wireless devices just around the user. For the rest of the paper, we will assume that the PA is running on a PDA that satisfies these communication requirements. At any one time, either the HDA or the PA is providing personal mobility service to the user. When the PDA is switched ON, the PA contacts the HDA to retrieve the user profile. From that point on until the PDA is switched OFF, the PA is responsible for executing the logic in the user profile, and the HDA would act only as a proxy for incoming call requests. To ensure that the HDA is aware of the status of the PA, replies to communication requests are sent back through the HDA. The HDA can detect when the PA is not running or the PDA is currently out of reach if the HDA does not see a reply to a forwarded call after a certain time-out period. The HDA would then switch into active mode, and handle the communication request according to the rules specified in its local copy of the user profile. The architecture of the Personal Agent has four major components: a Service Discovery Agent, a User Activity Monitor Agent (UAMA), a QoS Selection and Negotiation Agent (QSNA), and a Service Registry. Fig. 1 shows the architecture of the Personal Agent with its components. Following is a detailed description of each of these components. - Service Discovery Agent and Service Registry: the function of the Service Discovery Agent (SDA) is to search for all services in the WPAN of the user. The SDA provides the QoS Selection and Negotiation Agent (QSNA) (discussed below) with the list of currently available services. Since different services might be using different service discovery protocols (JINI [13], SDP [14], SLP [15]), the SDA shall act as a service discovery client in multiple service discovery protocols. Once a session has been started, the SDA periodically searches for services that are similar to the services currently used in the session, and stores this information in the Service Registry (SR). Information in the SR allows smooth service mobility, as discussed in Section 2.4. - User Activity Monitor Agent (UAMA): The UAMA keeps track of the current activity of the user (idle, busy…) as well as the services he is currently using. The UAMA assists the QSNA during the service selection phase by providing up-tothe-minute information about the user’s status. The UAMA can also incorporate information about the context of the user in the service selection process. Context information includes the location of the user, and whether he is by himself or surrounded by other people, as suggested in [11,16]. – QoS Selection and Negotiation Agent (QSNA): based on the session requirements2, the information in the user profile, the list of available services
2
Session requirements may be described using the Session Description Protocol (SDP) carried in the SIP INVITE message.
Support for Personal and Service Mobility in Ubiquitous Computing Environments
1051
User Profile
Call Request QoS Selection and Negotiation Agent Selected Services
Service Discovery Agent
User Activity Monitor Agent
Service Registry
Fig. 1. Components of the Personal Agent
provided by the SDA, and the user’s current activity provided by the UAMA, the QSNA selects the best services that satisfy the session requirements, and complying to the preferences of the user (described in the user profile). The QSNA might also generate a mix and match several currently available services to satisfy the requirements of the session. The QSNA implements the service and QoS parameter values selection algorithm presented in the next section. 2.3 Service and QoS Parameter Values Selection Algorithm To provide personal mobility in a ubiquitous environment, a decision has to be made concerning (a) the service(s) (camera, speakers, microphone…) that should be used among the discovered services that meet the requirements of the communication session, and (b) what Quality of Service (QoS) parameter values (frame rate, frame resolution, audio quality…) should be used for each service. These decisions are affected by many factors including the media type of the exchanged content, the hardware and software capabilities of the devices where the services are running (also called device limitations), the preferences of the user and finally his current context. When the PA receives an incoming call through the HDA, and assuming that the user is willing to accept calls now, the first step in the selection process consists of selecting the services that are required for establishing the communication session. Based on the session requirements, the QSNA queries the SDA for the necessary services for the session. For instance, a communication session that includes the receipt of audio media requires the discovery and selection of an audio playing service. The second step in the selection process consists of selecting the QoS parameter values for each service, such as the audio quality for an audio service and the video frame rate and resolution for a video service. This selection is based on the hardware/software limitations of the device, network condition, and the preferences of the user. For instance, the network access line (e.g. modem or wireless connection) of one of the devices may limit the maximum throughput of that device Also, a video content that requires a high-resolution display eliminates the possibility of using the
1052
K. El-Khatib, N. Hadibi, and G. v. Bochmann
display service of a monochrome device, especially if the user expressed his desire to receive only high quality video. This selection process also deals with the case when multiple similar services exist in the environment of the user. The QSNA always selects the services that give the user the highest possible satisfaction. We base the selection of QoS parameters for each service on the concept of maximizing the user’s satisfaction. In [8], we presented an extension to the work presented in [17], wherein, the QoS parameters for each service are selected based on a user satisfaction function. In his/her profile, the user indicates a satisfaction function that maps a QoS parameter value to a satisfaction value in the [0..1] range. The user also assigns a weight value for each media type. The total satisfaction value of the user for a combination of services is based on weighted combination of his/her satisfaction with the media type and the individual QoS parameter value of the service. Using all possible combinations of QoS parameters of all available services, we select the combination that generates the maximum satisfaction within the restrictions of the device where the service is running and the preferences of the user. More details about the selection algorithm are given in [8]. 2.4 Support for Service Mobility During a communication session, a nomadic user in a ubiquitous environment might move away from one device and get closer to another device that provides a similar service. Service mobility is required since the life span of the communication session might be longer than the time that the currently used device is available in the user’s WPAN. When a device that is currently used becomes unavailable, the Personal Agent should switch to another device providing a similar service if available or it should inform the user about the disappearance of the service. For instance, a user moving away from his computer and entering the conference room should have, if desired, the multimedia session transferred from his computer to the TV and stereo system in the conference room. If the conference room does not have a TV set, the user should be warned that the video display service would be discontinued if he/she stays in the conference room. Since our architecture is designed to support nomadic users in ubiquitous environments, the architecture supports service mobility through service hand-off, transparently to the user. A smooth transparent service handoff requires continuous discovery of currently available services and updating of the service registry to provide smooth service transfer. The SDA periodically updates the information in the local service registry (SR). Updates are carried out only when a communication session is in progress. When a connection to a service is fading, the SDA informs the QSNA about a replacement service, which sends the coordinates of the replacement service to the other party. The Personal Agent may also use several heuristics during service selection such as selecting the services that have the longest life span, even with lower quality, in order to avoid service disruption and minimize the number of service hand-off.
Support for Personal and Service Mobility in Ubiquitous Computing Environments
1053
Fig. 2. Session establishment based on the Personal Agent
3
Usage Scenario (Continued)
In this section, we will elaborate more on the scenario presented in Section 1. We will assume that the SIP [18] signaling protocol is used to establish and maintain the communication session. The scenario of Alice trying to reach Bob, who is in the lounge area, is divided into five phases (Fig. 2), with the first phase executed only once when Bob switches ON his PDA. We will assume that Bob has enabled the service mobility option. Due to the space limitation, we will only give a short description of each phase:
1054
K. El-Khatib, N. Hadibi, and G. v. Bochmann
- Startup Phase: The Personal Agent retrieves the user’s profile from the Home Directory Agent (Messages 1-2). - Session Initiation Phase: Bob’s Home Directory Agent forwards the request to the Personal Agent (Messages 3-5). The SDA uses the service discovery protocol to discover the available services for the session and update the SR (Messages 68). The QSNA selects from the SR the services for the session based on the session requirements, the user profile, network condition, and device/service profile, as described in Section 2.4. The QSNA might also mix-and-match several devices to provide compound services. The Personal Agent sends back to Alice the address of the selected services. (Messages 9-11) - Data Exchange Phase: The data is exchanged between Alice’s device and the selected devices from Bob’s environment. - Session Maintenance Phase: As long as the session is still running, the SDA periodically queries the environment for services that are similar to the services used in the session. This information is used to update the SR in order to reduce the delay in service mobility. (See Section 2.4 for more details). The SDA will also keep querying for services that temporary were not available but were optional for the session. When Bob moves to the conference room, the SDA detects all the audio and video services of the conference room. (Messages 12-14) - Service Hand-off Phase: In case a service that is currently used becomes unavailable because of the mobility of the user, the SDA informs the QSNA of the replacement service(s) (in this scenario, the replacement services are the services of the conference room). The QSNA in turns sends an update message to Alice with the new address of the service. (Messages 15-16)
4 Conclusion In this paper, we have presented an architecture for supporting personal mobility in ubiquitous environments. The architecture allows nomadic users to benefit from the availability of large number of hidden services in a ubiquitous environment to establish communication sessions. To construct this architecture, we introduced a new component that we called the Personal Agent (PA) that acts on behalf of the user during the service discovery and selection process. The Personal Agent also provides support for service mobility through periodic updates of currently available services into a local service registry. We have also shown the functionality of the Personal Agent during a typical communication session using an example scenario. Presently, we have implemented the architecture and we are studying its feasibility and usability.
Acknowledgment. The authors would like to thank Eric Zhen Zhang for his helpful comments and for implementing a prototype of the architecture.
Support for Personal and Service Mobility in Ubiquitous Computing Environments
1055
References 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18.
Schulzrinne, H.: Personal mobility for multimedia services in the Internet. In European Workshop on Interactive Distributed Multimedia Systems and Services (IDMS), (Berlin, Germany), Mar. 1996. http://www.ubiq.com/weiser Haartseen, J., Naghshineh, M., Inouye, J.: Bluetooth: Vision, Goals, and Architecture. ACM Mobile Computing and Communications Review, Vol. 2, No. 4, October 1998, pp. 38–45 Anerousis, N. et. al.: TOPS: An architecture for telephony over packet networks,” IEEE Journal of Selected Areas in Communications, Jan. 1999 Roussopoulos, M., Maniatis, P., Swierk, E., Lai, K., Appenzeller, G., Baker,M.: Personlevel Routing in the Mobile People Architecture. Proceedings of the USENIX Symposium on Internet Technologies and Systems, October 1999 Kahane, O., Petrack, S.: Call Management Agent System: Requirements, Function, Architecture and Protocol. IMTC Voice over IP Forum Submission VOIP97-010, 1997 Wang, H. et. al.: ICEBERG: An Internet-core Network Architecture for Integrated Communications. IEEE Personal Communications (2000): Special Issue on IP-based Mobile Telecommunication Networks He, X., El-Khatib, K., Bochmann, G.v.: A communication services infrastructure including home directory agents. U. of Ottawa, Canada. May 2000 Barbeau, M., Azondekon, V., Liscano, R.: Service Selection in Mobile Computing Based on Proximity Confirmation Using Infrared. MICON 2002 Schneider, G., Hoymann, C., Goose, S.: Ad-hoc Personal Ubiquitous Multimedia Services via UpnP. Proceedings of the IEEE International Conference on Multimedia and Exposition, Tokyo, Japan; Aug. 2001 Beigl, M., Schmidt, A., Lauff, M., Gellersen, H.W.: The UbicompBrowser. Proceedings of the 4th ERCIM Workshop on User Interfaces for All, October 1998, Stockholm, Sweden IEEE802.11: http://grouper.ieee.org/groups/802/11/main.html JINI (TM): Http: //java.sun.com/product/JINI/. 1998 Bluetooth Special Interest Group (SIG): Service Discovery Protocol,” SIG Specification version 1.0, Volume 1 Core, part E, pp 233–384 Guttman, E., Perkins, C., Veizades, J., Day, M.: Service Location Protocol. Version 2. http://ietf.org/rfc/rfc2608.txt. Horvitz, E., Jacobs, A., Hovel, D.: Attention-Sensitive Alerting. In: Proceedings of UAI ‘99, Stockholm, Sweden, July 1999, pp. 305–313. San Francisco: Morgan Kaufmann A. Richards, A., Rogers, g., Witana, V., Antoniades, M.: Mapping User Level QoS from a Single Parameter. In Second IFIP/IEEE International Conference on Management of Multimedia Networks and Services, Versailles, November 1998 Handley, M., Schulzrinne, H., Scholler, E., Rosenberg, J.: SIP: Session Initiation Protocol. RFC2543, IETF, March 1999
Dynamic Layouts for Wireless ATM Michele Flammini1 , Giorgio Gambosi2 , Alessandro Gasparini1 , and Alfredo Navarra1,3 1
3
Dip. di Informatica, University of L’Aquila, Italy. {flammini,gasparin,navarra}@di.univaq.it 2 Dip. di Matematica, University of Rome “Tor Vergata”, Italy.
[email protected] MASCOTTE project, INRIA/Universit´e de Nice–Sophia Antipolis, France.
[email protected] Abstract. In this paper we present a new model able to combine quality of service (QoS) and mobility aspects in wireless ATM networks. Namely, besides the standard parameters of the basic ATM layouts, we introduce a new one, that estimates the time needed to reconstruct the virtual channel of a wireless user when it moves through the network. QoS guarantee dictates that the rerouting phase must be imperceptible. Therefore, a natural combinatorial problem arises in which suitable trade-offs must be determined between the different performance measures. We first show that deciding the existence of a layout with maximum hop count h, load l and distance d is NP-complete, even in the very restricted case h = 2, l = 1 and d = 1. We then provide optimal layout constructions for basic interconnection networks, such as chains and rings.
1
Introduction
Wireless ATM networks are emerging as one of the most promising technologies able to support users mobility while maintaining the QoS offered by the classical ATM protocol for Broadband ISDN [2]. The mobility extension of ATM gives rise to two main application scenarios, called respectively End-to-End WATM and WATM Interworking [13]. While the former provides seamless extension of ATM capabilities to users by allowing ATM connections that extend until the mobile terminals, the latter represents an intermediate solution used primarily for high-speed transport over network backbones by exploiting the basic ATM protocol with additional mobility control capabilities. Wireless independent subnets are connected at the borders of the network backbone by means of specified ATM interface nodes, and users are allowed to move among the different wireless
Work supported by the IST Programme of the EU under contract number IST1999-14186 (ALCOM-FT), by the EU RTN project ARACNE, by the Italian project REAL-WINE, partially funded by the Italian Ministry of Education, University and Research, by the French MASCOTTE project I3S-CNRS/INRIA/Univ. Nice–Sophia Antipolis and by the Italian CNR project CNRG003EF8 – “Algoritmi per Wireless Networks” (AL-WINE).
H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 1056–1063, 2003. c Springer-Verlag Berlin Heidelberg 2003
Dynamic Layouts for Wireless ATM
1057
subnets. In both scenarios, the mobility facility requires the efficient solution of several problems, such as handover (users movement), routing, location management, connection control and so forth. A detailed discussion of these and other related issues can be found in [13,6,5,19,17]. The classical ATM protocol for Broadband ISDN is based on two types of predetermined routes in the network: virtual paths or VPs, constituted by a sequence of successive edges or physical links, and virtual channels or VCs, each given by the concatenation of a proper sequence of VPs [15,14,18]. Routing in virtual paths can be performed very efficiently by dedicated hardware, while a message passing from one virtual path to another one requires more complex and slower elaboration. A graph theoretical model related to this ATM design problem has been first proposed in [12,7]. In such a framework, the VP layouts determined by the VPs constructed on the network are evaluated mainly with respect to two different cost measures: the hop count, that is the maximum number of VPs belonging to a VC, which represents the number of VP changes of messages along their route to the destination, and the load, given by the maximum number of virtual paths sharing an edge, that determines the size of the VP routing tables (see, e.g., [8]). For further details and technical justifications of the model for ATM networks see for instance [1,12]. While the problem of determining VP layouts with bounded hop count and load is NP-hard under different assumptions [12,9], many optimal and near optimal constructions have been given for various interconnection networks such as chain, trees, grids and so forth [7,16,10,11,20,4] (see [21] for a survey). In this paper we mainly focus on handover management issues in wireless ATM. In fact, they are of fundamental importance, as the virtual channels must be continually modified due to the terminals movements during the lifetime of a connection. In particular, we extend the model of [12,7] in order to combine QoS and mobility aspects in wireless ATM networks. Typical handover managements issues are the path extension scheme, in which a VC is always extended by a virtual path during a handover [5], or the anchor-based rerouting and the nearest common node rerouting [13,3], that involve the deletion of all the VPs of the old VC and the addition of all the VPs of the new one after a common prefix of the two VCs. Other handover strategies can be found in [13,6,5]. Starting from the above observations, besides the standard hop count and load performance measures, we introduce the new notion of virtual channel distance, that estimates the time needed to reconstruct a virtual channel during a handover phase. In order to make the rerouting phase imperceptible to users and thus to obtain a sufficient QoS, the maximum distance between two virtual channels must be maintained as low as possible. Therefore, a natural combinatorial problem arises in which suitable trade-offs must be determined between the different performance measures. The paper is organized as follows. In the next section we introduce the model, the notation and the necessary definitions. In Section 3 we provide hardness
1058
M. Flammini et al.
results for the layout construction problem. In Section 4 and 5 we provide optimal layouts for chains and rings, respectively. Finally, in Section 6, we give some conclusive remarks and discuss some open questions.
2
The WATM Model
We model the network as an undirected graph G = (V, E), where nodes in V represent switches and edges in E are point-to-point communication links. In G there exists a subset of nodes U ⊆ V constituted by cells with corresponding radio stations, i.e., switches adapted to support mobility and having the additional capability of establishing connections with the mobile terminals. A distinguished source node s ∈ V provides high speed services to the users moving along the network. We observe that, according to the wireless nature of the system, during the handover phase mobile terminals do not necessarily have to move along the network G, but they can switch directly from one cell to another, provided that they are adjacent in the physical space. It is thus possible to define a (connected) adjacency graph A = (U, F ), whose edges in F represent adjacencies between cells. A layout Ψ for G = (V, E) with source s ∈ V is a collection of paths in G, termed virtual paths (VPs for short), and a mapping that defines, for each cell u ∈ U , a virtual channel V CΨ (u) connecting s to u, i.e., a collection of VPs whose concatenation forms a shortest path in G from s to u. Definition 1. [12] The hop count hΨ (u) of a node u ∈ U in a layout Ψ is the number of VPs contained in V CΨ (u), that is |V CΨ (u)|. The maximal hop count of Ψ is Hmax (Ψ ) ≡ maxu∈U {hΨ (u)}. Definition 2. [12] The load lΨ (e) of an edge e ∈ E in a layout Ψ is the number of VPs ψ ∈ Ψ that include e. The maximal load Lmax (Ψ ) of Ψ is maxe∈E {lΨ (e)}. As already observed, when passing from a cell u ∈ U to an adjacent one v ∈ U , the virtual channel V CΨ (v) must be reconstructed from V CΨ (u) changing only a limited number of VPs. Once fixed V CΨ (u) and V CΨ (v), denoted as V CΨ (u, v) the set of VPs in the subchannel given by the longest common prefix of V CΨ (u) and V CΨ (v), this requires the deletion of all the VPs of V CΨ (u) that occur after V CΨ (u, v), plus the addition of all the VPs of V CΨ (v) after V CΨ (u, v). The number of removed and added VPs, denoted as D(V CΨ (u), V CΨ (v)), is called the distance of V CΨ (u) and V CΨ (v) and naturally defines a channel distance measure dΨ between pairs of adjacent nodes in A. Definition 3. The channel distance of two nodes u and v such that {u, v} ∈ F (i.e., adjacent in A) is dΨ (u, v) = D(V CΨ (u), V CΨ (v)) = hΨ (u) + hΨ (v) − 2|V CΨ (u, v)|. The maximal distance of Ψ is Dmax (Ψ ) ≡ max{u,v}∈F {dΨ (u, v)}. It is now possible to give the following definition on WATM layouts. Definition 4. A layout Ψ with Hmax (Ψ ) ≤ h, Lmax (Ψ ) ≤ l and Dmax (Ψ ) ≤ d is a h, l, d-layout for G, s and A.
Dynamic Layouts for Wireless ATM
1059
In the following, when the layout Ψ is clear from the context, for simplicity we will drop the index Ψ from the notation. Moreover, we will always assume that all the VPs of Ψ are contained in at least one VC. In fact, if such property does not hold, the unused VPs can be simply removed without increasing the performance measures h, l and d.
3
Hardness of Construction
In this section we show that constructing optimal dynamic layouts is in general an NP-hard problem, even for the very simple case h = 2 and l = d = 1. Notice that when d = 1, for any two cells u, v ∈ U adjacent in A = (U, F ), during an handover from u to v by definition only one VP can be modified. This means that in every h, l, 1-layout Ψ , either V C(v) is a prefix of V C(u) and thus V C(v) is obtained from V C(u) by adding a new VP from u to v, or vice versa. In any case, a VP between u and v must be contained in Ψ . As a direct consequence, the virtual topology defined by the VPs of Ψ coincides with the adjacency graph A. Theorem 1. Given a network G = (V, E), a source s ∈ V and an adjacency graph A = (U, F ), deciding the existence of a 2, 1, 1-layout for G, s and A is an NP-complete problem. For h = 1, any l and any d, the layout construction problem can be solved in polynomial time by exploiting suitable flow constrictions like the ones presented in [9].
4
Optimal Layouts for Chain Networks
In this section we provide optimal layouts for chain networks. More precisely, we consider the case in which the physical graph is a chain Cn of n nodes, that is V = {1, 2, . . . , n} and E = {{v, v + 1}|1 ≤ v ≤ n − 1}, and the adjacency graph A coincides with Cn . Moreover, without loss of generality, we take the leftmost node of the chain as the source, i.e. s = 1, as otherwise we can split the layout construction problem into two equivalent independent subproblems for the left and the right hand sides of the source, respectively. Finally, we always assume d > 1, as by the same considerations of the previous section the virtual topology induced by the VPs of any h, l, 1-layout Ψ coincides with the adjacency graph A and thus with Cn . Therefore, the largest chain admitting a h, l, 1-layout is such that n = h + 1. In the following we denote by u, v the unique VP corresponding to the shortest path from u to v in Cn and by s, v1 v1 , v2 . . . vk , v or simply s, v1 , v2 , . . . , vk , v the virtual channel V C(v) of v given by the concatenation of the VPs s, v1 , v1 , v2 , ..., vk , v. Clearly, s < v1 < v2 < . . . < vk < v. Definition 5. Two VPs u1 , v1 and u2 , v2 are crossing if u1 < u2 < v1 < v2 . A layout Ψ is crossing-free if it does not contain any pair of crossing VPs.
1060
M. Flammini et al.
Definition 6. A layout Ψ is canonic if it is crossing-free and the virtual topology induced by its VPs is a tree. According to the following definition, a h, l, d-layout for chains is optimal if it reaches the maximum number of nodes. Definition 7. Given fixed h,l,d and a h, l, d-layout Ψ for a chain Cn , Ψ is optimal if no h, l, d-layout exists for any chain Cm with m > n. We now prove that for every h,l,d, the determination of an optimal h, l, dlayout can be restricted to the class of the canonic layouts. Theorem 2. For every h, l, d, any optimal h, l, d-layout for a chain is canonic. Motivated by Theorem 2, in the remaining part of this section we focus on canonic h, l, d-layouts for chains, as they can be the only optimal ones. Let us say that a tree is ordered if it is rooted and for every internal node a total order is defined on its children. As shown in [12], an ordered tree induces in a natural way a canonic layout and vice versa. Therefore, there exists a bijection between canonic layouts and ordered trees. We now introduce a new class of ordered trees T (h, l, d) that allows to completely define the structure of an optimal h, l, d-layout. Informally, denoted as T (h, l) the ordered tree corresponding to optimal layouts with maximum hop count h and load l without considering the distance measure [11], T (h, l, d) is a maximal subtree of T (h, l) with the additional property that the distance between two adjacent nodes in the preorder labelling of the ordered tree, and thus between two adjacent nodes in the induced layout, is always at most d. Moreover, the containment of T (h, l, d) in T (h, l) guarantees that the hop count h and the load l are not exceeded in the induced layout. The definition of T (h, l, d) is recursive and the solution of the associated recurrence gives the exact number of the nodes reached by an optimal h, l, dlayout. Before introducing T (h, l, d), let us define another ordered tree that is exploited in its definition. Definition 8. Given any h, l, d, T (h, l, d) is an ordered tree defined recursively as follows. T (h, l, d) is obtained by joining the roots of min{h, d − 1} subtrees T (i, l − 1, d) with h − min{h, d − 1} + 1 < i ≤ h in such a way that the root of T (i − 1, l − 1, d) is the rightmost child of the root of T (i, l − 1, d). A last node is finally added as the rightmost child of T (h − min{h, d − 1} + 1, l − 1, d). Trees T (0, l, d) and T (h, 0, d) consist of a unique node. Definition 9. The ordered tree T (h, l, d) is defined recursively as the join of the roots of the tree T (h − 1, l, d) and the tree T (h, l − 1, d) in such a way that the root of T (h − 1, l, d) is the rightmost child of the root of T (h, l − 1, d). Trees T (0, l, d) and T (h, 0, d) consist of a unique node. The following lemma establishes that T (h, l, d) is the ordered tree induced by an optimal h, l, d-layout.
Dynamic Layouts for Wireless ATM
1061
Lemma 1. The layout Ψ induced by T (h, l, d) is a h, l, d-layout. Moreover, every canonic h, l, d-layout Ψ induces an ordered subtree of T (h, l, d). Let Tn (h, l, d) and Tn (h, l, d) denote the number of nodes in T (h, l, d) and in T (h, l, d), respectively. Directly from Definition 8 and 9, it follows that h Tn (h, l, d) = Tn (h, l − 1, d) + Tn (h − 1, l, d) = k=0 Tn (k, l − 1, d), where the value of every Tn (k, l − 1, d) for 0 ≤ k ≤ h is obtained by the following recursive equation: 1 if l = 0 or h = 0, min{h,d−1}−1 Tn (h, l, d) = 1 + j=0 Tn (h − j, l − 1, d) otherwise. Before solving the above recurrence, we recall that given n+1 positiveintegers m m, k1 , . . . , kn such that m = k1 + · · · + kn , the multinomial coefficient k1 ,...,k n is defined as k1 !·k2m! !·····kn ! . Lemma 2. For every h, l, d, Tn (h, l, d) = l h−1
i=0 j=0
0 ≤ kd−2 ≤ kd−3 ≤ . . . ≤ k2 ≤ k1 ≤ i k1 + k2 + . . .+kd−2 = j
i i−k1 , k1 −k2 , . . . , kd−3 −kd−2 , kd−2
.
The following theorem is a direct consequence of Lemma 1, Lemma 2 and Definition 9. Theorem 3. For every h, l, d, the maximum numberof nodes reachable on a h chain network by a h, l, d-layout is Tn (h, l, d) = 1 + k=1 Tn (k, l − 1, d). More details will be shown in the full version of the paper.
5
Optimal Layouts for Ring Networks
In this section we provide optimal layouts for ring networks Rn with V = {0, 1, . . . , n − 1} and E = {{i, (i + 1) mod n}|0 ≤ i ≤ n − 1}. Again we assume that the adjacency graph A coincides with Rn and without loss of generality we take s = 0 as the source node. Moreover, we let d > 1, since as remarked in Section 3, no layout with maximum distance 1 exists for cyclic adjacency graphs. Notice that in any h, l, d-layout Ψ for Rn , by the shortest path property, if n is odd the nodes in the subring [1, n2 ] are reached in one direction from the source, say clockwise, while all the remaining ones anti-clockwise. This means that Ψ can be divided into two separated sublayouts Ψc and Ψa respectively for the subchains of the nodes reached clockwise in Ψ , that is [0, n2 ], and anticlockwise, that is from n2 to 0 in clockwise direction, extremes included. However, the results of the previous section for chains do not extend in a trivial way, as a further constraint exists for the final nodes n2 and n2 , that are adjacent in A and thus must be at distance at most d in Ψ . A similar observation holds when n is even.
1062
M. Flammini et al.
As for chains, let us say that a h, l, d-layout Ψ for rings is optimal if it reaches the maximum number of nodes. Moreover, let us call Ψ canonic if the clockwise and anticlockwise sublayouts Ψc and Ψa are both crossing-free and the virtual topologies induced by their VPs are trees. The following lemma is the equivalent of Theorem 2 for rings. Lemma 3. For every h, l, d, there exists an optimal h, l, d-layout for rings that is canonic. Starting from Lemma 3, we generalize the ordered tree T (h, l, d) to T (h, l, d, t) by adding a further parameter t ≤ h, which fixes the hop count of the rightmost leaf to t. Roughly speaking, T (h, l, d, h) = T (h, l, d) and T (h, l, d, d − 1) = T (h, l, d). More precisely, T (h, l, d, t) is defined recursively as the join of the roots of min{h, t} subtrees T (i, l − 1, d) for h − min{h, t} < i ≤ h in such a way that for i < h the root of a T (i, l − 1, d) is the rightmost child of the root of a T (i+1, l−1, d), plus a final node as rightmost child of T (h−min{h, t}+1, l−1, d). h Thus, Tn (h, l, d, t) = 1 + k=h−min{h,t}+1 Tn (k, l − 1, d) Lemma 1 extends directly to T (h, l, d, t), that in turn corresponds to an optimal h, l, d-layout for a chain with the further property that the rightmost node (opposite of the source) has hop count t. Therefore, it is possible to prove the following theorem. Theorem 4. The maximum number of nodes reachable on a ring network by a h, l, d-layout is 2Tn (h, l, d, d2 ) − ((d + 1) mod 2), with Tn (h, l, d, d2 ) = h 1 + k=h−min{h, d }+1 Tn (k, l − 1, d). 2
6
Conclusion
We have extended the basic ATM model presented in [12,7] to cope with QoS and mobility aspect in wireless ATM networks. This is obtained by adding a further measure, the VCs distance, that represents the time needed to reconstruct connecting VCs when handovers occur and must be maintained as low as possible in order to avoid the rerouting mechanism to be appreciated by the mobile users. We have shown that finding suitable trade-offs between the various performance measures is in general an intractable problem, while optimal constructions have been given for chain and ring topologies. Among the various questions left open, we have the extension of our results to more general topologies. Moreover, another worth investigating issue is the determination of layouts in which the routed paths are not necessarily the shortest ones, but have a fixed stretch factor or even unbounded length. Finally, all the results should be extended to other communication patterns like all-to-all.
References 1. S. Ahn, R.P. Tsang, S.R. Tong, and D.H.C. Du. Virtual path layout design on ATM networks. In Proceedings of INFOCOM, pages 192–200, 1994.
Dynamic Layouts for Wireless ATM
1063
2. I. Akyildiz, J. McNair, J. Ho, H. Uzunalioglu, and W. Wang. Mobility management in next-generation wireless systems. In Proceedings of the IEEE, volume 87, pages 1347–1384, 1999. 3. Bora A. Akyol and Donald C. Cox. Rerouting for handoff in a wireless ATM network. In Proceedings of the 5th IEEE International Conference on Universal Personal Communications (ICUPC), 1996. 4. L. Becchetti, P. Bertolazzi, C. Gaibisso, and G. Gambosi. On the design of efficient ATM schemes. In Proceedings of SOFSEM, volume 1338 of LNCS, pages 375–382. Springer-Verlag, 1997. 5. M. Cheng, S. Rajagopalan, L. Chang, G. Pollini, and M. Barton. PCS mobility support over fixed ATM networks. IEEE Communications Magazine, 35:82–92, 1997. 6. C. Chrysostomou, A. Pitsillides, and F. Pavlidou. A survey of wireless ATM handover iusses. In Proceedings of the 3th International Symposium of 3G Infrastructure and Services (3GIS), pages 34–39, 2001. 7. I. Cidon, O. Gerstel, and S. Zaks. A scalable approach to routing in ATM networks. In Proceedings of the 8th International Workshop on Distributed Algorithms (WDAG), volume 857 of LNCS, pages 209–222. Springer-Verlag, 1994. 8. R. Cohen and A. Segall. Connection management and rerouting in ATM networks. In Proceedings of INFOCOM, pages 184–191, 1994. 9. T. Eilam, M. Flammini, and S. Zaks. A complete characterization of the path layout construction problem for ATM networks with given hop count and load. In Proceedings of the 24th Int. Colloquium on Automata, Languages and Programming (ICALP), volume 1256 of LNCS, pages 527–537. Springer-Verlag, 1997. 10. O. Gerstel, I. Cidon, and S. Zaks. The layout of virtual paths in ATM networks. IEEE/ACM Transactions on Networking, 4(6):873–884, 1996. 11. O. Gerstel, A. Wool, and S. Zaks. Optimal layouts on a chain ATM network. In Proceedings of the 3th Annual European Symposium on Algorithms (ESA), volume 979 of LNCS, pages 508–522. Springer-Verlag, 1995. 12. O. Gerstel and S. Zaks. The virtual path layout problem in fast networks. In Proceedings of the 13th ACM Symposium on Principles of Distributed Computing (PODC), pages 235–243, 1994. 13. Jerry D. Gibson. The Mobile Communications Handbook, Second Edition. CRC Press in cooperation with IEEE Press, 1999. 14. R. H¨ andler and M.N. Huber. Integrated Broadband Networks: an introduction to ATM-based networks. Addison-Wesley, 1991. 15. ITU recommendation. I series (B-ISDN), Blue Book, November 1990. 16. E. Kranakis, D. Krizanc, and A. Pelc. Hop-congestion tradeoffs in ATM networks. In Proceedings of the 9th IEEE International Parallel and Distributed Processing Symposium (IPDPS), pages 662–668, 1995. 17. G. Parry. Wireless ATM MAC protocols - a literature survey. In WARP Project URL http://vera.ee.und.ac.za/coe/warp, 1999. 18. C. Partridge. Gigabit Networking. Addison Wesley, 1994. 19. D. Sobirk and J. M. Karlsson. A survey of wireless ATM MAC protocols. In Proceedings of the International Conference on the Performance and Management of Complex Communication Networks (PMCCN). Chapman & Hall, 1997. 20. L. Stacho and I. Vrto. Virtual path layouts for some bounded degree networks. In Proceedings of the 3rd Colloquium on Structural Information and Communication Complexity (SIROCCO), pages 269–278. Carleton University Press, 1996. 21. S. Zaks. Path layouts in ATM networks. In Proceedings of SOFSEM, volume 1338 of LNCS, pages 144–160. Springer-Verlag, 1997.
Modeling Context-Aware Behavior by Interpreted ECA Rules 1
Wolfgang Beer1, Volker Christian , Alois Ferscha1, and Lars Mehrmann2 1
Johannes Kepler University Linz, Department for Practical Informatics, Altenbergerstrasse 69, 4040 Linz Austria
[email protected], {voc, ferscha}@soft.uni-linz.ac.at http://www.soft.uni-linz.ac.at 2 Siemens AG, CT SE2, Otto-Hahn-Ring 6, 81730 Munich, Germany
[email protected] Abstract. The software architecture of distributed systems is about to change due to new requirements of modern mobile devices. New network techniques, like ad-hoc radio communication or peer-to-peer networks allow mobile devices to sense their environment and to interact with other devices dynamically. This paper presents a cutting-edge way to describe objects and their interaction, also called context, as well as the possibility to configure such interaction scenarios. A lookup mechanism collects information about the environment and a rolebased classification is responsible for identifying possible interaction partners. Furthermore the configuration of scenario behavior with context rules is introduced. Finally a comparison with already existing context frameworks is given and a practical emergency scenario configuration is shown.
1 Introduction More and more radio based wireless networks seem to become the standard for mobile device interaction. For many tasks it is not necessary to be a participant of a global area network, but to connect to an ad hoc network between several mobile devices. Even personal area networks (PANs) get more and more popular, e. g. in an ad hoc communication between a cellular phone, a handheld organizer and a headset. Depending on fixed networks, traditional software architecture for distributed applications contains aspects that harden its use in ad hoc communication environments. Client server architecture depends on a reliable central server that offers services. In ad hoc communication the network structure changes rapidly and normally it is not possible to set up a globally accessible server. Therefore a mobile device has to act as a server and as a client at the same time, to provide services like voice phone, or to use services like calendar function or headset output. The first question that comes to our mind, when using ad hoc communication, is how to find other entities and how to describe their functionality. Moreover, it is necessary to know how to describe the kind of interaction, which is possible with this new entity H. Kosch, L. Böszörményi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 1064–1073, 2003. © Springer-Verlag Berlin Heidelberg 2003
Modeling Context-Aware Behavior by Interpreted ECA Rules
1065
that appeared. A set of devices is possibly able to solve higher-level problems that the single device is not able to fulfill. Therefore, the single device has to be able to get context information about its environment, i.e. to find compatible partners, in order to solve joint problems. Embedded or personal mobile devices have to exchange sensor and actuator information, to model a digital representation of the real world. With this representation, digital devices should get sensitivity for the context of a human user, e. g. the location or the environment noise and the problems the user would like to solve, e. g. to switch to a louder sound profile. The digital devices could analyze the historical development of this context representation, to propose solutions for future problems. So, embedded wireless devices offer the chance to improve the kind of interaction between human users and their digital helpers [10][ 1].
2 Description of Digital and Non-digital Objects In traditional applications the programmer is responsible for connecting client and server programs, to solve a certain kind of problem. The programmer is aware of the interfaces, the network addresses, the functionality and maybe the reliability of the different participants. In modern peer-to-peer, ad-hoc communication environments the programmer does not automatically get this information. If a device appears in another communication range, it is totally unknown, which device it is and what services it offers, or demands. Therefore it is necessary to get a semantic description of the unknown objects attributes and services. Various research projects deal with the problem of semantic information modeling and distribution, often with the goal to extend the possibilities of smart software agents and reasoning mechanisms. Originally, the XML based W3C standard RDF (Resource Description Framework) [8] and its schema language were developed to meet the requirements of semantic information retrieval on the web. Various research projects deal also with the topic of joint terminology, like the definition of ontology [11]. Generally, we distinguish between two ways of describing and classifying objects:
Closed world assumption: Data and services are completely known at design time. Therefore, it is easy to define a suitable semantic model, in order to handle this information. It is not possible to add any unknown element in a closed world assumption. Moreover, parts of information are explicitly excluded from the context world model, in order to simplify reasoning [13]. Open world assumption: Data and services are not known at design time. Therefore, it is necessary to derive all the information from the semantic description of the participating agents. This issue is hard to realize and a major problem in AI research [9].
Another important aspect, when we are speaking about the description of objects, is the classification of objects. There are two different methods of classifying objects:
1066
W. Beer et al.
Static classification: The class hierarchy is completely known at design time. Once an object appears, its type either is part of the known class hierarchy, or its type is unknown. The class hierarchy cannot change at runtime. Role based classification: An unknown object is classified by a library of attribute information. The object’s set of attributes is compared with attribute sets that are known. This library of attributes and known attribute sets (we call them attribute templates) can be extended at runtime. Naturally an object is able to fulfill more than one attribute template, so it acts in more than one role. Objects can extend their functionality at runtime, i.e. their attribute template is able to change at dynamically.
To support role-based classification with an open world assumption, our framework is based on the work of XEROX Parc in 1995 [9]. Every object, called entity, is a container for a set of attributes. This set of attributes fully describes the entity’s functionality. A specific set of attributes is called an attribute template. It is responsible for classifying an entity at runtime. An entity is likely to act in more than one role. As it is shown in Figure 1, an entity, which owns the attributes {Name, Birthday, SocNumber} would be classified as a person in the European union and in the United States.
Fig. 1.
Entity classification with attribute templates
Without the social security number, the entity would be classified as a person in Europe but not in the United States. So the role based classification of entities is based on the set of attribute templates that are registered locally on the mobile device. The classification of objects is not a task of the context middleware but of the applications that run on it. Imagine a chat application that lists all persons in the local environment. Therefore, the chat application itself has to provide an attribute template that defines how the chat application specifies a person object.
Modeling Context-Aware Behavior by Interpreted ECA Rules
1067
3 Framework Architecture Overview As already mentioned in section 2, an entity is classified by the set of attributes it provides at runtime. The set of attributes possibly changes at runtime, when an entity loads or unloads attributes over the network or from a local storage. Therefore an entity has to reference a class loader and a transport layer, to load and to deploy new attributes. The class loader itself retrieves class information, either from a local attribute repository or through the transport layer. Because of the recursive nature of the attribute-entity architecture, it is possible for an entity to react as an attribute within another entity, as it is shown in Figure 2.b.).
Fig. 2. a.) Shows dynamic attribute loading at run time, while b.) shows a simple recursive attribute-entity architecture
Between every context information flow, including the loading or unloading of attributes, the interpreter checks whether any additional actions should be performed. These additional actions are specified in context rules. ECA (Event Condition Action) rules are able to react on certain entity states, where the event specifies the context event an attribute throws [3]. Context rules are described more detailed in Section 4. Figure 2.a.) shows how an entity is able to load an attribute, called Heart Monitor, and how the entity deploys it to another entity over a transport layer. The entities contain a set of attributes, which may be updated at runtime. In this upgrading process special attributes are loaded, which are necessary to fulfill a certain context scenario, see Section 6. The transport layer does not depend on any specific protocol, it can be changed at runtime. At the moment a TCP socket implementation is available as well as a SOAP-HTTP implementation. We plan to implement an additional JXTA transport layer, as far as JXTA provides reasonable performance on mobile devices.
1068
W. Beer et al.
3.1 XML Configuration of Entities Entities exist in containers, which manage their lookup mechanism, transport and lifecycle. A container has the possibility to host a performance-optimal collection of entities. A desktop computer would probably host a larger set of entities, while a mobile device is limited to a small set. Containers can exchange entities at runtime in order to optimize performance or to minimize network load. To minimize network load, it is possible to host a personal entity on a desktop computer when the user is in the office. When the user leaves his office, his entity description travels with him, hosted on his PDA. At the container startup time, a set of entities is loaded by the Dynamic Loader attribute of the container. XML configuration files inform the container, which initial set of entities should be loaded and which initial sets of attributes the entities own. The attribute-specific configuration is also located in separate XML configuration files, which were referenced by the entity configurations. The following figure shows how a simple container configuration is modeled in XML on a mobile device.
Fig. 3.
XML configuration of containers and entities
3.2 Lookup Communication in ad-hoc mode requires a mechanism that discovers entities, which are inside the communication range of an entity. In order to find a service provider in an ad-hoc network it is necessary to announce the presence of mobile devices in the environment. For a small number of service providers, it is possible to use Broadcast or Multicast IP solutions to announce their presence in the network. Broadcasting of service provider access information is not scalable and therefore not directly used for larger networks. Discovery protocols like SLP (Service Location Protocol) and the discovery mechanism of JINI (Java Intelligent Network Infrastructure) introduce hybrid solutions, where Multicast IP is used for searching in local networks and fixed addresses are used for global service discovery [12].
Modeling Context-Aware Behavior by Interpreted ECA Rules
1069
The lookup mechanism implemented in context framework also uses a hybrid solution where fixed service provider addresses are used for global service discovery. Alternatively, it is possible to use Broadcast or Multicast IP service provider announcement for local area discovery. A context container is responsible for announcing and receiving the presence of entities and their services. Additionally, the context lookup informs about the set of attributes of an entity, which enables the role-based classification. A difference to JINI is, that our solution runs on Java Personal edition version 1.1.8 and is lightweight enough to run on mobile devices. Another difference is, that our lookup solution sends XML based lookup information. The use of XML is a performance drawback, but one of the major disadvantages of JINI is, that only Java based service providers are supported, through the use of Java Object Serialization.
4 Dynamic Interaction of Entities through Context Rules In any context scenario, it is important that the entities can update or change their relations at runtime. As it is described in Section 2, an entity is able to dynamically classify unknown objects that appear in its environment with attribute templates. In this connection we defined a context rule: “A context rule specifies the reaction of an entity on a specified context state and defines therefore the event based interaction between entities” An entity is able to sense its environment with sensors and to change the information in the scenarios world model, as shown in Figure 3.a.). The sensing mechanism could be either event-triggered or time-triggered, as it was already observed in [2]. The framework has to transform the lower level sensor input into information that fits in the framework’s context world model. In the context framework the set of entity attributes contain the high-level context information about an object. So the values of these attributes provide an entity state. A context event is triggered by an attribute, if its state changes and it would like to inform its entity. To react on a specific entity state, it is necessary to define a context rule inside the entity. Context rules define an ECA (Event-Condition-Action) matching service [3], as it is already known from active database research [7]. 4.1 Context Rule Syntax We defined a configuration syntax, which is easy to understand for humans and more compact than XML-coded ECA definitions: Rules = “rules” [ Targets ] “{“ { Rule } “}”. Targets = “for” EntityOrTemplate { “,” EntityOrTemplate }. Rule = “on” Event [ “if” Condition ] Action
The non terminal symbol EntityOrTemplate describes an entity or a template which defines a set of similar entities as target for the rule. Thus it is possible to bind the rule to a specific entity or to a specific role, specified with a template name. Because of
1070
W. Beer et al.
very limited space, the non terminal symbols Event, Condition and Action are not shown in EBNF here. The following example shows a set of rules, that is bound to all entities, which act in the role of a Person: rules for { ... }
Fig. 4. a.) Shows the context information transformation and event triggering, while b.) shows the lookup and interface WSDL description delivery.
An entity owns a rule interpreter, which is responsible for catching specific events and reacting on them. Also the attribute loading events or the lookup events can be caught by the rule interpreter. The context rule interpreter is able to load an initial set of rules from a file, referenced by the entity’s XML configuration. After the container startup, the entity is able to receive new context rules over the transport layer. These new context rules are registered with the interpreter. Considering the easy plain-text syntax of our context rules, it is also possible to register new context rules from mobile devices like cellular phones or PDAs. Distributed rule deployment is a convenient feature to develop a context scenario on a laptop computer and deploy it to different mobile devices. Attribute Interface Description with WSDL Distributed scenario development and deployment is also supported by the interface description of attributes with WSDL (Web Service Definition Language). As the lookup service collects information, which set of attributes an unknown entity provides, the entity is also able to deliver WSDL information for its attributes, as it is
Modeling Context-Aware Behavior by Interpreted ECA Rules
1071
shown in Figure 3.b.). With these types of information it is easy to generate context rules automatically or to use a visual builder tool to generate context behavior. Rule Consistency One major problem of the context rule management and deployment is to maintain the consistence of rules. This problem statement is already solved by expert systems, or generally by artificial intelligence research. In fact expert system shells like CLIPS or Jess [5] offer the possibility to reason about entities and their relations. For this purpose it is planned to integrate the Jess library into the context framework, in order to manage entity relationship reasoning. At the moment the interpreter itself triggers actions according to incoming events, but it is not able to check the consistence of new rules. Advantages of Interpreted Context ECA Rules One of the most important advantages of controlling complex interaction scenarios with interpreted ECA rules is the high degree of flexibility to change the interaction topology at runtime. Therefore, it is possible to change the interaction partners and the interaction itself at runtime. With the use of attribute templates the rules for handling specific events can be routed to a group of entities that fulfill a specific role. The current implementation of our ECA rule syntax enables the dynamic delivery and lifecycle management of rules at runtime. The rules are coded with the use of a proprietary rule syntax, which enables short and concise creation of new rules, also on mobile devices. Another possible module would be a visual builder tool, to support the creation of context rules, or even the creation of whole interaction scenarios without programming. These aspects, of context modeling through ECA rules, allow nontechnical people to control smart environments and devices without knowledge of a complex system, which a smart environment definitely is. As an example for an abstract view on complex scenario modeling, the Washington’s Department of Bioengineering developed a tool called Labscape. With Labscape it is possible to model complex scenarios in a laboratory [6].
5 Example Context Scenario To demonstrate the abilities of the context middleware, a complex emergency scenario simulation was built (aware of the fact that real-time applications normally are not implemented in Java). For an compact, out of the box demonstrator we simulated indoor location sensors with RFID transponders and active readers (in a real world outdoor scenario we would use differential GPS and GPRS as transmission protocol). The scenario starts with an automatic emergency call, triggered by a heart patients PDA-HeartMonitor device: rules for { on HeartMonitor.Alarm() { <EmergencyDispatcher>.EmergencyCall:Alarm($Patient); } }
1072
W. Beer et al.
The lookup identifies all entities, acting in the role of an EmergencyDispatcher, to which the emergency call is delivered to. When the emergency dispatcher receives an emergency call a rule catches this call and delivers some reaction possibilities to the human emergency dispatcher: rules for <EmergencyDispatcher> { on EmergencyCall.Alarm(Patient p) { Map.ShowPatient(p); Map.ShowNearestAmbulances(p.location); Map.ShowNearestHospital(p.location); } }
When the human emergency dispatcher reacts on the emergency call, the chosen ambulance car is informed: rules for <EmergencyDispatcher> { on EmergencyCall.SendAmbulance(Ambulance a, Patient p, Hospital h) { a.Alarm:SendTo(p, h); } }
The three context rules, that are shown above, are a small and primitive subset of the mass of context rules that form the complete emergency scenario. They should just explain the purpose of such an abstraction layer.
Fig. 5.
RFID reader with PDA device and a model emergency car with sensor equipment.
5.1 Emergency Scenario Hardware Setup The hardware setup is based on three Siemens Loox devices for hosting the patient, the mobile doctors and the ambulance car entities. In order to receive sensor information about the proximity location of entities RFID (Radio Frequency Identification)
Modeling Context-Aware Behavior by Interpreted ECA Rules
1073
readers were used. For network ad-hoc communication purposes, wireless IEEE802.11b compact flash cards are used. The opened device is shown in Figure 4. To get proximity location information, the presentation floor material is tagged with RFID transponders. The devices are now able to find themselves with the lookup mechanism and to communicate over the context containers transport layer. The emergency scenario is modeled with ECA context rules, in order to define the behavior and the interaction between the emergency entities.
6
Conclusion
The goal of this work was, to show how complex ad-hoc scenarios could be modeled by defining ECA rules for each entity, which is relevant for the scenario. A very flexible and stable context middleware software framework was implemented and tested within an example scenario. Many different simulated entities (emergency cars 1-10, mobile doctors 1-5, 1-10 ambulance drivers and nurses) were hosted in mobile context containers, to show the flexibility of our context middleware. The scenario is not bound to any fixed behavior, but could be changed at runtime through distributed ECA rule deployment. Major future issues will be the visual composition of ECA context rules and a lightweight security model to restrict access to the context model.
References 1. 2. 3.
4.
5. 6. 7. 8. 9. 10. 11. 12. 13.
A. K. DEY, G. D. ABOWD: Toward a Better Understanding of Context and ContextAwareness. GIT, GVU Technical Report GIT-GVU-99-22, June 1999. A. FERSCHA, S. VOGL, W. BEER: Ubiquitous context sensing in wireless environments, 4th DAPSYS, Kluwer Academic Publishers 2002 Diego López de Ipiña: An ECA Rule-Matching Service for Simpler Development of Reactive Applications, Published as a supplement to the Proceedings of Middleware 2001 at IEEE Distributed Systems Online, Vol. 2, No. 7, November 2001 J. PASCOE, N. Ryan, D. Morse: Issues in developing context-aware computing. Handheld and Ubiquitous Computing, Nr. 1707 in Lecture Notes in Computer Science, pages 208– 221, Heidelberg, Germany, September 1999. Springer-Verlag JESS (Java Expert System Shell), http://herzberg.ca.sandia.gov/jess/ L. ARNSTEIN et al., „Labscape: A Smart Environment for the Cell Biology Laboratory”, IEEE Pervasive Computing, July-September 2002, pp. 13–21. PATON N.W. and Diaz O.: Active Databases Survey, ACM Computing Surveys, Vol. 31 No. 1, pp. 63–103, March 1999 RDF (Resource Description Framework), http://www.w3.org/RDF/ R. WANT, W. SCHILIT, et. al.: The PARCTAB Ubiquitous Computing Experiment, Technical Report CSL-95-1, Xerox Palo Alto Research Center, March 1995. SENTIENT COMPUTING, AT&T Laboratories, http://www.uk.research.att.com/spirit/ SEMANTIC WEB Organisation, http://www.semanticweb.org SLP (Service Location Protocol), http://www.ietf.org/rfc/rfc2608.txt T. KINDBERG, et al.: people, places, things: web presence for the real world
A Coordination Model for ad hoc Mobile Systems Marco Tulio Valente1 , Fernando Magno Pereira2 , Roberto da Silva Bigonha2 , and Mariza Andrade da Silva Bigonha2 1 2
Department of Computer Science, Catholic University of Minas Gerais, Brazil Department of Computer Science, Federal University of Minas Gerais, Brazil
[email protected], {fernandm,bigonha,mariza}@dcc.ufmg.br Abstract. The growing success of wireless ad hoc networks and portable hardware devices presents many interesting problems to system engineers. Particular, coordination is a challenging task, since ad hoc networks are characterized by very opportunistic connections and rapidly changing topologies. This paper presents a coordination model, called PeerSpaces, designed to overcome the shortcomings of traditional coordination models when used in ad hoc networks.
1
Introduction
With the advent of ad hoc networks, mobile devices can detach completely from the fixed infrastructure and establish transient and opportunistic connections with other devices that are in communication range. Designing applications on these dynamic and fluid networks presents many interesting problems. Particularly, coordination is a challenging task. Since a user may find itself in a different network at any moment, the services available to him change along the time. Thus, computation should not rely on any predefined and well known context. Specifically when operating in ad hoc mode, coordination should not assume the existence of any central authority, since the permanent availability of this node can not be granted. Communication should also be uncoupled in time and space, meaning that two communicating entities do not need to establish a direct connection to exchange data nor must know the identity of each other. Recently, shared space coordination models, inspired by Linda [4], are being considered for communication, synchronization and service lookup in mobile computing systems. The generative communication paradigm introduced by Linda is based on the abstraction of a tuple space. Processes communicate by inserting, reading and removing ordered sequences of data from this space. In traditional Linda systems, like TSpaces [8] and JavaSpaces [3], the tuple space is a centralized and global data structure that runs in a pre-defined service provider. In the base station scenario this server can easily be located in the fixed network. However, if operation in ad hoc mode is a requirement, the fixed infrastructure simply does not exist. This suggests that standard client/server implementations of Linda are not suitable to ad hoc scenarios, since they assume a tight coupling between client and servers and the permanent availability of the latter. H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 1074–1081, 2003. c Springer-Verlag Berlin Heidelberg 2003
A Coordination Model for ad hoc Mobile Systems
1075
This paper formalizes our attempts to customize and adapt shared space coordination models to applications involving mobile devices with ad hoc network capabilities. The model formalized in the paper, called PeerSpaces, has primitives for local and remote communication, process mobility and service lookup. In order to answer the new requirements posed by ad hoc mobile computing systems, PeerSpaces departs from traditional client/server architectures and push towards a completely decentralized one. In the model, each node (or peer) has the same capabilities, acting as client, shared space provider and as router of messages. In order to provide support to operation in ad hoc mode, service lookup is distributed along the network and does not require any previous knowledge about its topology. The paper is organized as follows. In Section 2 we informally present the PeerSpaces model, including its main design goals, concepts and primitives. In Section 3 we give the formal semantics of the model in terms of a small language derived from the π-calculus. Besides a precise specification of the model, the semantics presented in this section supports formal reasoning about applications built on PeerSpaces. Section 4 compares the model with similar efforts. Finally, Section 5 concludes the paper.
2
The PeerSpaces Model
The main concepts used in PeerSpaces are the following: Hosts. The model assumes that hosts are mobile devices. Each host has its own local tuple space and a running process. A host is written hg [ P, T ], where h is the name of the host, P is the process running in the host, T is its local tuple space, and g is the group of the host. In PeerSpaces, the name of a host is different from the name of all other hosts. The model also assumes a infinite set H of possible host names. Groups. Hosts in the model are logically organized in groups. Each group has a name and can also contain subgroups, creating a tree structure. The group of a host is denoted by a tuple g1 , . . . , gn , that specifies the path from the root group g1 to the leaf group gn where the host is located. For example, the tuple pucminas, cs, proglab denotes the set of hosts in the proglab group, which is a subgroup of the group cs, which is nested in the root group pucminas. Two groups can have the same name, as long they are not subgroups of the same group. Network. Mobile hosts in the model are connected by a wireless and ad hoc network. As usual in such networks, connectivity is transient and determined by the distance among hosts. Consequently, the topology of the network is continuously changing. In PeerSpaces, a network with hosts h1 , h2 , . . . , hn is denoted by: h1g1 [ P1 , T1 ] | h2g2 [ P2 , T2 ] | . . . | hngn [ Pn , Tn ] , E
1076
M.T. Valente et al.
where g1 , g2 , . . . , gn are the group of the hosts and E : H × H is a relation representing connections among hosts. The presence of a pair (hi , hj ) in E, denoted by hi 1 hj , indicates that host hi is in communication range with host hj . This relation is in continuous change to reflect reconfigurations in the network. PeerSpaces also defines a set of primitives to assemble applications using the previous defined concepts. We spend the rest of this section describing such primitives. Local Primitives. The local tuple space of any host is accessed using the traditional in , rd and out primitives from Linda. Furthermore, there is a chgrp g primitive, used to change the group of the current host to the one specified by tuple g. Process Mobility. Processes in PeerSpaces are mobile in order to model the behavior of mobile agents. A mobile agent is a process that can move among sites carrying computation and accessing resources locally. In wireless environments, agents are a powerful design tool to overcome latency and to embed autonomous computations. In the model, the primitive move h.P is used to move a process to node h, where its execution continues as P . If host h is not connected, the operation blocks until a connection to such host is established. Remote Primitives. Crucial to the scalability and efficiency of any coordination model for mobile computing systems is the design of the remote operations. Thus, from the beginning PeerSpaces departs from the idea of providing seamless access to a global and centralized space. Instead, there are primitives that operate in the remote space of a well-known host h: out h, v; in h, p, x and rd h, p, x, where v is a tuple, p is a pattern and x is a variable. These operations are merely remote implementations of the traditional Linda primitives and thus do not impact in the overall performance of the system. As its local version, the remote out h, v primitive is asynchronous. The primitive is used when a process wants to leave a information to be consumed later in another host. In order to model its asynchronous behavior, the operation is executed in two steps. In the first step, a tag is added to the tuple v to indicate that it should be transfer as soon as possible to the destination host h. The tagged tuple, denoted by vh , is then outputted in the local space of the host h that requested the operation. In the second step, tuple vh is transferred to the space of host h as soon as it is connected to h and the tag is removed from the tuple. Since both steps are not atomic, while the tuple is “misplaced” in the source node it can be retrieved by an operation like in vh . For example, this operation can be called by a garbage collector process in charge of reclaim tuples that are waiting for the connection of their destination host for a long time. Lookup Primitive. Without a lookup primitive the remote operations described above have little use, since a mobile host may not know in advance the
A Coordination Model for ad hoc Mobile Systems
1077
name h of a service provider in its current network. Moreover, since the system is designed to support operation in ad hoc mode, the lookup service must not be centralized in a single host, but must be distributed along the federation of connected devices. In order to accomplish such requirements, there is in PeerSpaces the following primitive: find g, p. This primitive queries hosts in group g for tuples matching pattern p in a distributed way. All matching tuples found in group g are copied asynchronously to the local space of the host that has called the operation. The semantics of PeerSpaces does not assume any specific routing protocol for propagation of lookup queries. Continuous Queries. Often it is useful to query a group of hosts for a resource and keep the query effective until such resource is available. In this way, a client does not need to periodically send lookup queries to detect new resources that may become available since the last query was issued. In PeerSpaces, lookup queries that remain active after their first execution are called continuous queries. Continuous queries in PeerSpaces have a lifetime parameter, used to automatically garbage collect the query after its expiration. Continuous lookup queries are issued adding the lifetime t to the find primitive: find g, p, t. This primitive will search the hosts of group g for all currently available resources matching pattern p and for resources that may become available in t units of time after the query was issued.
3
Formal Semantics
The ultimate goal of our research is to deploy a coordination middleware for ad hoc mobile computing systems. In order to achieve this goal we have initially defined the formal semantics of PeerSpaces. The formalization presented next uses an operational semantics based on the asynchronous π-calculus [6]. The π-calculus is a good basis as it provides a small, elegant and expressive concurrent programming language. The main departure from π in our semantics is the use of generative communication instead of channel-based communication. Table 1 summarizes the syntax of our core language. We assume an infinite set H of names, used to name hosts and lookup queries. Meta-variables h and x range over H. Basic values, ranged over by v and g, consist of names and tuples. Tuples are ordered sequences of values v1 , . . . , vn . A tuple space T is a multiset of tuples. We use the symbol ? ∈ H to denote the distinguished unspecified value. A program is composed by the network N, the relation E and a global set of names X. The relation E : H × H represents the connectivity map of the network. The names used over several hosts in the system are recorded in the set X, ensuring their unicity. Each host h is member of a group g and has a running process P and a local tuple space T . Processes are ranged by P and Q. Similar to the π-calculus, the simplest term of our language is the inert process 0, which denotes a process with no behavior at all. The term P | Q denotes two processes
1078
M.T. Valente et al. Table 1. Syntax
P rog ::= N , E , X N ::= ε | H | N H ::= hg [ P, T ] P ::= 0 | P | Q | ! P | (ν x) P | out v | in v, x.P | rd v, x.P | find g, p, t | chgrp g | move h.P
running in parallel. The term ! P denotes a infinite number of copies of P , all running in parallel. The restriction operator (ν x) P ensures that x is a fresh and unguessable name in the scope of P. Similar to Linda, the primitive operations out, in and rd provide access to the local tuple space. Since the out operation is asynchronous it does not have a continuation P . The same happens to the find and chgrp primitives. We assume that non-continuous lookup queries can be simulated by defining the lifetime equal to zero. Finally, the move operation simulates the behavior of single thread mobile agents. The operational semantics of our calculus is summarized in Tables 2 and 3. The semantics is defined in terms of a reduction relation →, a structure congruence ≡ between processes and a set of pattern matching rules. Table 2 summarizes the core language semantics, which is basically Linda with multiple tuple spaces. A reduction N , E , X → N , E , X defines how the configuration N , E , X reduces in a single step computation to N , E , X . Initially, there are three reduction rules describing the effects on the configuration of each standard Linda primitive. The output operation, out v, asynchronously deposits a tuple in the local space (rule L1). The input, in v, x.P , and read, rd v, x.P , operations try to locate a tuple v that matches v (rules L2 and L3). If one is found, free occurrences of x are substituted for v in P , denoted as P {v /x}. In the case of the input, the tuple is removed from the space. The next set of rules defines a structural congruence relation ≡ between processes (SC1 to SC7) and hosts (SC8 to SC10). As in the π-calculus, such rules define how processes can be syntactically rearranged in order to allow the application of reductions. In such rules, we write f n(P ) to denote the set of free names in process P. The definition of pattern matching, written v ≤ v , allows for recursive tuple matching. Values match only if they are equal or if the unspecified value occurs on the left hand side. Table 3 extends the core language with the primitives proposed in PeerSpaces. The find g , p, t operation deposits a tuple representing a service lookup query in the local space (rule P1). Such query is a tuple in the format k, g , p, t, h, where k is a fresh name that identifies the query, g defines the group where the query will be performed, p is a pattern for the desired service, t is the lifetime and h is the name of the current host. The operation chgrp g just
A Coordination Model for ad hoc Mobile Systems
1079
Table 2. Core Language Operational Semantics
Reductions Linda Primitives hg [ out v | P, T ] | N , E , X → hg [ P, v ∪ T ] | N , E , X
(L1)
hg [ in v, x.P | Q, v ∪ T ] | N , E , X → hg [ P {v /x} | Q, T ] | N , E , X
(L2)
hg [ rd v, x.P | Q, v ∪ T ] | N , E , X → hg [ P {v /x} | Q, v ∪ T ] | N , E , X
(L3)
The rules are subjected to the following side conditions: (L2) if v ≤ v (L3) if v ≤ v Structural Congruence Rules P |Q≡Q|P (SC1) (ν x) (ν y) P ≡ (ν y) (ν x) P (SC5) !P ≡ P | !P (SC2) P ≡ Q ⇒ (ν x) P ≡ (ν x) Q (SC6) (P | Q) | R ≡ P | (Q | R) (SC3) (ν x) (P | Q) ≡ P | (ν x) Q (SC7) P |0≡P (SC4) P ≡ Q ⇒ hg [ P, T ] ≡ hg [ Q, T ] (SC8) hg [ (ν x) P, T ] | N , E , X ≡ hg [ P, T ] | N , E , x ∪ X (SC9) (SC10) hg [ P, T ] | N , E , X ≡ N | hg [ P, T ] , E , X The rules are subjected to the following side conditions: (SC7) if x ∈ / f n(P ) (SC9) if x = h, x ∈ / f n(N ), x ∈ /X Pattern Matching Rules v≤v
?≤ v
v1 ≤ v1 . . . vn ≤ vn v1 . . . vn ≤ v1 . . . vn
changes the group of the current host to the one specified by tuple g (rule P2). If such group does not exist, it is created. The move h .P operation changes the location of the continuation process P to host h if this host is connected (rule P3). Otherwise, the operation remains blocked until the engagement of h . Reduction rule Q1 defines how lookup queries are propagated in the network. Basically, any host that holds a query k, g , p, t, h can propagate it to a connected host h in group g , if g matches g and the query is not yet present in h . If such conditions are satisfied, the query is inserted in the local space of h and a process P is added in parallel with the other processes running in this host. This process continuously read tuples matching the pattern p and then use a remote output operation to send the results to the local space of the host h that has issued the query.
1080
M.T. Valente et al. Table 3. PeerSpaces Operational Semantics
Reductions PeerSpaces Primitives hg [ find g , p, t | P, T ] | N , E , X → hg [ (ν k) P, k, g , p, t, h ∪ T ] | N , E , X (P1) hg [ chgrp g | P, T ] | N , E , X → hg [ P, T ] | N , E , X
(P2)
hg [ move h .P | Q, T ] | h g [ P , T ] | N , E , X → hg [ Q, T ] | h g [ P | P , T ] | N , E , X
(P3)
Query Propagation hg [ P, k, g , p, t, h ∪ T ] | h g [ P , T ] | N , E , X → hg [ P, k, g , p, t, h ∪ T ] | h g [ P | P , k, g , p, t, h ∪ T ] | N , E , X
(Q1)
Network Reconfiguration E ⇒ E N , E , X → N , E , X
(N1)
The rules are subjected to the following side conditions: (P3) if h 1 h (Q1) if (h 1 h ) ∧ (g g ) ∧ (k, g , p, t, h ∈ T ) ∧ P = ! (rd p, x.out h, x) Group Matching Rule g1 = g1 . . . gn = gn g1 . . . gn g1 . . . gn . . . gm
The last reduction rule introduces a new type of reduction ⇒ used to describe reconfigurations in the network and consequently in the connectivity relation E. Basically, this rule dictates that changes in E should be propagated to the current configuration. However, we left ⇒ reductions unspecified in the semantics, since they are dependent on the physical location of each host and on technological parameters of the subjacent network. There is also a special pattern matching rule for group names. Two groups g and g matches, written g g if all subgroups in g are equal to equivalent subgroups in g , which can also have extra nested subgroups.
4
Related Work
Many characteristics of PeerSpaces have been inspired in file sharing applications popular in the Internet. Particularly, the peer to peer network created by Gnutella [5] over the fixed Internet presents many properties that are interesting in mobile settings, like absence of centralized control, self-organization and adaptation to failures. PeerSpaces is an effort to transport and adapt such
A Coordination Model for ad hoc Mobile Systems
1081
characteristics to mobile computing systems. This explains the choice of Linda shared spaces as the prime coordination infrastructure for PeerSpaces. Lime [7] introduces the notion of transiently shared data space to Linda. In the model, each mobile host has its own tuple space. The contents of the local spaces of connected hosts are transparently merged by the middleware creating the illusion of a global and virtual data space. Applications in Lime perceive the effects of mobility by atomic changes in the contents of this virtual space. The scalability and performance weakness of Lime have motivated the proposal of CoreLime [2], where in name of simplicity and scalability the idea of transiently shared spaces is restricted to the set of mobile agents running in a host. Another work proposing an alternative semantics to the notion of transiently shared spaces is [1].
5
Conclusions
In this paper we have presented and formalized PeerSpaces, a coordination model for mobile computing systems. The model was designed to overcome the main shortcoming of shared spaces coordination models when used in ad hoc wireless networks – the strict reliance on the traditional client/server architecture – while preserving the main strengths of such models – the asynchronous and uncoupled style of communication. PeerSpaces can be used as the building block of ad hoc mobile systems like file sharing, groupware, mobile commerce and message systems. Acknowledgments. We would like to thank Jan Vitek and Bogdan Carbunar for the discussions that led to this paper.
References 1. N. Busi and G. Zavattaro. Some thoughts on transiently shared tuple spaces. In Workshop on Software Engineering and Mobility, May 2001. 2. B. Carbunar, M. T. Valente, and J. Vitek. Corelime a coordination model for mobile agents. In International Workshop on Concurrency and Coordination, volume 54 of Electronic Notes on Theoretical Computer Science. Elsevier Science, July 2001. 3. E. Freeman, S. Hupfer, and K. Arnold. JavaSpaces Principles, Patterns, and Practice. Addison-Wesley, 1999. 4. D. Gelernter. Generative communication in Linda. ACM Transactions on Programming Languages and Systems, 7(1):80–112, Jan. 1985. 5. Gnutella Home Page. http://gnutella.wego.com. 6. R. Milner. Communicating and Mobile Systems: the Pi-Calculus. Cambridge University Press, 1999. 7. A. L. Murphy, G. P. Picco, and G.-C. Roman. Lime: A middleware for physical and logical mobility. In Proceedings of the 21st International Conference on Distributed Computing Systems, May 2001. 8. P. Wycko, S. W. McLaughry, T. J. Lehman, and D. A. Ford. TSpaces. IBM Systems Journal, 37(3):454–474, Aug. 1998.
Making Existing Interactive Applications Context-Aware Tatsuo Nakajima, Atsushi Hasegawa, Tomoyoshi Akutagawa, Akihiro Ibe, and Kouji Yamamoto Department of and Computer Science, Waseda University 3-4-1 Okubo Shinjuku Tokyo 169-8555 JAPAN
[email protected] Abstract. We propose a new approach to build context-aware interactive applications. Our approach enables us to use existing GUI-based interactive applications although a variety of interaction devices can be used to control them. This means that we can adopt traditional GUI toolkits to build context-aware applications. The paper describes design and implementation of our middleware that enables existing applications to be context-aware, and presents some examples to show the effectiveness of our approach.
1
Introduction
Context-awareness[4,11] will reduce complexities to access our environments since the environments recognize our situation to fill many parameters that are chosen by us in traditional approaches. The parameters are retrieved by various sensors embedded in our environments. However, implementing context-aware applications is hard for usual programmers. Also, we already have many existing GUI-based interactive applications. In the future, these applications will like to be modified to support context-awareness, but the modification is not easy. In this paper, we propose a new approach to build context-aware applications. Our middleware enables the GUI-based interactive applications to be contextaware without modifying them. Also, programmers can adopt traditional GUI toolkits such as GTK+ and Java Swing to build context-aware applications. Our current work focuses on home computing environments where home appliances such as TV and VCR are connected by a home network. Most of current home appliances do not take into account context-awareness. Also, standard middleware for home computing has adopted traditional GUI toolkits such as Java AWT. Therefore, our middleware is attractive to build context-aware home computing applications in an easy way.
2
Ubiquitous Computing Applications
In ubiquitous computing environments, one of the most important problems is how to interact with a variety of objects embedding computers. The interaction H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 1082–1090, 2003. c Springer-Verlag Berlin Heidelberg 2003
Making Existing Interactive Applications Context-Aware
1083
between us and computers embedded in various objects has been studied. However, it is necessary for us to consider how to manage these interaction devices. Let us consider that a user uses a mobile device to control home appliances or to retrieve interesting information. In this case, applications running on the device need to take into account the mobility of devices. Also, we like to use various display devices to show control panels of home appliances. For example, when a user sits on a sofa, he will use his PDA to control the appliances. The control panel is displayed on the PDA, and he navigates the panel to control appliances. However, if he can find a large display near him, he likes to use it to control the appliances. The control panel should be displayed on the large display, and the panel can be navigated from his PDA. Moreover, if the display is a touch panel, he will navigate the panel by touching it. We believe that it is important to use a variety of interaction devices according to his situation, and one interaction device is not enough to cover various situations. In this case, a user interface middleware to offer context-awareness is necessary to change various interaction devices according to the situations. There are many researches to develop context-aware applications[6,11]. Also, there is a generic framework to develop context-aware applications[4]. However, we believe that there are two important issues that should be considered to build context-aware applications. The first issue is how to retrieve context information. We need various sensors and common interface to provide higher level abstraction to abstract various sensor information. For example, context toolkit[4] and sentient information framework[5] provide high level abstraction to hide details of context information retrieved from the sensors. The second issue is a mechanism to adapt software structure according to retrieved context information[9]. Traditionally, most of context-aware applications change their software structures in an ad-hoc way, and it is not easy to take into account new context information. We believe that it is important to adopt a framework to build adaptive applications in a systematic way. Although there are many researches to attack the problems described above, it is not easy to develop context-aware applications. Because implementing adaptive applications is still ad-hoc, and programmers need to learn advanced programming techniques such as design patterns and aspect-oriented programming to build context-aware applications that can take into account new situations. Also, we need to reimplement existing applications if we like to modify the applications to support context-awareness. For example, if a context-aware application needs to display video streams according to the location of a user. The application should implement the redirection of the video streams to transmit them to computers that is desirable to display them[1]. We believe that we need a simpler approach to build context-aware applications. Especially, it is important to build context-aware applications from existing applications to build ubiquitous computing environments in a practical way. Our middleware based on a thin-client architecture enables us to build context-aware applications in a simple way. Existing applications are completely separated from our middleware components. Our middleware enables us to inte-
1084
T. Nakajima et al.
Input Devices keyboard mouse event User Interface Middleware bitmap image
Existing Applications or Application with standard GUI toolkits
Output Devices Context-aware device selection and bitmap image adaptation
Fig. 1. Basic Architecture
grate new interaction devices and context-awareness without modifying existing applications.
3 3.1
Design and Implementation Basic Architecture
Fig. 1 shows an overview of our middleware infrastructure. In the architecture, an application generates bitmap images containing information such as control panels, photo images and video images. The approach is simple because popular operating systems provide a mechanism to retrieve bitmap images generated by applications. Also, these applications can receive keyboard and mouse events to be controlled. The user interface middleware receives bitmap images from applications and transmits keyboard and mouse events. The role of the middleware is to select appropriate interaction devices by using context information. Also, input/output events are converted to keyboard/mouse events according to the characteristics of interaction devices. Our system uses the thin-client system to transfer bitmap images to draw graphical user interface, and to process mouse/keyboard events for inputs. The usual thin-client system such as Citrix Metaframe[3], Microsoft Terminal Server[7], Sun Microsystems Sun Ray[13], and AT&T VNC(Virtual Network Computing) system[10] consists of a viewer and a server. The server is executed on a machine where an application is running. The application implements graphical user interface by using a traditional user interface system such as the X window system. The bitmap images generated by the user interface system are transmitted to a viewer that is usually executed on another machine. On the
Making Existing Interactive Applications Context-Aware
1085
Application
Input Device
Input Management Module Inp u
Inp ut E
t Sp
Plug and Play Management Module
ec
ven ts
Universal Interaction Protocol
Output Spec Output Event s
UniInt Server
Output Device
Output Management Module
UniInt Proxy
Universal Interaction Module
Fig. 2. System Architecture
other hand, mouse and keyboard events captured by the viewer are forwarded to the server. The protocol between the viewer and the server are specified as a universal interaction protocol. The system is usually used to move a user’s desktop according to the location of a user[6], or shows multiple desktops on the same display, for instance, both MS-Windows and the X Window system. In our system, we replace the viewer of a thin-client system to the UniInt(Universal Interaction) proxy that forwards bitmap images received from a UniInt server to an output device. Also, UniInt proxy forwards input events received from an input interaction device to the UniInt server. In our approach, a server of any thin-client systems can be used as the UniInt server. Our system consists of the following four components as shown in Fig. 2. – – – –
Home Computing Application UniInt Server UniInt Proxy Input/Output Interaction Devices
Home computing applications[12] generate graphical user interface for currently available home appliances to control them. For example, if TV is currently available, the application generates user interface for the TV. On the other hand, the application generates the composed GUI for TV and VCR if both TV and VCR are currently available. The UniInt server transmits bitmap images generated by a window system using the universal interaction protocol to a UniInt proxy. Also, it forwards mouse and keyboard events received from a UniInt proxy to the window system. In our current implementation, we need not to modify existing servers of thinclient systems, and any applications running on window systems supporting a UniInt server can be controlled in our system without modifying them. The UniInt proxy is the most important component in our system. The UniInt proxy converts bitmap images received from a UniInt server according to
1086
T. Nakajima et al.
the characteristics of output devices. Also, it converts events received from input devices to mouse or keyboard events that are compliant to the universal interaction protocol. The UniInt proxy chooses a currently appropriate input and output interaction devices for controlling appliances. To convert interaction events according to the characteristics of interaction devices, the selected input device transmits an input specification, and the selected output device transmits an output specification to the UniInt proxy. These specifications contain information that allow a UniInt proxy to convert input and output events. The last component is input and output interaction devices. An input device supports the interaction with a user. The role of an input device is to deliver commands issued by a user to control home appliances. An output device has a display device to show graphical user interface to control appliances. 3.2
Implementation of UniInt Proxy
The current version of UniInt proxy is written in Java, and the implementation contains four modules as shown in Fig. 2. The first module is the universal interaction protocol module that executes the universal interaction protocol to communicate with a UniInt server. The replacement of the module enables us to use our system with different thin-client systems. The module can use the same module implemented in a viewer of a thin-client system. The second module is the plug and play management module. The module collects information about currently available interaction devices, and builds a database containing information about respective interaction devices. The third module is the input management module. The module selects a suitable input interaction device by using the database contained in the plug and play management module. The last module is an output management module. The module also selects a suitable output interaction device. Also, the module converts bitmap images received from the universal interaction module according to the output specification of the currently selected output interaction device. Management of Available Interaction Devices: The plug and play management module detects currently available input and output devices according to context information. The module implements the Universal Plug and Play Protocol to detect currently available interaction devices. In our system, we assume that all interaction devices can be connected to the Internet. An interaction device transmits advertisement messages using the simple service discovery protocol(SSDP). When a UniInt proxy detects the messages, it knows the IP address of the interaction device. Then, the UniInt proxy transmits an HTTP GET request to the interaction device. We assume that each interaction device contains a small Web server, and returns an XML document. The XML document contains information about the interaction devices. If the interaction device is an input device, the file contains various attributes about the device, which are used for the selection of the most suitable device. For an output device, the file contains information about the display size and the attributes for the device. The plug and play management module maintains a database containing all information about currently detected interaction devices.
Making Existing Interactive Applications Context-Aware
1087
Adaptation of Input and Output Events: The role of the input management module and the output management module is to determine the policies for selecting interaction devices. As described in the previous section, all information about currently available interaction devices are stored in a database of the plug and play management module. The database provides a query interface to retrieve information about interaction devices. Each entry in the database contains a pair of an IP address and a list of attributes for each interaction device, then the entry whose attributes are matched to a user’s preference provided in a query is returned. The current implementation of the input management module receives all input events from any currently available input devices. When a new input device is detected, a new thread for receiving input events from the newly detected device is created. All input events are delivered to the universal interaction module, and they are processed by applications eventually. The output management module converts bitmap images received from the universal interaction module according to the display size of an output device. The size is stored in the database of the plug and play management module. When an output device is selected, the display size is retrieved from the database. The bitmap image is converted according to the retrieved information, then it is transmitted to the selected output device. 3.3
Current Status
Our system have adopted the AT&T VNC system[10] as a thin-client system, and the VNC server can be used as the UniInt server without modifying it. The current prototype in our HAVi-based home computing system[12], where HAVi is a standard specification for digital audio and video, emulates two home appliances. The first one is a DV viewer and the second one is a digital TV emulator. Our application shows a graphical user interface according to currently available appliances as described in the previous section. Also, the cursor on a screen that displays a graphical user interface can be moved from a Compaq iPAQ. However, if the device is turned off, the cursor is controlled by other devices such as a game console. It is also possible to show a graphical user interface on the PDA device according to a user’s preference. Also, the current system has integrated cellular phones to control home appliances. NTT Docomo’s i-mode phones have Web browsers, and this makes it possible to move a cursor by clicking special links displayed on the cellular phones. In our home computing system, Linux provides an IEEE 1394 device driver and an MPEG2 decoder. Also, IBM JDK1.1.8 for the Java virtual machine is used to execute the HAVi middleware component. Fig. 3 contains several photos to demonstrate our system. Currently, our home computing applications are executed on HAVi, and a control panel is written by using Java AWT. In the demonstration, if both a DV camera and a digital TV tuner are simultaneously available, the control panels for them are combined as one control panel as shown in the photo(Top-Left). The control panel can be navigated by both a cellular phone(Top-Right) and a game
1088
T. Nakajima et al.
console(Bottom-Left). Also, the control panel can be displayed and navigated on a PDA(Bottom-Right). Our middleware proposed in this paper enables us to use various interaction devices and to interact with the appliances in a context-aware way. By integrating home appliances with our middleware, a user is allowed to choose the most suitable interaction device according to his situation.
Fig. 3. Controlling Our HAVi based Home Computing System
4
Scenarios of Our Approach
In this section, we present two scenarios to show the effectiveness of our approach. The first scenario enables us to interact with existing applications running on MS-Windows or Linux in a location-aware fashion. The system has been implemented, and shows the effectiveness of our approach. The second scenario is a location-aware video phone. 4.1
Location-Aware Interaction
The system enables a user to control a home appliance in a location-aware way. In the future, our home will have many display devices to show control panels
Making Existing Interactive Applications Context-Aware
1089
for controlling home appliances. A user usually likes to use the nearest display device to him to control a variety of home appliances. For example, if a user sits on a sofa, the control panel of a home application is displayed on his PDA. On the other hand, he is in front of a display device. The control panel is shown on the display device. In this case, if the display is a touch panel, he can access the control panel by touching the display. However, if the display is a normal display, he can navigate the control panel from his PDA or a game console. 4.2
Ubiquitous Video Phones
The second example is a ubiquitous video phone that enables us to use a video phone in various ways. In this example, a user speaks with his friend by using a telephone like a broadband phone developed by AT&T Labolatories, Cambridge[2]. The phone has a receiver like traditional phones, but it also has a small display. When the phone is used as a video phone, the small display renders video streams transmitted from other phones. The display is also able to show various information such as photos, pictures, and HTML files that are shared by speakers. Our user interface system makes the phone more attractive, and we believe that the extension is a useful application in ubiquitous computing environments. When a user needs to start to make a dinner, he will go to his kitchen, but he likes to keep to talk with his friend. However, a phone is put in a living room, and the traditional phone receiver is not appropriate to continue the conversation with his friend in the kitchen because his both hands may be used for cooking. In this case, we use a microphone and a speaker in the kitchen so that he can use both hands for making the dinner while talking with his friend. In the future, various home appliances such as a refrigerator and a microwave provide displays. Also, a kitchen table may have a display to show a recipe. These displays can be used by the video phone to show a video stream. In a similar way, a video phone can use various interaction devices for interacting with a user. The approach enables us to use a telephone in a more seamless way. Our system allows us to use a standard VoIP application running on Linux. The application provides a graphical user interface on the X window system. However, our system allows a user to be able to choose various interaction styles according to his situation. Also, if his situation is also changed, the current interaction style is changed according to his preference. Currently, we are incorporating an audio stream in our system, and it is easy to realize the example described in the section by adopting our approach.
5
Conclusion and Future Work
In the paper, we have proposed a new approach to build ubiquitous computing applications. Our middleware enables us to convert existing applications to context-aware applications without modifying existing applications. Therefore, a programmer needs not to learn new APIs to write context-aware applications.
1090
T. Nakajima et al.
Currently, we are extending our middleware to support video streams and audio streams. Also, we are implementing a location-aware video phone described in this paper. Our system can transmit video streams to any output devices if the bandwidth is enough to process the video streams. We like to incorporate a dynamic QOS control[8] scheme in our system. This means that the quality of video streams is changed according to the processing power of a machine to execute UniInt proxy programs. To support audio streams, we are working on an audio snoop program in a Linux kernel. The program runs in the kernel, and captures events about audio streams. The program communicates to UniInt proxy, and the UniInt proxy transmits audio streams to appropriate speakers, and receive audio streams from appropriate microphones. The audio streams can be converted according to the capabilities of microphones and speakers similar to other interaction devices.
References 1. J. Bacon, J Bates, and D. Halls, “Location-Oriented Multimedia”, IEEE Personal Communications, Vol.4, No.5, 1997. 2. AT&T Laboratories, Cambridge, “Broadband Phone Project”, http://www.uk.research.att.com/bphone/apps.html. 3. Citrix Systems, “Citrix Metaframe 1.8 Background”, Citrix White Paper, 1998. 4. A.K. Dey, D. Salber, and G.D. Abowd, “A Conceptual Framework and a Toolkit for Supporting the Rapid Prototyping of Context-Aware Applications”, HumanComputer Interactions, Vol.16, No.2–4, 2001. 5. Diego Lopez de Ipina, “Building Components for a Distributed Sentient Framework with Python and CORBA”, In Proceedings of the 8th International Python Conference, 2000. 6. Andy Harter, Andy Hopper, Pete Steggles, Andy Ward, Paul Webster, “The Anatomy of a Context-Aware Application”, In Proceedings of the 5th Annual ACM/IEEE International Conference on Mobile Computing and Networking, 1999. 7. Microsoft Corporation, “Microsoft WIndows NT Server 4.0: Technical Server Edition, An Architecture Overview”, Technical White Paper 1998. 8. T. Nakajima “A Dynamic QOS Control based on Optimistic Processor Reservation”, In Proceedings of the IEEE International Conference on Multimedia Computing and Systems, 1996. 9. T. Nakajima “Adaptive Continuous Media Applications in Mobile Computing Environment”, In Proceedings of the IEEE International Conference on Multimedia Computing and Systems, 1997. 10. T.Richardson, et al., “Virtual Network Computing”, IEEE Internet Computing, Vol.2, No.1, 1998. 11. B.N. Schilit, N.Adames, and R. Want, “Context-Aware Computing Applications”, In Proceedings of the IEEE Workshop on Mobile Computing Systems and Applications, 1994. 12. K.Soejima, M.Matsuda, T.Iino, T.Hayashi, and T.Nakajima, “Building Audio and Visual Home Applications on Commodity Software”, IEEE Transactions on Consumer Electronics, Vol.47, No.3, 2001. 13. Sun Microsystems, “Sun Ray 1 Enterprise Appliance”, http://www.sun.com/products/sunray1/.
Benefits and Requirements of Using Multi-agent Systems on Smart Devices Cosmin Carabelea¹, Olivier Boissier¹, and Fano Ramparany² ¹ SMA/SIMMO/ENS Mines de Saint-Etienne, France {cosmin.carabelea, olivier.boissier}@emse.fr ² DIH/OCF/France Télécom R&D
[email protected] Abstract. Due to the emergence of Internet and embedded computing, humans will be more and more faced to an increasing intrusion of computing in their day-to-day life by what it is called now smart devices. Agent characteristics like pro-activeness, autonomy and sociability and the inherent distributed nature of multi-agent systems make them a promising tool to use in the smart devices applications. This paper tries to illustrate the benefits of using multi-agent systems on smart communicating objects and discusses the need of using multiagent platforms. We then present an overview of multi-agent platforms created for use on small devices (i.e. devices with limited computing power) and for each reviewed multi-agent platform we try to identify what are its main characteristics and architectural concepts.
1 Introduction In the near future it is to be expected that humans will be more and more faced to an increasing intrusion of computing (in the form of smart devices) in their day-to-day life. The term smart device designates any physical object associated with computing resources and able to communicate with other similar objects via any physical transmission medium, and logical protocol, or with humans via standard user interface. The scale spans from big smart devices such as PDAs, to small ones such as RFID Tags. We are mainly interested in devices that share the following characteristics: tight memory constraints, limited computing power (due to low power consumption), limited user interface peripheral, embedded in the physical world and exhibiting real time constraints. The increasing number of such devices makes us envision a myriad of applications in which users will shortly need to be intelligently assisted in their interactions with these computing entities. The evolution surely will produce several users in interaction with several devices, so we need to introduce in these intelligent objects some cooperating and social reasoning capabilities. For many years, multi-agent systems have proposed a new paradigm of computation based on cooperation and autonomy. An intelligent agent is "a computer H. Kosch, L. Böszörményi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 1091–1098, 2003. © Springer-Verlag Berlin Heidelberg 2003
1092
C. Carabelea, O. Boissier, and F. Ramparany
system, situated in some environment, that is capable of flexible and autonomous action in order to meet its design objectives" [8]. A multi-agent system is a federation of software agents interacting in a shared environment, that cooperate and coordinate their actions given their own goals and plans. Many properties of this metaphor of computing make them attractive for tackling with the requirements presented above: • Pro-activity: agents can pro-actively assist their users and try to achieve goals given the evolution of the environment; they can help them in discovering information, in interacting. • (Adjustable) autonomy: agents are able to try to achieve their own goals, but are also able to take into-account and adapt their autonomy according to their users’ constraints. • Self-awareness: agents are conscious of themselves in the sense that they have a model of their goals, of their own competences, of their state. Although the multi-agent approach may seem suitable for the smart communicating objects, one has to solve the problems of the actual deployment of agents on those small devices. This paper tries to identify the requirements for executing agents on such devices (operating system, communication facilities, computational power, etc.) and, in the meantime, we try to identify the constraints that the use of small devices is imposing to the multi-agent architectures and services. The remainder of the paper is structured as follows. In the next section we will present our arguments on why the utilization of multi-agent systems on smart devices is benefical. Section 3 describes what a multi-agent platform is and what are the FIPA standards for those platforms; we then present some of the main characteristics of several multi-agent platforms for small devices. Finally, in Section 5 the utility of using multi-agent systems and platforms on small devices is discussed and we are drawing some conclusions and tracing directions for future work.
2 Why Use Multi-agent Systems on Smart Devices? Pervasive and ubiquitous computing will involve more and more objects endowed with computing and communication ressources, called smart devices. These smart devices should be able to sense their environment and react to changes in it. In order to alleviate the user information overload, it will be necessary to enable these objects to communicate directly with one another, while in the same time they should be able to communicate with the user. Due to the high number of these devices, we will have to reduce the number of objects needing explicit control from the user, thus making these devices autonomous. Another way of simplifying the human – smart device interaction is to provide smart devices with knowledge and reasoning capabilities, so that most of the repetitive tasks involved in the interaction could be automated. An intelligent agent is "a computer system, situated in some environment, that is capable of flexible and autonomous action in order to meet its design objectives" [8]. Beside the (adjustable) autonomy, an agent has other properties. It should be able to
Benefits and Requirements of Using Multi-agent Systems on Smart Devices
1093
detect and react to changes in its environment. Agents can pro-actively assist their users and try to achieve goals given the evolution of the environment; they can help them in discovering information, in interacting. We can have several agents acting in a common environment, thus obtaining what is called a multi-agent system. From the multi-agent point of view, an agent should be able to interact with other agents, i.e., communicate, cooperate or negotiate with them. Also, an agent should be able to reason about the existing organizational structures in the system. It is no surprise that the multi-agent applications try to solve the same problems as the applications on smart devices. Some of these problems are concerned on how to realize the interaction between the diverse entities in the system, while others on how the agents or the smart devices should adapt themselves to new situations. The smart devices environment is a very dynamic and open one, devices can enter or leave it at any time. Multi-agent systems research is also interested in the modelization of open systems: what check-in/check-out procedures are needed, how to ensure the service discovery in an open system and so on. The problem of trust and reputation is also tackled in both domains. It is clear for us that intelligent agents have characteristics desirable for smart devices. Moreover, the multi-agent systems already solve or try to solve similar problems with those of smart devices, so one can beneficiate from these solutions. Of course, an application for smart devices can be created without the multi-agent technology, but by using it, one can take advantage of the methodologies and solutions provided by the multi-agent paradigm. And we should not forgot the reusability: a multi-agent system created for a specific problem can be very easily used to solve another one. In other words, using agents on smart devices is not the the only perspective of solving the problem, but it might prove better than others.
3 How to Use Multi-agent Systems on Small Devices? A multi-agent platform is a software infrastructure used as an environment for agents’ deployment and execution. It should provide comfortable ways for agent programmers to create and test agents and it can be viewed as a collection of services offered to developers, but also to agents at runtime. As depicted in the previous characterization, the platform is an execution environment for the agents in the sense that it should allow one to create, execute and delete agents. Secondly, the platform should act as a middleware between the operating system and the applications (agents) running on it, as depicted in Fig.1. The developer will then be able to create agents for a platform and use them on all the systems that support the platform without changing the code. More than that, a platform should hide from the developer the communication details between the agents. If, for example, there are several communication protocols available on the device (e.g. HTTP, RMI, Bluetooth, etc.), the platform should choose the appropriate protocol for each situation and use it. This will allow one to create agents that are independent of the communication protocols available on the machine, leaving the actual sending of the information as a task for the platform.
1094
C. Carabelea, O. Boissier, and F. Ramparany
Multi-agents application Multi-agent platform OS
TCP/IP, Bluetooth, etc. Hardware
Fig. 1. Coarse-grained view of a device executing an agent platform and an agent application
Because the choice of an agent platform has a great influence on the way the agents are designed and implemented, FIPA1 has produced standards that concern how an agent platform should be structured [6], i.e. what are the basic services it should offer (e.g. a message transport service, an agent management service, a directory service). These standards exist to ensure a uniform design of the agents, independent of the multi-agent platform. Thanks to it, for instance, at the moment several agent platforms are deployed in several cities of the world on which agents can go and execute and cooperate with other agents [1]. In the next chapter we will give a brief description of several agent platforms designed for small (and wireless) devices, on which the limitations of computing resources have a great influence on the complexity of the applications. For each platform we will try to put in evidence the targeted devices and the wireless technologies it works with, as well as if it is FIPA-compliant or not and what are its most relevant characteristics. All the platforms we have investigated are Java-based, so we will try to specify for each one what is the Java Virtual Machine (VM) it needs.
4 Multi-agent Platforms for Small Devices Due to the limited resources available, the agent platforms for the small devices need to have a light architecture. Most of these platforms are designed to run completely (i.e. with all the services they provide) on the small device, while others are keeping some of the services (or sometimes all) on a server and they are downloaded on the device only when needed. From this point of view, we can classify existing platforms in three categories replicating the conceptual classification presented in [14]: Portal platforms that don’t execute the agents on the small devices, but on other hosts. The small device is used only as an interface with the user. Embedded platforms are executed entirely on the small device(s), together with one or several agents. Surrogate platforms that are executing only partially on the small device, while some part of them is executed on other hosts.
1
Non-profit organization aimed at producing standards for the interoperation of heterogeneous software agents. http://www.fipa.org
Benefits and Requirements of Using Multi-agent Systems on Smart Devices
1095
In what follows, we will present the basic characteristics of several platforms from all categories. 4.1 Portal Platforms – MobiAgent The MobiAgent [13] system architecture consists of three main components: a handheld mobile wireless device, an Agent Gateway and the network resources. The Agent Gateway is executing an agent platform and the agents for that platform. When the user wants to delegate a task to an agent, the mobile device connects to the gateway and downloads an interface that will configure the agent for her. The agent will perform its task and it will later report the results via the same mechanism. Note that neither the platform, neither the agents are running on the device, the only application running there is a MIDlet that configures the agent. This approach does not impose anything to the platform or to the agents running on the Agent Gateway; the platform running there may be FIPA-compliant, but this is not mandatory. This infrastructure is available on any device supporting J2ME/CLDC/MIDP, including mobile phones. The connection between the device and the gateway is done via HTTP over a wireless network. Unfortunately, no links for downloading the platform are currently available. 4.2 Surrogate Platforms – kSaci kSACI [9] is a smaller version of the SACI platform [15] suited to the kVM. SACI is an infrastructure for creating agents that are able to communicate using KQML [10] messages. Each SACI agent has a mailbox to exchange messages with the others. Its architecture also contains a special agent, called the facilitator, offering the white- and yellow-pages services of the system. The architecture of the platform is not FIPAcompliant. The kSACI platform is usable on small devices running the kVM, but the platform is not entirely situated on the small device. In fact, the facilitator is on a server and on every small device there is only one agent running. This agent should contain a mailbox used to exchange messages, but it doesn’t know how to pass messages to the others; it communicates via HTTP with a proxy running on a desktop machine that further passes the messages to and from other agents. This solution makes the agent lighter, so this platform can be used with devices as small as a mobile phone or a twoway pager. 4.3 Embedded Platforms Several multi-agent platforms exist. Among them few are suited to be embedded on small devices. We have investigated four of them, but, due to space limitations, only two are presented here. The interested reader is invited to [4] for more information about the two other platforms, MAE and microFIPA-OS.
1096
C. Carabelea, O. Boissier, and F. Ramparany
4.3.1 AgentLight AgentLight [2] is an agent platform with the objectives of being capable of deployment on fixed and mobile devices with various operating systems and that can operate over both a fixed and a wireless network. Its authors are using a set of profiles that allows one to configure the platform for several devices and/or operating systems. Also, this platform aims to be FIPA-compliant and operating system agnostic. The architecture of AgentLight is called “half-daisy” and consists of an agent container running on each device on which several agents can be placed. If the two agents on the same device communicate with each other, the agent container does the communication internally, but if the agents are on separate devices, the agent container acts as a proxy. Each agent is seen as a set of rules and it is using for execution an inference engine provided by the agent container. This architecture is not entirely FIPA-compliant (e.g. the directory service is missing) and it is still work in progress. It can be run on devices with J2ME/CLDC/MIPD and with the kVM and the smallest device targeted is a mobile phone, although for the moment there are not any tests to prove a mobile phone can support the platform, especially the inference engine that can be quite heavy at the execution time. 4.3.2 LEAP The Lightweight Extensible Agent Platform (LEAP 3.0) is probably the most known agent platform for small devices. It is the result of the LEAP project [12] that had the objective of creating a entirely FIPA compliant platform with similar objectives to the AgentLight Platform: be capable of deployment on fixed and mobile devices with various operating systems and deployed in wired or wireless networks. Since the last version, 3.0, LEAP is an add-on of the JADE platform [7] and it uses a set of profiles that allows one to configure it for execution on various machines, OS and Java VM. The architecture of the platform is modular and is organized in terms of modules among which some can be mandatory, requested by FIPA specifications, while other are optional. The mandatory ones are the kernel module that manages the platform and the lifecycle of agents and the communication module, which handles the heterogeneity of communication protocols. Some of the modules are device-dependent (e.g. the communication module), while others are independent of the characteristics of the device (e.g. a security plug-in for the communication module). The platform is operating system agnostic and it can run on devices ranging from mobile phones and PDAs to workstations. It can work over wireless networks that support TCP/IP, like WLAN or GPRS. As in AgentLight, the architecture of LEAP splits the platform in several containers, one for every device/workstation used, with one or several agents in each container. These containers are responsible for passing the messages between agents by choosing the appropriate communication protocol available. One of these containers has a special role and it is called main-container; it contains two special agents that implement the FIPA specifications to provide the white and yellow pages services. This main-container cannot be executed on a small device, but only on a PC. In order to allow the execution of containers even on very limited devices, LEAP has
Benefits and Requirements of Using Multi-agent Systems on Smart Devices
1097
two execution modes: stand-alone and split. In the former, the container is completely embedded on the small device, while in the former, a part of the container is executed on a PC. Table 1. Characteristics of multi-agent platforms for small devices Platform
MobiAgent
kSACI
AgentLigh t
Connection to SD
portal
surrogate
embedded
surrogate / embedded
mobile phone
mobile phone
mobile phone
it can be
no
Smallest targeted device FIPAcompliant No. of agents on device Available for download JavaVM
LEAP
MAE
micro FIPA-OS
embedd ed
embedded
mobile phone
PDA
PocketPC
yes (?)
yes
no
yes
several
several (pref. 1)
0
1
several
several (pref. 1)
no
yes
yes
yes
no
yes
kVM
kVM
kVM
various
various
PersonalJava
5 Discussion and Conclusions In this article we have presented some benefits of using multi-agent systems with the smart devices. But this usage of multi-agent systems on small devices has its drawbacks, i.e. the need for an existing infrastructure that will allow the creation and execution of agents and the communications between them: a multi-agent platform. We have presented a set of multi-agent platforms for small devices, underlining some of their major characteristics and architectural concepts. We have used the three main directions proposed in [14] to bind smart devices and multi-agent platforms: portal, surrogate and embedded. It was beyond the scope of this paper to identify one of these platforms as being the best. We believe there doesn’t exist such a platform, but each one of them is suited for different applications. Table 1 summarizes the most important characteristics we have identified for each platform. As future work we intend to set up a benchmark to evaluate these platforms within several scenarios; the results obtained will then allow us to analyse the platforms’ performances in different situations and what is the real amount of computational resources (processing power, memory size) needed by each one. We have begun these practical tests by deploying LEAP on a Siemens S55 mobile phone. So far, the results are encouraging: the phone supports the execution of the platform and of an agent, although with a low execution speed. When using a multi-agent platform on a small device, there is a high risk that the device resources will not be enough for both the platform and the agent(s) running on
1098
C. Carabelea, O. Boissier, and F. Ramparany
it. Although the use of a platform can greatly reduce the developing cost and increase portability, it is clear that a trade-off should be found between the services the platform offers and its size. One might eventually arrive in the situation of using a small device like a mobile phone and a platform designed to work on it (like LEAP) and to discover that the platform is too big for the device’s capabilities or there is no room left for any agents [3]. As for the FIPA-compliancy, some projects such as Extrovert-Gadgets [5] for instance, question its utility for multi-agent applications on small devices. The advantage of compelling to some standards is clear, but these standards have not been designed for the particular characteristics of the small and wireless devices. A good example is presented in [11] where the LEAP platform is used, but some of its mandatory modules, which make it FIPA-compliant, are useless for some applications (e.g. ad-hoc networks).
References 1. 2. 3. 4. 5. 6. 7. 8.
9.
10. 11. 12. 13.
14. 15.
AgentCities: http://www.agentcities.org AgentLight – Platform for Lightweight Agents: http://www.agentlight.org Berger, M. et al.: Porting Distributed Agent-Middleware to Small Mobile Devices. In Proc. of the Ubiquitous Computing Workshop, Bologna, Italy (2002) Carabelea, C., Boissier, B.: Multi-agent platforms for smart devices: dream or reality?. In Proc. of the Smart Objects Conference (SOC'03), Grenoble, France (2003), p. 126-129. E-Gadgets: http://www.extrovert-gadgets.net FIPA Abstract Architecture Spec.: http://www.fipa.org/repository/architecturespecs.html JADE – Java Agents DEvelopment Framework: http://jade.cselt.it Jennings, N.R., Sycara, K., Wooldridge, M.: A Roadmap of Agent Research and Development. Int. J. of Autonomous Agents and Multi-Agent Systems 1 (1) (1998), p. 7– 38 kSACI: http://www.cesar.org.br/~rla2/ksaci/ Labrou, Y., Finin, T.: A Semantics approach for KQML – A General Purpose Communication Language for Software Agents. In Proc. of Int. Conf. on Information and Knowledge Management (1994) Lawrence, J.: LEAP into Ad-Hoc Networks. In Proc. of the Ubiquitous Computing Workshop, Bologna, Italy (2002) LEAP – the Lightweight Extensible Agent Platform: http://leap.crm-paris.com Mahmoud, Q.H.: MobiAgent: An Agent-based Approach to Wireless Information Systems. In Proc. of the 3rd Int. Bi-Conference Workshop on Agent-Oriented Information Systems, Montreal,Canada (2001) Ramparano, F., Boissier, O.: Smart Devices Embedding Multi-agent Technologies for a Pro-active World. In Proc. of the Ubiquitous Computing Workshop, Bologna, Italy (2002) SACI – Simple Agent Communication Infrastructure: http://www.lti.pcs.usp.br/saci/
Performance Evaluation of Two Congestion Control Mechanisms with On-Demand Distance Vector (AODV) Routing Protocol for Mobile and Wireless Networks . Azzedine Boukerche University of Ottawa, Canada
[email protected] Abstract. In this paper, we focus upon the congestion control problem in on-demand distance vector routing (AODV) protocol for mobile and wireless ad hoc networks. We propose to study two mechanisms to deal with the congestion problem within AODV routing protocol. First, we investigate a randomized approach of AODV protocol. then we present a preemptive ad hoc on-demand distance vector routing protocol for mobile ad hoc networks. We discuss the implementation of both algorithms, and report on the performance results of simulation of several scenarios using the ns-2 ad hoc network simulator.
1
Introduction
Ad hoc wireless networks are expected to play an increasingly important role in future civilian and military settings where wireless access to wired backbone is either ineffective or impossible. However, frequent topology changes caused by node mobility make routing in ad hoc wireless networks a challenging problem [3,4,5,6]. In this paper, we are concerned with the congestion problem and packets dropped that are mainly due to the path break in the ad hoc network where the topology changes frequently, and the packet delay due to finding a new path and the transfer time. As a consequence, we present a Randomized AODV protocol (R-AODV), and Pr AODV , an extension to the AODV routing protocol using a preemptive mechanism. While R-AODV uses a randomized approach to divert routing packets from congested paths to reliable and less congested paths, Pr AODV scheme a preemptive approach to find a new path before the existing path breaks, thereby switching to this new path. Our main focus in this paper is to study the feasibility and the effectiveness of both congestion control mechanisms in the proposed protocol.
Part of this research is supported by Canada Research Chair program and Texas Advanced Research Program grant ATP/ARP
H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 1099–1108, 2003. c Springer-Verlag Berlin Heidelberg 2003
1100
2
A. Boukerche
The Basic AODV Protocol
AODV uses an on-demand based protocol to discover the desired path, while using hello packets to keep track of the current neighbors. Since it is an on demand algorithm, it builds routes between nodes only upon request by the source nodes. It maintains these routes as long as they are needed by the sources. It is loop-free and self-starting. Earlier studies have also shown that AODV protocol scales quite well when we increase the number of mobile nodes [2]. AODV allows mobile nodes to obtain routes quickly for new destinations and does not require mobile nodes to maintain routes to destinations that are not in active communication. It also allows mobile nodes to respond to situations associated with broken link and changes in network topology in a timely manner, as we shall see later. AODV uses a destination broadcast id number to ensure loop freedom at all times, since the intermediate nodes only forward the first copy of the same request packet. The sequence number is used in each node, and the destination sequence number is created by the destination for any route information it sends to requesting nodes. It also ensures the freshness of each established route. The route is updated if a new reply is received with the greater destination sequence or the same destination sequence number but the route has fewer hops. Therefore, this protocol will select the freshest and shortest path at any time. • Path discovery: When a route is needed, the source invokes a path discovery routine. It has two steps: the source sends out a request and the destination return a reply. In what follows, we will discuss these two steps. Request phase: When a route to a new destination is needed, the node uses a broadcast RREQ to find a route to that destination. A route can be determined when the RREQ reaches either the destination itself or an intermediate node with a fresh route to the destination. The fresh route is an unexpired route entry for the destination associated. The important step during the request phase is that a reverse path from the destination to the source can be set up. When a source node wants to find a path to a destination, it broadcasts a route request packet to its neighbors. The neighbors update their information for the source node, set up backwards pointers to the source node in their route tables, and rebroadcast the request packet. Reply phase: When the request arrives at the destination or at an intermediate node that has a path to that destination a reply packet is returned to the source node along the path recorded. While the reply packet propagates back to the source, nodes set up forward pointers to the destination, and set up the forward path. Once the source node receives the reply packet, it may begin to forward data packets to the destination. • Data Packet Forwarding: After the reply packet returns from the destination, the source can begin sending out the packet to the destination via the new discovery path, or the node can forward any enqueued packets to a destination if a reverse path is set up and the destination of the reverse path is the destination of the path.
Performance Evaluation of Two Congestion Control Mechanisms
1101
A route is considered active as long as there are data packets periodically traveling from the source to the destination along that path. Once the source stops sending data packets, the links will time out and eventually be deleted from the intermediate node routing tables. • Dealing with the broken links: If a link breaks while the route is active, the packets that are flowing in that path can be dropped and an error message is sent to the source or a local repair routine will take over. If the node holding the packets is close to the destination, this node invokes a local repair route. It enqueues the packet and finds a new path from this node to the destination. Otherwise, the packets are dropped, and the node upstream of the break propagates a route error message to the source node to inform it of the now unreachable destination(s). After receiving the route error, if the source node still desires the route, it can re-initiate a route discovery mechanism. However, we can add a preemptive protocol to AODV and initiate a rediscovery routine before the current routes goes down. In the next section, we will discuss this preemptive approach in more detail.
3
Randomized AODV (R-AODV) Protocol
AODV protocol is extended with a drop factor that induces a randomness feature to result in Randomized Ad-Hoc On-Demand Routing (R-AODV) protocol. During the route discovery process, every intermediary or router nodes between the source and the destination nodes makes a decision to either broadcast/forward the RREQ packet further towards the destination or drop it. Before forwarding a RREQ packet, every node computes the drop factorwhich is a function of the inverse of the number of hop counts traversed by the RREQ packet. This drop factor lies in the range of 0 to 1. Also, the node generates a random number from 0 to 1. If this random number is higher than the drop factor, the node forwards the RREQ packet. Otherwise, the RREQ packet is dropped. Dropping of RREQ packets does not necessarily result in a new route discovery process by the source node. This is due to the fact that the original broadcast by the source node results in multiple RREQ packets via the neighbors and this diffusing wave results quickly in a large number of RREQ packets traversing the network in search of the destination. A major proportion of these packets are redundant due to the fact that in the ideal case, a single RREQ packet can find the best route. Also, a number of these packets diffusing in directions away from the destination shall eventually timeout. Hence, in R-AODV, the aim is to minimize on these redundant RREQ packets, or alternatively, drop as much as possible of these redundant RREQ packets. The drop policy is conservative and its value becomes lesser with higher number of hops. As RREQ packets get near the destination node, the chances of survival of RREQ packets is higher. Hence, the first phase of the route discovering process, that is, finding the destination node, is completed as soon as possible and a RREP packet can be transmitted from the destination node back to the source node.
1102
A. Boukerche
In R-AODV, the dropping of redundant RREQ packets reduces a proportion of RREQ packets that shall never reach the destination node, resulting in a decrease of network congestion. Hence, the ratio of the number of packets received by the nodes to the number of packets sent by the nodes, namely, throughput, should be higher in R-AODV compared to AODV. The following algorithm is used in the decision making process of whether to drop the RREQ packets by the intermediary or routing nodes. Step 1: Calculate drop_factor drop_factor = (1/(Hop_count_of_RREQ_packet + 1)) Step 2: Calculate a random value in the range of 0 to 1. Step 3: If (random_value > drop_factor) then broadcast/forward RREQ_packet else drop RREQ_packet
4
Adaptive Preemptive AODV Protocol
In this section, we introduce the preemptive protocol first, then we discuss how this preemptive protocol is added into the original AODV, The preemptive protocol initiates a route rediscovery before the existing path breaks. It overlaps the route discovery routine and the use of the current active path, thereby reducing the average delay per packet. During the development of Pr AODV development, we have investigated several preemptive mechanisms. In this paper, we will settle on the following two approaches: (i) Schedule a rediscovery in advance: In this approach, when a reply packet returns to the source through an intermediate node, it collects the information of the links. Therefore, when the packet arrives at the source, the information about the condition of all links will be known, including the minimum value of the lifetime of the links. Hence, we can schedule a rediscovery Trediscovery time before the path breaks. (ii) Warn the source before the path breaks: Some mechanisms are needed to take care of finding which path is likely to break. We can monitor the signal power of the arrived packets as follows: when the signal power is below a threshold value, we begin the pingpong process between this node and its immediate neighbor nodes. This node sends to its neighbors in the upstream a hello packet called ping, and the neighboring nodes will respond with a hello packet called pong. Such ping-pong messages should be monitored carefully. In our approach, when bad packets were received (or we timeout on ping packets) a warning message should be sent back to the source during the monitoring period. Upon receiving a warning message, a path rediscovery routine is invoked.
In our preemptive AODV protocol, we combine the above two preemptive mechanisms, and add them to the original AODV. Pr AODV protocol is built based on the following assumptions. Future work will directed to eliminate such assumptions. – Congestion is caused only due to the path breaks. Other factors such as the limited bandwidth available on wireless channels and the intensive use of resources in an ad hoc network cannot affect our protocol. We assume that the network has enough bandwidth available, and each node has enough capacity to hold packets.
Performance Evaluation of Two Congestion Control Mechanisms
1103
– The route discovery time can be estimated. We estimate the discovery time by monitoring the request and reply packets, as the sum of the time required to process and monitor these two packets. – We can detect when a link breaks by monitoring the transfer time of packets between the two neighboring nodes. Based on the above assumptions, we present our Pr AODV as the following: (i) Schedule a rediscovery: We have added an additional field, which we refer to as a lifetime, in the reply packet to store the information of the links along the new discovered path. Each link associates a time-to-live, a parameter that is set up in the request phase. The lifetime field contains the minimum value of the time-to-live of the links along a path. The reply packet also has an additional field, a timestamp, which indicates the time that the request packet was sent out. Let us now see how we compute RT T , the estimated time needed to discover a new path. When the reply packet arrives at the source node, it computes RT T as (arrivalT ime − timestamp), then schedule a rediscovery at time (lif etime − RT T ). We use RT T as our estimated value for the time needed to discover a new path. (ii) Warning the source: Recall that a preemptive protocol should monitor the signal power of the receiving packet. In our approach, we monitor the transfer 0 time for the hello packets. According to [1], Pn = P r n , where P0 is a constant for each link and n is also a constant, we can see that the distance, which connects to the transfer time of the packet via each link, has a one-to-one relationship to the signal power. Thus, by monitoring the transfer time of packets via each link, we can identify when the link is going to break. In our protocol, when a node (destination) receives a real packet other than AODV packets, it sends out a pong packet to its neighbor (source), i.e., where the packet comes from. Note that there is no need to send this pong packet if we can include a field in the real packet to store the current time the packet is sent. However, in the original packet, it doesn’t have such a field, hence, we need an additional pong packet to get such information. In both ping and pong packets, we include a field containing the current time that the packet is sent. When the source monitoring the transfer time is greater than some specific value c1 , it begins the ping-pong process. If more than two pong packets arrived with the transfer time greater than a constant c2 , the node will warn the source of the real packet. Upon receiving the warning message, the source checks if the path is still active. This can easily be done, since each route entry for a destination associates a value, rt last send, which indicates if the path is in use or not. If the path is still in use, then a rediscovery routine is invoked. Theoretically, these two cases can take care of finding new paths before current paths break. The first case is quite helpful if the data transmission rate is slow. If only one data packet is sent out per time-expired period, then even the current route is still needed. Note that this entry may not have packets to go through it during the next time period. This can happen if the interval between
1104
A. Boukerche
two continuous packets is greater that the timeout period of the current route. The second case can be used to find a new path whenever a warning message is generated. However, those two cases are also based on some assumptions, such as the span of a link which can be estimated by the time out of ping packets and the values of those constants are accurate. Once we break those assumptions, the test results will be unexpected.
5
Simulation Experiments
We have implemented both R-AODV and Pr AODV protocols using ns-2, version 2.1b8. The implementation of both algorithms is based on the CMU Monarch extension. The source-destination pairs are spread randomly over the network. By changing the total amount of traffic sources, we can get scenarios with different traffic loads. The mobility models used in our experiments to investigate the effect of the number of nodes are 500m x 500m field with different number of nodes. Each node starts its journey from a random location to a random destination with a randomly chosen speed uniformly distributed between 0-20m/sec. Once the destination is reached, another random destination is targeted after a pause. Varying the pause time changes the frequency of node movement. In each test, the simulation time is 1000 seconds. The experimental results were obtained by averaging several trial runs. In order to evaluate the performance of R-AODV and Pr AODV ad hoc routing protocols, we chose the following metrics: – Routing path optimality: the difference in the average number of hops between the optimal (shortest) routing path and the real routing path. – Throughput: (i.e., packet delivered ratio) is a ratio of the data packets delivered to the destination to those generated by the CBR sources. – Average delay per packet: this includes all possible delays caused by buffering during route discovery latency, queuing at the interface queue, retransmission delays at the MAC, and propagation and transfer time. – Packet arrived: this is the number of packets that are successfully transferred to the destination. 5.1
Simulation Results
In this section, we report on the performance results of simulation of several workload models of several scenarios we have used to evaluate the performance the both R-AODV and the preemptive ad hoc routing protocols. AODV vs. R-AODV. Recall that the use of the drop factor in R-AODV will to a drop of some forwarding packets but the drop rate is conservative enough not to let the source node to result in a loop of initiation of route discoveries. The routing path optimality is illustrated in figure 1. As we can see, the routing path optimality is better in R-AODV compared to AODV on an average.
Performance Evaluation of Two Congestion Control Mechanisms
1105
We also observe that the routing path optimality is about 20% better in R-AODV compared to AODV on an average. larger percentage were observed with a larger field. This is mainly due to the induction of the dropping factor in R-AODV to reduce the node congestion. Figures 2 shows the comparison of AODV with R-AODV on packet overflow due to the limited packet buffer space. It is observed that the number of packets overflowed in R-AODV is lower on the average. This is due to the induction of the drop factor in R-AODV to decrease node congestion. recall that the packets overflow is dependent on the dynamic network topology. It decreases with an increase in the pause time of the nodes in both AODV and R-AODV. This implies that for more stable networks, wherein the nodes are static for higher quantum of time, the latency in finding a route to a node is relatively less. The throughput is given in figures 3. As we can see, R-AODV exhibits a higher throughput on the average since most of the forwarding packets that were dropped were redundant packets and allows more packets to reach their destination without being dropped due to reduce node congestion. The higher stable network dynamics result in a higher throughput. Overall, since throughput is higher for R-AODV, lesser number of packets that were sent out by the nodes in the network were lost during the route discovery process, thus reducing network congestion in R-AODV when compared to AODV. Figure 4 show improved delay performance in most cases in R-AODV. Especially, for larger networks and higher mobility rate, the performance improvement shows a obvious contrast compared to AODV. For high mobility rate, the routes are frequently renewed with the newly discovered neighbors. Consequently the route update rate increases and R-AODV has more chances to find alternative paths due to decrease node congestion. The delay drops as the pause time increase. A main reason attributed to this observation is that as node mobility decrease, the route update interval increases since low node mobility results in less route updates. Thus, delay increases depending on node mobility. The overhead is presented in figure 5 which show similar characteristics for a given pause time in both R-AODV and AODV. AODV vs. Pr AODV . In general, in both AODV and Pr AODV , the number of nodes does affect their performance. We have observed that with the preemptive protocol, Pr AODV can significantly improve the performance of the throughput, the average delay per packet, and the packet delivered. The pause time and the packet rate can also affect its performance. In the course of our experiments, we have observed that Pr AODV has much better performance than AODV when the packet rate is low. In our experiments, we set the traffic flow to 10-pairs, the interval traffic rate to 4, and the pause time to 600. As illustrated in Figure 6 our results indicate that that the number of nodes does not have any affect on both protocols, AODV and Pr AODV . However, the preemptive AODV protocol indicates that it is more stable than AODV. This is mainly due to the fact that Pr AODV balances the traffic quite efficiently to a new path before the current one breaks. The best
1106
A. Boukerche 1 R-AODV AODV
1 R-AODV AODV
0.8
Average Difference between Optimal Path and Actual Path
Average Difference between Optimal Path and Actual Path
0.8
0.6
0.4
0.2
0.6
0.4
0.2
0 0
50
100
150
200
250
300
350
400 0
Pause Time (second), 100 nodes, 500m x 500m network
0
Fig. 1. Average Route Path Optimality – 500m x 500m
100 150 200 250 300 Pause Time (second), 100 nodes, 500m x 500m network
350
400
Fig. 2. Overflow – 500m x 500m
Throughput - R-AODV Throughput - AODV
1.004
50
Delay - R-AODV Delay - AODV
0.09 0.08
1.002
0.07 1
Delay
Throughput
0.06 0.998
0.996
0.05 0.04 0.03
0.994
0.02 0.992 0.01 0.99 0
50
100 150 200 250 300 Pause Time (second), 100 nodes, 500m x 500m network
350
400
Fig. 3. Throughput – 500m x 500m
0
50
100 150 200 250 300 Pause Time (second), 100 nodes, 1000m x 1000m network
350
400
Fig. 4. Delay – 500m x 500m Packets Received - R-AODV Packets Received - AODV
7860
7840
7820
Overhead
7800
7780
7760
7740
7720
7700 0
50
100 150 200 250 300 Pause Time (second), 100 nodes, 500m x 500m network
350
400
Fig. 5. Overhead – 500m x 500m
performance is observed when the number of nodes is set to 35, as illustrated in Figure 6. In our experiments, we have noticed that this type of behavior is always true with only 5 traffic pairs as well. In our next set of experiments, we wish to study the effect of the number of traffic on the performance of of Pr AODV . Based upon our initial experiments, (see above), we choose a scenarios with 35 nodes and 5 traffic pairs. We also set the pause time at 600 seconds. Our results are as indicated in Figure 7. In both protocols, we observe an increase of the the number of the packet arrivals as we increase amount of traffic. The throughput obtained with Pr AODV is lower when compared to the AODV protocol, and the delay obtained with the preemptive AODV protocol is lower when compared to the original AODV. This is mainly due the preemptive paradigm used in Pr AODV and the number of packets that are dropped during the simulation.
Performance Evaluation of Two Congestion Control Mechanisms 1110
1107
4500 AODV PAODV
AODV PAODV 4000
1105 3500 1100
Arrive
Arrive
3000
1095
2500
2000 1090 1500 1085 1000
1080
500 20
25
30
35
40
45
50
2
0.996
4
6
8
10 number_of_traffic
12
14
16
18
0.996 AODV PAODV
AODV PAODV
0.995
0.995
0.994 0.994
Throughput
Throughput
0.993
0.992
0.993
0.992
0.991 0.991 0.99 0.99
0.989
0.988
0.989 20
25
30
35
40
45
50
2
4
6
8
10
12
14
16
18
number_of_traffic
0.08
0.036
AODV PAODV
AODV PAODV 0.07 0.034
0.06 0.032
Delay
Delay
0.05 0.03
0.04 0.028 0.03
0.026
0.02
0.01
0.024 20
Fig. 6. nodes
6
25
30
35
40
45
Effect of the number of
50
2
4
6
8
10 number_of_traffic
12
14
16
18
Fig. 7. The effect of the amount of traffic pairs
Conclusion
In this paper, we have studied the congestion problem in mobile and wireless networks. and we have presented two congestion control mechanisms for AODV ad hoc routing protocol, which we refer to as, R-AODV, a Randomized AODV protocol, and Pr AODV , an extension to the AODV routing protocol using a preemptive mechanism. We have discussed both algorithms and their implementations. We have also reported a set of simulation experiments to evaluate their performance when compared to the original AODV on-demand routing protocol. Our results indicate that both R-AODV and Pr AODV protocols are easy to scale, the number of mobile nodes does not affect their performance. and the traffic is quite well balanced among the availiable paths in the network. While our results indicate that both protocols increase the number of arrival packets within
1108
A. Boukerche
the same simulation time, they decrease the delay per packet when compared to the original AODV scheme.
References 1. Preemptive Routing in Ad Hoc Network URL: http://opal.cs.binghamton.edu/ nael/research/papers/mobicom01.pdf 2. A. Boukerche, “Simulation Based Comparative Study of Ad Hoc Routing Protocols” 34th IEEE/ACM/SCS Annual Simulation Symposium, April 2001, pp. 85–92. 3. A. Boukerche, S. K. Das, and A. Fabbri “Analysis of Randomized Congestion Control with DSDV Routing in Ad Hoc Wireless Networks” Journal of Parallel and Distributed Computing, pp. 967–995, Vo. 61, 2001. 4. D.B. Johnson, D.A. Maltz, Dynamic Source Routing in Ad Hoc Wireless Networks, in: Mobile Computing, Editors: T. Imielinski and H.F. Korth (Kluwer Academic Pub. 1996), Ch.5, 153–181. 5. C.E. Perkins, Ad Hoc On Demand Distance Vector (AODV) Routing, IEFT Internet Draft, available at: http://www.ieft.org/internet-drafts/draft-ietfmanet-aodv-02.txt (November 1998). 6. C.E. Perkins, P. Bhagwat, Highly Dynamic Destination-Sequenced Distance Vector Routing (DSDV) for Mobile Computers, Proceedings of ACM SIGCOMM’94, (1994), 234–244.
Towards an Approach for Mobile Profile Based Distributed Clustering Christian Seitz and Michael Berger Siemens AG, Corporate Technology, Information and Communications, 81730 Munich, Germany,
[email protected] [email protected] Abstract. We present a new application for mobile ad hoc networks, which we call Mobile Profile based Distributed Clustering(MPDC), which is a combination of mobile clustering and data clustering. In MPDC each mobile host is endowed with a user profile and while the users move around, hosts with similar profiles are to be found and a robust mobile cluster is formed. The participants of a cluster are able to cooperate or attain a goal together. We adapt MPDC to a taxi sharing application, in which people with similar destinations form a cluster and could share a taxi or other public transportation resources.
1
Introduction
Technology developments like mobile devices and mobile communication have formed a new computing environment which is referred as mobile computing, in which an entire new class of distributed applications has been created. An mobile ad hoc network consist of hosts travelling through physical space and communicating in an opportunistic manner via wireless links. In the absence of a fixed network infrastructure, the mobile hosts must discover each other’s presence and establish communication patterns dynamically. The structure of an ad hoc mobile network is highly dynamic. In an ad hoc network two hosts that want to communicate may not be within wireless transmission range of each other, but could communicate if other hosts between them are also participating in the ad hoc network and are willing to forward packets for them. The absence of a fixed network infrastructure, frequent and unpredictable disconnections, and power considerations render the development of ad hoc mobile applications a very challenging undertaking. We present a new mobile ad hoc network application area, which me call Mobile Profile based Distributed Clustering (MPDC). In MPDC each mobile host is endowed with a user profile and while the users move around, hosts with similar profiles are to be found and a mobile cluster is formed. We apply MPDC to a taxi sharing scenario. If a train arrives at a railway station or an airplane at an airport the different passengers may have the same destination e. g. a hotel. This destination address is part of a user profile and H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 1109–1117, 2003. c Springer-Verlag Berlin Heidelberg 2003
1110
C. Seitz and M. Berger
stored on a mobile device. While people are waiting at the baggage terminal the mobile devices exchange the profiles and try to find small groups with similar destinations. The rest of the paper is organized as follows. Section 2 gives an overview of related work of other clustering problems. In section 3 we outline the architecture of MPDC, define the ad hoc network model and make some assumption. The next section describes the used algorithms in our approach to MPDC and finally, section 5 concludes the paper.
2
Problem Classification and Related Work
Mobile Profile based Distributed Clustering comprises three main problems. The first problem is the dynamic behavior of an ad hoc network, where the number of hosts and communication links permanently changes. Furthermore, an expandable profile has to be defined and a mechanism must be created to compare profile instances. Finally, similar profiles have to be found in the ad hoc network and the corresponding host form a robust cluster, in spite of the dynamic behavior of the ad hoc network. The term clustering is used in the research areas databases, data mining and in the mobile networks are. Clustering in mobile networks describes the partitioning of a mobile network in several, mostly disjoint, clusters [1,2]. The clustering process comprises the determination of a cluster head in a set of hosts. A cluster is a group of hosts, all able to communicate with the clusterhead. This clustering takes place at the network layer and is used for routing purposes. Clustering in the database or data mining area encompasses the search for similar data sets in huge data bases. In the surveys of Fasulo [4] or Fraley and Raftery [5] an overview of many algorithms for that domain can be found. Maitra [8] and Kolatch [7] examine data-clusters in distributed databases. In Mobile Profile based Distributed Clustering mobile hosts are equipped with wireless transmitters, receivers, and a user profile. They are moving in a geographical area and form an ad hoc network. In this environment hosts with similar profiles have to be found. Therefore, MPDC must combine the two aforementioned clustering approaches to accomplish its objective. The problems, arising by means of the motion of the hosts could be solved by methods used in the mobile network area. Searching for similar profiles is based on algorithms of data clustering. Both methods must be adapted to MPDC, e. g. while in the database area millions of data sets must be scanned, in the MPDC application at the utmost one hundred other hosts are present. In contrast to data sets in databases ad hoc hosts move around and are active, i. e. they can publish their profile by their own. There is other work that analyzes ad hoc clustering algorithms. Roman et al. [10] deals with consistent group membership. They assume that the position of each host is known by other hosts and two hosts do only communicate with each other, if it is guaranteed that during the message exchange the transmission
Towards an Approach for Mobile Profile Based Distributed Clustering
1111
range will not exceed. In our environment obtaining position information is not possible, because such data is not always available, e. g. inside of buildings.
P ro file
P ro file M a tc h in g
C lu s te rin g
C lu s te rin g
P ro file M a tc h in g
T o p o lo g y C re a tio n
T o p o lo g y C re a tio n
In itia to r D e te rm in a tio n
In itia to r D e te rm in a tio n
P ro file
Fig. 1. Architecture
3
Architecture
In this section the architecture of MPDC is presented, which consists of three layers. With each layer at least one agent is associated. Figure 1 envisages the layered architecture of MPDC. The lowest layer is the Initiator Detection layer, which assigns the initiator role to some hosts. An initiator is needed in order to guarantee, that the algorithm of the next layer is not started by each host of the network. This layer does not determine one single initiator for the whole ad hoc network. It is sufficient, if the number of initiator nodes is only reduced. The Virtual Topology layer is responsible for covering the graph G with another topology, e. g. a tree or a logical ring. This virtual topology is necessary to reduce the number of messages, that are sent by the mobile hosts. First experiences showed, that a tree is the most suitable virtual topology and therefore we will only address the tree approach in this paper. The next layer is the most important one, the Clustering layer, which accomplishes both, the local grouping and the decentralized clustering. Local grouping comprises the selection of hosts which are taken into account for global clustering. Decentralized clustering encompasses the exchange of the local groups with the goal to achieve a well defined global cluster. The Profile Matching module, depicted with a dashed box in figure 1 is responsible for comparing two profiles. Below, two definition are given to distinguish local grouping and decentralized clustering. Definition: A local group gi in a graph G(V,E) is a subset of vertices Vgi ⊆ V with similar profiles according to a node vi . Furthermore, there is a similarity operator σ, with σ(vi , V ) = Vgi . Graph G consists of |V | local groups, which together build a set, denoted with G. Definition: A decentralized cluster C is the intersection of all local groups gi in a graph G(V,E). A decentralized cluster C is obtained, if σ is applied to all Vgj .
4
Algorithms
In this section the network model is defined and the algorithms for each layer are presented. The used middleware for MPDC is an ad hoc multi agent platform
1112
C. Seitz and M. Berger
which is described in Berger and Watzke [3], where each agent platform hosts an ad hoc management agent that makes itself known to its neighbors by generating a beacon at regular intervals and by listening to signals from other hosts around. 4.1
The ad hoc Network Model
An ad hoc network is generally modelled as an undirected graph G0 = (V0 , E0 ) as depicted in figure 2a. The vertices vi of G0 represent mobile hosts. If the distance between vi and vj is below the transmission range rt , the two vertices are connected by an edge eij . Due to the motion of the vertices, a graph G0 as shown in figure 2a is only a snapshot of an ad hoc network, because in consequence of the mobility of the hosts G0 will change.
a )
b )
Fig. 2. Graph and a possible Spanning tree
Assumptions on the the mobile nodes and network are: – Each mobile device has a permanent, constant unique ID. – The transmission range of all hosts is rt . – Each host knows all its neighbors and its associated IDs. 4.2
Initiator Determination
At first, initiators must be determined who are allowed to send the first messages. Without initiators all hosts start randomly sending messages with the result that the algorithm in the next layer cannot start. We are not in search of one single initiator, we only want to guarantee, that not all hosts start the initiation. There are active and a passive methods to determine an initiator. The active approach starts an election algorithm (see Malpani et al. [9]). These algorithms are rather complex and a lot of messages are sent. They guarantee that only one leader is elected and in case of link failures that another host takes the initiator role. This is not necessary for MPDC, because the initiator is only needed once and it matters little if more than one initiator is present. Therefore, we decided for a passive determination method, which is similar to Gafni and Bertsekas [6]. By applying the passive method no message is sent to determine an initiator. Since each host has an ID and knows all neighbor IDs, we only allow a host being an initiator, if its ID is larger than all IDs of its neighbors. The initiator is in charge of starting the virtual topology algorithm, described in the next section.
Towards an Approach for Mobile Profile Based Distributed Clustering
4.3
1113
Virtual Topology Creation
Having confined the number of initiators, the graph G0 can be covered with a virtual topology (VT). Simulations showed that a spanning tree is a promising approach for a VT and therefore we will only describe the spanning tree VT in this paper. A spanning tree spT(G) is a connected, acyclic subgraph containing all the vertices of the graph G. Graph theory guarantees, that for every G a spT(G) exists. Figure 2b shows a graph with one possible spanning tree. The Algorithm Each host keeps a spanning tree sender list (STSL). The STSL contains the subset of a host’s neighbors belonging to the spanning tree. The initiator, determined in the previous section, sends a create-message furnished with its ID to all its neighbors. If a neighbor receives a create-message for the first time, this messages is forwarded to all neighbors except for the sender of the createmessage. The host adds each receiver to the STSL. If a host receives a message from a host which is already in the STSL, it is removed from the list. To identify a tree, the ID of the initiator is always added to each message. It may occur that a host already belongs to another tree. Under these circumstances the message is not forwarded any more and the corresponding host belongs to two (more are also possible) trees. In order to limit the tree size a hop-counter ch is enclosed to each message and is each time decremented, the message is forwarded. If the counter is equal to zero, the forwarding process stops. By using a hop-counter it may occur that a single host does not belong to any spanning tree, because all tree around are large enough, i. e. ch is reached. The affiliation of that host is not possible, because tree nodes do not send messages in case the hop-counter’s value is zero. When time elapses and a node does notice it does still not belong to a tree, an initiator determination is started by this host. Two cases must be distinguished. In the first one the host is surrounded only by tree nodes, in the other case a group of isolated hosts are existing. In both cases, the isolated host contacts all its neighbors by sending an initmessage, and if a neighbor node already belongs to a tree it answers with a joinmessage. If no non-tree nodes are around, the single node chooses arbitrarily one of the neighbors and joins the tree by sending an join-agree-message, to the other hosts a join-refuse-message is sent. If another isolated host gets the init-message, a init-agree-message is returned and the host sending the initmessage becomes the initiator starts creating a new tree. Evaluation The reason for creating a virtual spanning tree is the reduction of messages needed to reach an agreement. Let n be the number of vertices and let e be the number of edges in a graph G. If no tree is built each messages must be forwarded to all its neighbors, which results in 2e − n + 1 messages. Overlaying a graph with a virtual spanning tree, the number of forwarded messages is reduced to n − 1 plus 2e − n + 1 messages for tree creation. Determining the factor A,
1114
C. Seitz and M. Berger
when a tree becomes more profitable leads us to A = e = 2n, the amortization A results in 3n+1 2n+2 .
2e−n+1 2(e−n+1) .
If on the average
Table 1. Relation of edges and vertices in an arbitrary graph #vertices n 9 19 26 50
#edges e 26 45 71 135
amortization A 1.22 1.33 1.27 1.28
To confirm this formal work, some arbitrary ad hoc networks were investigated (see table 1 ). The amortization value A does never exceed 1.4 (theoretical value), that means if each host sends only two messages to all of its neighbors, less messages are sent then without the spanning tree. In the equation above, the tree maintenance cost are not taken into account. If a new host comes into transmission range or an other host goes away, additional messages must be sent to re-establish the virtual topology. 4.4
Local Grouping: Optimizing the Local View
In this section the subset of neighbor hosts are determined, which initially belong to a host’s local cluster, called a group. The algorithms presented in this and the following section depend on the used profile, which on its part depends on the application. We describe the grouping and clustering algorithms using the taxi sharing application with a very simple profile that only consists of a X- and Yvalue, representing the destination of a person. These points are plotted in figure 3a in a coordinate system, which is the local view of the black point. The grey points are the X, Y -values of the points the black one is able to communicate with. Besides, all hosts are currently located around the origin of the coordinate system, which can always be achieved by rotating and translating the coordinate system. y y
x
x a )
b )
Fig. 3. Local Grouping of a mobile host
Towards an Approach for Mobile Profile Based Distributed Clustering
1115
The algorithm starts by creating an ellipse from the origin of the coordination system to the local’s point destination (Pd ), see the left ellipse in figure 3b. An ellipse was chosen because it is a continuous shape, whereas a rectangle is not. The height (the semi minor half-axis) of the ellipse is in fixed relation to the length of the ellipse. All points inside this oval already belong to the local group. In order to interchange local groups, a more compact shape, e. g. polygon, including all points is desirable. After creating the first oval, the local group has to be enlarged, if other points are still present. Therefore, a new destination point Pd must be found, which acts as new ending point of another oval (see second oval in figure 3b). This point must meet the following requirements: – Pn must not harm the structure of the existing oval group, i. e. no Pn is allowed which result in a cycle, loop or a helix. – To build the local group with as little ovals as possible, Pn must include as many other points as possible. To guarantee the first requirement, the angle between the first oval and the potential next one has to be considered and may not exceed a specified value. In order to satisfy the second requirement, for each point P, which is not yet in an oval, it is calculated, how many other points would be in that new oval if it may become the next Pd and how many points are excluded in becoming further Pd points. The point with the largest difference becomes the new Pd . This polygon has to be merged with the polygon of the first oval, because for the clustering we need one single polygon, that contains the whole group. Merging polygons is a tricky undertaking, because the merged polygon should not be larger than the ovals. But simply connecting two polygons does not meet this requirement. Therefore, additionally virtual profile points Pv must be inserted to reduce the size of a merged polygon. Having determined a local group it is easy to proceed, when a host with a new profile appears. If the new point is inside the polygon, no changes are to be done. If it is outside the polygon and could become a new Pd the polygon is adjusted. In all other cases the point does not become a member of the group. 4.5
Decentralized Clustering: Achieving the Global View
In the previous section each host has identified its neighbor hosts that belong to its local group gi . These local groups must be exchanged and a global cluster has to be achieved. The algorithm presupposes no special initiator role. Each host may start the algorithm and it can even be initiated by more than one host contemporaneously. Initially, each host sends a cluster-message with its group-polygon enclosed to its neighbors which are element of the spanning tree. If a message arrives, the enclosed polygon is taken and it is intersected with it current local view of the host to get a new local view. This new local view is forwarded to all neighbors except for the sender of the received message. If a node has no other outgoing
1116
C. Seitz and M. Berger
edges and the algorithm has not terminated, the message is sent back to the sender. If node receives two messages from different hosts, only one message is forwarded in order to reduce the number of messages. If the algorithm has terminated, each host has the same lokal view, i. e. the global is achieved. A critical point is to determine the termination of the clustering process. The algorithm terminates in at most 2 · dG = 4 · ch steps. If a host receives this amount of messages, the clustering is finished. But, if the tree is smaller or larger than it is supposed to be, waiting until 2 · dG are received is no termination criteria. For that reason, the real hop counter must be enclosed to a cluster message. If isolated points are adopted, ch increases and the new ch value must be announced.
5
Conclusion
In this paper we presented a new ad hoc applications called Mobile Profile based Distributed Clustering (MPDC). Each mobile host is endowed with its user’s profile and while the user walks around clusters are to be found, which are composed of hosts with similar user profiles. The ad hoc network is covered with a virtual topology in order to reduce the number of messages. Each host determines a set of hosts, that belong to its local group. Finally, the local groups are exchanged and a global cluster is achieved. We are simulating MPDC by means of a taxi sharing application and analyzing MPDC with respect to performance issues. But currently, the grouping and clustering depends on the used profile. One major goal will be to find a more generic solution, for other domains with more complex profiles.
References 1. S. Banerjee and S. Khuller. A clustering scheme for hierarchical control in multihop wireless networks. Technical report, Univ. of Maryland at College Park, 2000. 2. S. Basagni. Distributed clustering for ad hoc networks. In Proceedings of the IEEE International Symposium on Parallel Architectures, Algorithms, and Networks (ISPAN), Perth., pages 310–315, 1999. 3. M. Berger, M. Watzke, and H. Helin. Towards a FIPA approach for mobile ad hoc environments. In Proceedings of the 8th International Conference on Intelligence in next generation Networks (ICIN), Bordeaux, 2003. 4. D. Fasulo. An analysis of recent work on clustering algorithms. Technical report, University of Washington, 1999. 5. C. Fraley and A. E. Raftery. How many clusters? Which clustering method? Answers via model-based cluster analysis. The Computer Journal, 41(8):578–588, 1998. 6. E. M. Gafni and D. P. Bertsekas. Distributed algorithms for generating loopfree routes in networks with frequently changing topology. IEEE Transactions on Communications, COM-29(1):11–18, January 1981. 7. E. Kolatch. Clustering algorithms for spatial databases: A survey. Technical report, Department of Computer Science, University of Maryland, College Park, 2001.
Towards an Approach for Mobile Profile Based Distributed Clustering
1117
8. R. Maitra. Clustering massive datasets. In statistical computing at the 1998 joint statistical meetings., 1998. 9. N. Malpani, J. Welch, and N. Vaidya. Leader election algorithms for mobile ad hoc networks. In Proc. of the Fourth Int. Workshop on Discrete Algorithms and Methods for Mobile Computing and Communications, pages 96–103, 2000. 10. G.-C. Roman, Q. Huang, and A. Hazemi. Consistent group membership in ad hoc networks. In Int. Conference on Software Engineering, pages 381–388, 2001.
Simulating Demand-Driven Server and Service Location in Third Generation Mobile Networks Geraldo Robson Mateus, Olga Goussevskaia, and Antonio A.F. Loureiro Department of Computer Science, Federal University of Minas Gerais, Brazil {mateus,olga,loureiro}@dcc.ufmg.br
Abstract. In this work we present a model for a 3G mobile network, emphasizing some of its main characteristics, such as provision of various types of services, including Voice, Video and Web; classification of users according to their mobility behavior in different geographical areas and during different times of day; varying demand generation patterns for each class of user; and exploration of the knowledge about the geographical location of users in order to improve the resource and information distribution over the network. A mobility simulator was implemented to generate a 24-hour demand in a typical metropolitan area and an integerprogramming problem was proposed to optimize the server and service allocation in a dynamic fashion. Since the problem turned out to be NPHard, a heuristic method was developed in order to solve it. The obtained results were near optimal. The proposed approach allows a personalized and demand driven server and service location in 3G networks.
1
Introduction
Third-generation mobile telecommunication systems and technologies are being actively researched worldwide. Due to their ”mass market nature”, they should be designed as high-capacity systems, able to cope with the envisaged overwhelming traffic demands. The influence of mobility on the network performance is expected to be strengthened, mainly due to the huge number of mobile users. Another important characteristic, that is going to be part of 3G mobile systems, is the ability to determine the geographic location of mobile units [6]. This property will originate a whole new class of services that are based on client’s location [9]. Among them are emergency services, traffic information and fleet management. Moreover, the ability to locate mobile units (MUs) will enable more sophisticated network management techniques and more precise resource and information distribution over the network. The growth of the competition in this sector is notable as well, making the ability to reduce system costs, in order to offer more competitive prices, critical in this scenario. A typical mobile network environment consists of mobile hosts, fixed hosts and certain access points. The fixed hosts are all connected to a backbone (e.g., the Internet). MUs usually do not contact them directly, but use hosts closely located as access points to the backbone in order to minimize the distance which has to be bridged by a mobile connection line. Inserting such an environment into H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 1118–1128, 2003. c Springer-Verlag Berlin Heidelberg 2003
Simulating Demand-Driven Server and Service Location
1119
a metropolitan area brings out some difficulties to the problem of where to locate the access points of the network and how to make the association between these points and the MUs. Such difficulties arise due to the variety of users’ behaviors in metropolitan regions, as well as due to the dynamic demand distribution. For example, business regions present a higher concentration of people during working hours, whereas residential regions have quite an opposite behavior. Two questions that arise in this context are: (i) How to locate the servers over a geographic area in such a way that their capacity will not be wasted while the demand is concentrated somewhere else?, and (ii) How to locate the servers in such a way that location-dependent information will be delivered efficiently? In this work we formulate and solve the problem of how to automatically distribute network resources according to a user demand, which may vary with time and space, so that the equipment maintenance and service provision costs are minimized. It means that, given a system composed of a set of servers, a set of services, and a set of mobile units, the attendance of the demand is made by those servers that are most closely located to the mobile users and that have the lowest maintenance cost at that moment. Such subsets of active servers may change from time to time, according to the users’ mobility behavior and changes in demand distribution. The result is a sequence of activations and deactivations of servers so that the demand is satisfied and there is no capacity waste. The proposed approach gives a personalized, demand driven information supply. The problem was treated as follows. A mobility simulator was implemented to represent a hypothetical city, a typical contemporary metropolitan area. The city was divided into geographical zones, such as a City Center and some suburbs, and populated with different groups of MUs, whose mobility behavior, as well as demand for different kinds of services was generated in order to simulate a typical 24-hour day input data for the model. Given the generated demand at a certain period of time, a server allocation was made in order to attend this demand. The allocation was achieved by modeling the system state as an integerprogramming problem and by, afterwards, running an optimization algorithm on it. The output from the optimization process was used to reconfigure the system by activating or deactivating the selected servers and attending the current demand. The resulting integer programming problem turned out to be NP-Hard [5], so a heuristic method was necessary in order to solve it. Lagrangean Relaxation technique [3,10], integrated to Subgradient Optimization [10], was chosen for this task, and the obtained solution was quite close to the optimum. The performance of the proposed approach was measured relatively to what we call a “Common” allocation approach, which was also implemented for the sake of comparison. The Common approach is based on the way server allocation is made in regular cellular systems. The performed experiments demonstrated the system’s sensitivity to different kinds of parameters, such as user mobility, demand distribution over time and space, time of day, as well as the area type where events occur. The results show a significant system cost reduction. The rest of this paper is organized as follows. Section 2 presents the developed mobility simulator. Section 3 contains a detailed description of the optimization
1120
G.R. Mateus, O. Goussevskaia, and A.A.F. Loureiro
process, including the formulated integer programming model, and the heuristic technique applied. Section 4 provides some experimental results and their analysis. Finally, Section 5 discusses our conclusions and future work.
2
Mobility Simulator
Geographical Area. Transportation theory [1] divides the geographic area under study into area zones. The division is based on criteria related to population density and natural limits (e.g., rivers, highways). Considering mobile telecommunication requirements, it seems reasonable to assume that an area zone equals to a network area (e.g., macrocell, local exchange area). The area zones are connected via high-capacity routes, which represent the most frequently selected streets for movement support. It is worth noting that areas outside the city can also be modeled as area zones which attract a relatively low number of trips. Based on the above observations, we chose to model the simulated geographic area as a twenty-kilometer-radius radial city, composed of four area types: city center, urban, suburban, and rural. Figure 1 is a representation of such a city. The model consists of 32 area zones (eight per city area type), four peripheral (one per area type) and four radial high-capacity routes.
50 40 30 % 20 10 0 er s O th lace ces rkp iden Wo R es
Fig. 1. The city area model consisting of area zones connected via high-capacity routes.
Center Urban Suburban Rural
Fig. 2. The distribution of movement attraction points over the city area.
Movement attraction points represent locations that attract population movements and at which people spend considerable time periods. Examples are workplaces, residences, and malls. Each movement attraction point characterizes the population group it attracts. In our simulator, we consider movement attraction points to be residences, workplaces, and other points (e.g., shopping centers, parks). Figure 2 presents the assumed distribution of movement attraction points over the whole city area. Note that within a certain area type (e.g., urban, suburban) the movement attraction points are uniformly distributed.
Simulating Demand-Driven Server and Service Location
1121
Population. The population is divided into mobile unit groups according to the mobility characteristics of the individuals and on the kind of demand they generate [11]. The groups created are 24-hour delivery boy, common worker, housekeeper, and taxi driver. A movement table was associated to each of these groups in order to determine their mobility behavior. This was done dividing the day into time periods, and associating a probability to a group of being at a certain location at a given moment. A call distribution and a set of requested services are also associated to each group. Some examples are: 24h Delivery Boy: Call Distribution: Poisson with mean equal to 10 min; Service: Voice with duration exponentially distributed with mean equal to 80 s. Common Worker : Call Distribution: Poisson with mean equal to 14 min; Services: Voice with duration exponentially distributed with mean equal to 80 s, Web with duration exponentially distributed with mean equal to 180 s, Video with duration normally distributed with mean equal to 1 h. A speed variation is associated to each group. It is assumed that the city has a pre-defined average speed that serves as reference. For example, a delivery boy moves 10 km/h faster that the city average speed, whereas a common worker moves exactly at the average speed. The way MUs move depends on each group movement table, movement attraction point distribution and topology. The choice of a new destination is made by first choosing the destination area (using the group movement table, and the attraction point distribution), and then choosing a specific area zone and a particular coordinate at that area zone. These procedures are performed considering a uniform distributed probability. The route toward the destination is selected using the shortest path between the current and the destination positions as a sequence of radial and peripheral high-capacity route passages. In the beginning of the simulation, all mobile units are assigned a home place and a working location. Home locations are uniformly distributed among those area types with greater residential attraction point concentration, whereas workplaces are chosen inside those area types with greater workplace concentration. Cost definition is a very important part of the simulation process. Costs may vary in both time and space. It means that if we are interested in differentiate servers by their location, their use may be minimized or even avoided. In a similar way, if costs are strictly dependent on a given time period (day, year), their work can be reduced during those periods. Connection (or service attendance) costs can also be differentiated according to various factors. Among them are the physical distance between the server and the mobile unit, the number of channels required by a particular kind of service (Voice: 1 channel, Web: 3 channels, Video: 2 channels), time of the day, or even a priority associated to a particular user.
3
Optimization Process
Modeling. In order to build a mathematical model for a mobile network that supports multi-service provision in a demand driven way, we combined some well-known combinatorial optimization models, such as Incapacitated Location
1122
G.R. Mateus, O. Goussevskaia, and A.A.F. Loureiro
Problem and P-Median Location Problem, which have been widely studied [4,8, 10], and then add some new characteristics to them. The following notation is used: T : set of service classes; I: set of servers; U : set of active mobile units; dtj : binary parameter, expressing whether the mobile unit j ∈ U is demanding a service t ∈ T or not; pt : number of servers to be selected for every service t ∈ T ; ctij : variable location dependent cost of attending the demand dtj by server i ∈ I; fit : cost of activating server i ∈ I to provide service t ∈ T ; xtij : Boolean variable, which is set to 1, when the mobile unit j ∈ U , requiring service t ∈ T , is attended by the facility i ∈ I and set to 0, otherwise; yit :] Boolean variable, which is set to 1, when server i ∈ I is active to provide service t ∈ T , and set to 0, otherwise; The problem is formulated as follows: (M ): min ctij xtij + fit yit (1) t∈T i∈I j∈U
subject to:
xtij
t∈T i∈I
= dtj , ∀j ∈ U , ∀t ∈ T ,
(2)
yit
≥ pt ,
∀t ∈ T ,
(3)
xtij
≤ yit ,
∀t ∈ T , ∀i ∈ I , ∀j ∈ U ,
(4)
i∈I
i∈I
xtij yit
∈ {0, 1} , ∀ t, ∀i ∈ I , ∀j ∈ U ,
(5)
∈ {0, 1} , ∀t ∈ T , ∀i ∈ I.
(6)
The objective function (1) minimizes the total cost of activating the selected servers and attending the active MUs at each period. The set of constraints (2) ensures that each active MU demand for each class of service is attended by a unique server at each period. The set of constraints (3) imposes a minimum number of servers that must be selected for each class of service. The set of constraints (4) guarantees that no MU will be allocated to a non-selected server. Finally, sets (5) and (6) of constraints guarantee the integrality of variables. Solution. As long as the formulated problem turned out to be NP-Hard [5], what we propose in this work is a method to obtain a tight bound, or an approximate solution, to the mixed integer programming problem M described earlier. The developed strategy makes use of the Subgradient Optimization technique [10] to successively maximize the lower bounds obtained for the problem using the Lagrangean Relaxation [2,3,4,10]. A combination of these two techniques allows us to determine the quality of the solution obtained. We have tried several relaxations, and the best results have been obtained by choosing the set of constraints (2), disregarding the fact that for every active mobile unit j ∈ U and for every class of service t ∈ T , the demand dtj must be attended by one and exactly one server. Associating a set of multipliers λtj , ∀j ∈ U, t ∈ T to the relaxed set of constraints (2) and using the same notation as in (M ), the following Lagrangean problem (LP ) is formulated:
Simulating Demand-Driven Server and Service Location
(LP ) :
min
(ctij − λtj )xtij +
t∈T i∈I j∈U
fit yit −
t∈T i∈I
λtj dtj
1123
(7)
t∈T j∈U
Subject to (3), (4), (5) and (6) The lower bound for M is obtained by resolving the LP associated to it using the current values for the Lagrangean multipliers. The following algorithm was used to find this solution: Suppose a facility yit is selected, then: 1 , if (ctij − λtj ) ≤ 0 t xij = (8) 0 , otherwise It follows that the contribution of facility yit in (7) is measured by: βit = fit + min(0, ctij − λtj )
(9)
j∈U
Using the formulation above, the LP can be rewritten as: βit yit + λtj dtj (LP ) : min t∈T i∈I j∈U
(10)
t∈T j∈U
Subject to (3), (4), (5) and (6). The optimal solution to the relaxed problem LP is obtained by the following algorithm: 1. Order the βit parameter in increasing order, ∀i ∈ I , t ∈ T ; t 2. Make 1 if βit ≤ 0 , ∀i ∈ I , t ∈ T ; yi = t 3. If i∈I yi < pt , make yit = 1 in increasing order of beta parameter until the required minimum number pt of servers is selected, ∀t ∈ T ; 4. Make xtij = 1 if yit = 1 and (ctij − λtj ) ≤ 0, and xtij = 0 otherwise. The value of the lower bound ZLB is obtained by replacing the values of the obtained variables x and y in (7). The initial upper bound ZU B = ∞. As far as feasible solutions for the proposed problem are obtained, the upper bound is renovated by the lowest value found up to that moment for the objective function. A solution is obtained from the previous lower bound solution by setting: 1 , if βit is minimal ∀i ∈ I , t ∈ T xtij = (11) 0 , otherwise In this way the set of constraints (2) is satisfied and the resulting solution is feasible. The upper bound ZU P is obtained by replacing the new variables X in the objective function (7). As the lower and upper bounds are determined, their values are used in the subgradient optimization iterative process. The total cost of the procedure is polynomial, and the output is the resultant duality gap (minZU B − maxZLB ) or the optimal solution itself. In any case, a quality guarantee can be associated to the obtained solution.
1124
4
G.R. Mateus, O. Goussevskaia, and A.A.F. Loureiro
Experiments
In order to evaluate the performance of the proposed architecture, three different approaches were simulated, and their performances compared. Common Allocation Approach. It is a simulation of the way demand attendance is performed in today’s traditional cellular networks. The geographic area is divided into cells and usually one server (Radio Base Station) is responsible for service provision to all mobile units physically inside the cell region. Each area zone of the simulated city area was treated as a separate cell. Online Allocation Approach (OnAA). In this work, we propose the Online Allocation Approach as a better alternative to the Common Approach. Its main characteristic is a periodic reconfiguration of the system’s resources (a sequence of activations and deactivations of servers) based on the current demand distribution, and the current location of mobile units. Every fixed amount of time, all information regarding the system’s current state (i.e., which services are being requested, and from which locations) is collected and an integer programming problem is built. The Lagrangean Relaxation technique is then applied to find the closest solution to the optimum that was pre-defined as acceptable. The solution is then used to reconfigure the system’s state, what means the demand is attended with the lowest cost at that instant. While mobile units move across the city area, mobile network resources try to go along with the generated demand. Offline Allocation Approach (OffAA). It is similar to the OnAA, except for the fact that the integer programming problem is built for more than one time instance once and pre-solved off-line. The simulation is repeated twice using the same seed for random sequences. During the first time, all information regarding user mobility and demand generation is collected and periodically saved into a file. The integer programming problem is then built with a small difference–one new dimension, namely time, is added to the problem, and its solution is calculated for all simulated periods. The resulting problem presents a slightly higher computational complexity then the first one. However, because the computation was performed off-line, good results were obtained. During the second time the simulation was run, the pre-calculated solution was used to attend the demand. The purpose of this approach was to be able to compare the overall performance of a real-time system reconfiguration (OnAA) to the performance of an off-line system reconfiguration, under the supposition that user mobility and demand generation could be perfectly predicted. As we shall see in the next section, interesting results were obtained using this approach, what strongly motivates the study of mobility and demand prediction methods in mobile telecommunication systems. Lagrangean Relaxation Performance Evaluation. The parameters that we found interesting to evaluate regarding Lagrangean Relaxation performance are the number of subgradient optimization iterations and the number of optimal solutions obtained. The proximity of non-optimal solutions to the optimum was not analyzed in this work but, according to [7], for all cases analyzed here, the solution is less then 10% from the optimum.
1125
12 O ptim um (Y/N) Num .Iters
10
35
N um .Iterations /O ptim um (yes=1,no=0)
Num .Iterations /O ptim um (yes=1,no=0)
Simulating Demand-Driven Server and Service Location
8 6 4 2 0
O ptim um (Y/N )
30
N um .Iters
25 20 15 10 5
Tim e ofday (hours)
Fig. 3. Lagrangean Relaxation Evaluation for Online Approach (50 MUs).
8. 9
8. 8
8. 6
8. 3
8. 4
8. 1
7. 8
7. 9
7. 6
7. 3
7. 4
7. 1
8. 9
8. 8
8. 6
8. 4
8. 3
8. 1
7. 9
7. 8
7. 6
7. 4
7. 3
7. 1
0
Tim e ofday (hours)
Fig. 4. Lagrangean Relaxation Evaluation for OnlineApproach (200 MUs).
Figures 3 and 4 show the number of iterations of the subgradient optimization loop performed to obtain solutions to the integer programming problem M during the period from 7:00 am to 9:00 am. For 50 MUs, in all instances, optimal solutions were obtained in less than 20 iterations. For 200 MUs, however, we can observe one case, near 8.48 am (8.8 h), for which the optimum was not found in 300 iterations (the value of 30, in the figure, was put not to distort the proportion of other values). In this case an approximate solution was used. To measure the performance of Lagrangean Relaxation for the OffAA, experiments considering 50, 100, 200 and 500 users were performed. For less than 500 MUs, the optimum solution is found in less than 30 iterations. For 500 users, however, only an approximate solution could be achieved. System Costs. The evaluation of system costs was performed through the values of the objective function of the integer programming problem. It means that all costs involved in server activation (fit ), as well as all costs involved in demand attendance (ctij ), were taken into account. Figures 5 to 8 show a comparison between the costs of the three server allocation approaches. Results for different numbers of simulated mobile units, and different cost proportions are analyzed. In Figures 5 and 6 variable costs (ctij ) are, in average, 10 times higher then fixed costs (fit ). This proportion makes the system configuration more sensible to the distances between mobile units and servers. In Figures 7 and 8, on the other hand, fixed costs are, in average, 10 times higher then variable costs, what makes the system more sensible to server activation/maintenance costs. In all four figures, the predominance of Common Approach costs over other approaches, and the predominance of Online over Offline Approach costs, can be seen. This result could be predicted since the Common Approach does not consider any cost evaluation in its demand attendance process, whereas Online and Offline approaches do so, with the following difference between them: the Online Approach optimizes system costs based on one period information about the system’s state, and does it in real time. The Offline Approach has a “broader” view of the system’s state, as long as it operates over several periods of time at once, what allows it to do a better resource allocation, achieving lower system
1126
G.R. Mateus, O. Goussevskaia, and A.A.F. Loureiro
2000 6000
1500
5000 4000
1000
3000
Online Obj.Fun Com.Obj.Fun Offline Obj.Fun
500 0 7,4
7,5
7,6
7,7
7,75
7,8
2000
Offline Obj.Fun Com.Obj.Fun Online Obj.Fun
1000 0 7,4
7,5
Time of Day (hours)
7,6
7,7
7,75
7,8
Time of day (hours)
Fig. 5. System costs (50 users, variable Fig. 6. System costs (200 users, variable cost predomination). cost predomination). 80000 70000
160000
60000
140000 120000
50000 Online Obj.Fun Com.Obj.Fun Offline Obj.Fun
40000 30000 20000
100000 80000
Online Obj.Fun
60000 40000
Offline Obj.Fun
Com.Obj.Fun
20000 0
10000 0 7,4
7,5
7,6
7,7
Time of Day (hours)
7,75
7,8
7,4
7,5
7,6
7,7
7,75
7,8
Tim e of Day (hours)
Fig. 7. System costs (50 users, fixed cost Fig. 8. System costs (200 users, fixed cost predomination). predomination).
costs. It is worth noting that such allocation approach could be applied in a real system only if a very powerful prediction mechanism was available. Comparing Figures 5 and 6, it can be seen that, as the number of MUs grows, the relation among costs achieved by the three approaches remains and becomes even more explicit. The same fact can be observed comparing Figures 7 and 8. One more aspect to be analyzed is the influence of cost proportion over the results. When fixed costs are expressively lower then variable costs, it is expected that more facilities are selected, in order to minimize the distances between servers and users. On the other hand, when the proportion is inverted, a minimum number of facilities is going to be selected and, if capacity is not treated, a concentration of users is going to occur on those servers of lowest maintenance cost. This fact is reflected in Figures 7 and 8, which present a much more expressive difference between the optimization and non-optimization approaches. What happens in these cases is a selection of a very low number of servers, more likely in the suburban areas, what makes the overall system costs become much lower. However, this kind of solution is not very realistic, since capacity issues are bound to arise in this scenario. The purpose of these tests, though, is to show how sensitive the proposed architecture is to such parameters.
Simulating Demand-Driven Server and Service Location
5
1127
Conclusions and Future Work
In this work we have formulated and solved the problem of how to dynamically distribute network resources in third-generation mobile telecommunication systems, aiming to minimize the equipment maintenance and service provision costs. A mobility simulator was implemented in order to represent a hypothetical city, which was populated with different groups of users, whose mobility behavior and demand for different kinds of services was generated in order to simulate a typical 24-hour day behavior. Given the generated demand at a certain period of time, a server allocation was made in order to attend it. This was done by modeling the system as an integer-programming problem and by, afterwards, running an optimization algorithm on it. The output from the optimization process was used to reconfigure the system by activating the selected servers and deactivating the others. The problem turned out to be NP-Hard, so a heuristic method was necessary in order to solve it. We chose the Lagrangean Relaxation technique to do the task, and obtained good results in terms of the solution proximity to the optimum. The performed experiments demonstrated the system’s sensitivity to different kinds of parameters, such as user mobility, demand distribution over time and over space, time of day, type of area where the events occur, and the relation between fixed and variable costs of the system. A significant system cost reduction was also achieved. One of the limitations of the proposed model is the lack of capacity treatment, which is something we intend to improve. Some new constraints can be added to the proposed model in order to make it able to cope with capacity issues. Equations limiting the number of mobile units attended by each server and the radius covered by them are going to be elaborated. Some new features will probably be necessary in the developed mobility simulator in order to adapt it to this new scenario.
References 1. K.G. Abakoukmin. Design of transportation systems. 1986. 2. M.L. Fisher. The lagrangean relaxation for solving integer programming problems. Management Science, 27:1–18, 1981. 3. M.L. Fisher. An application oriented guide to lagrangean relaxation. Interfaces, 15(2):10–21, 1985. 4. R.D. Galv˜ ao and E.R. Santiba˜ nez-Gonzales. A lagrangean heuristic for the pk median dynamic location problem. European Journal of Operational Research, 58:250–262, 1992. 5. M.R. Garey and D.S. Johnson. Computers and Intractability: A Guide to the Theory of NP-Completeness. Freeman W.H. and Company, 1979. 6. J. Hightower and G. Borriello. A survey and taxonomy of location systems for ubiquitous computing. Univ. of Washington, Comp. Sci. and Eng., 2001.
1128
G.R. Mateus, O. Goussevskaia, and A.A.F. Loureiro
7. G.R. Mateus, O. Goussevskaia, and A.A.F. Loureiro. The server and service location problem in the third generation mobile communication systems: Modelling and algorithms. Workshop on Mobile Internet Based Services and Information Logistics, Austria, pages 242–247, 2001. 8. P.B. Mirchandani and R.L. Francis. Discrete Location Theory. Wiley, 1990. 9. Prasad Ranrjee. Overview of wireless personal communications: Microwave perspectives. IEEE Communications Magazine, pages 104–108, 1997. 10. C.R. Reeves. Modern Heuristic Techniques for Combinatorial Problems. Blackwell, 1993. 11. M.N. Rocha, G.R.Mateus, and S.L. Silva. Traffic simulation and the location of mobile units in wireless communication systems. First Latin American Network Operations and Management Symposium, pages 309–319, 1999.
Designing Mobile Games for a Challenging Experience of the Urban Heritage Francesco Bellotti, Riccardo Berta, Alessandro De Gloria, Edmondo Ferretti, and Massimiliano Margarone 1
DIBE – Dep.t of Biophysical and Electronic Engineering of the University of Genoa, Via dell’Opera Pia 11/a, 16145 Genova, Italy {franz, berta, adg, ed, marga}@dibe.unige.it http://www.elios.dibe.unige.it/
Abstract. Ubiquitous gaming is a new emerging research area, that is gaining focus from industry and academy as smart wireless devices are becoming ever more widespread. We have explored this field implementing a treasure hunt game aimed at enhancing the visitor’s experience of the heritage in a city area, such as Venice. The game relies on standard, commercially available hardware and middleware systems. The paper describes the pedagogical principles that inspire the game and the corresponding structure of VeGame. We also discuss how combining mobile gaming and an educational focus may deliver to the player a challenging and engaging experience of the heritage, as shown by early user tests. Extensive qualitative and quantitative evaluation will start in June 2003.
1
Introduction
Mobile computing can support the user in the exact moment of the requirement – during her/his normal daily-life activities –, without the need for relying on a wired desktop computer. Our research group has explored such issues in the fields of tourism and culture, designing and developing multimedia mobile guides on palmtop computers for tourists [1, 2]. In this context, mobile services support visitors in the most important moment of their education experience: when they are up close to the subject, whether they’re viewing a painting in a museum, a monument in a park, or an animal in the zoo. We are now interested in developing further this research field, exploring how mobile gaming can help tourists enhance their experience of art and history, favoring a pleasant and challenging interaction with documents and artifacts in a real urban context. 1.1
Designing Games to Support Acquisition of Knowledge
In this article we describe VeGame (Venice Game, Fig. 1) – a team-game played along the Venice’s narrow streets to discover the art and the history of the city – and H. Kosch, L. Böszörményi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 1129–1136, 2003. © Springer-Verlag Berlin Heidelberg 2003
1130
F. Bellotti et al.
present design choices made by the interdisciplinary project team in an iterative process exploiting feedback from early usability tests. VeGame has been designed and developed by the ELIOS lab (Electronics for the InfOrmation Society) of the University of Genoa in collaboration with the Future Center of Telecom Italia Lab of Venice.
Fig. 1. VeGame’s initial page. The game uses a PocketPC, with Human-Computer Interaction software developed by the ELIOS lab of the University of Genoa.
The focus of our work is on the end-user [3] and, thus, in content and HumanComputer Interaction (HCI) design: how mobile services can deliver a useful and pleasant experience to the tourist, who is a special kind of general user - typically computer-illiterate - and cannot dedicate much mental effort to learn interaction with the computer. Her/his attention, in fact, is mainly dedicated to the surrounding environment (e.g. a picture, a church, a palace) which she/he is visiting. Thus, the main contribution of this paper consists in presenting our design experience and discussing how mobile games may improve the quality of the tourist visit, by providing a challenging and compelling experience of the surrounding heritage and by favoring development and acquisition of knowledge directly in the moment of the visit. We are thus interested in exploring pedagogic principles able to support knowledge acquisition in the field and studying how technology can systematically exploit them [4]. In particular, we focus on constructivistic learning philosophies [5, 6], which rely on construction of knowledge by situating cognitive experiences in authentic activities [7], and are the current subject of the ubiquitous computing research community [4,8]. According to these principles, VeGame features three main learning modalities: Learn by doing (or playing) , inviting players to operate in the field, exploring the territory and observing attentively the documents, such as pictures, palaces, bridges, etc. Learn by thinking, stimulating reflection on what has been seen and told, critical reasoning and concatenation of experiences and clues. Learn through social interaction, inviting players – both from the same and other teams – to play together for cooperative purposes and to comment on their
Designing Mobile Games for a Challenging Experience of the Urban Heritage
1131
experience during the game. Moreover, we suggest players to interact with local citizens to answer specific questions on the local dialect and activities. 1.2
Challenges
Designing an edutainment experience in a city of art is a complex and challenging task, presenting issues that involve various technical and humanistic fields such as computer science, HCI, multimedia content design, game design, art, history, psychology and pedagogy. In order to face these issues we built a multidisciplinary team which has worked on the project since the early stages of understandings of settings, user needs capture, specification definition and feasibility study. Defining the specifications, the design team analyzed several issues and questions including how to deliver a convincing and compelling edutainment experience (i.e. entertaining and valid from an educational point of view), stimulate and enhance interaction of visitors with the artifacts and other elements of heritage available in the territory, deliver contents from various fields (e.g. sculpture, painting, history, geography, poetry, architecture, local dialect, religion, etc.), favor social interaction of players among them and with other tourists and citizens. In particular, we have analyzed how mobile computing technology can systematically contribute to achieve these targets.
2
VeGame
Relying on the above mentioned pedagogic principles, VeGame fosters an active exploration of the territory (streets, channels, squares, churches, palaces, etc.), its people and their activities (e.g. the famous Murano’s glaziers), in order to have a better understanding and appreciation of the city’s cultural heritage. The educational experience is delivered through a game frame, since we consider it particularly suited to capture and appeal a wide audience, and allows us to exploit specific aspects of computer games able to support education and knowledge acquisition in a constructivistic way [9]. Players are grouped in teams – every team is assigned a HP IPAQ loaded with the VeGame software. The suggested team size is limited to 2 players, so that every person can actively participate to the games, in particular considering the limited screen size of palmtop computers. The game can be played by several teams simultaneously (e.g. on adequately advertised special events, on school trips, etc.), or independently by tourist who want to visit Venice. VeGame challenges players and stimulates competition by assigning score as a reward for the players’ activity. There is a session standings (for special events) and a hall of fame for all participants. We have designed a game structure which closely resembles a treasure hunt game. The participating teams are invited to go through a sequence of stages, which are points of interest distributed in the city, such as important churches, palaces and squares (e.g. Fig. 2a).
1132
F. Bellotti et al.
(a)
(b)
Fig. 2. (a) The map of the urban area covered by VeGame, showing the stages and the path done until current stage, represented as a solid line. (b) The stage-menu with 5 games (in this case: HistoricalQuiz, RightPlace, VisualQuiz, Couples and VideoGame) selectable by the player.
At each stage, the player is presented with a stage-menu, which leads to the five games available at that stage (Fig. 2b). The games are instances of 13 predefined typologies. The game typologies were designed by our multidisciplinary team (partially inspired by common real-world games), and then refined according to early tests with users and experts. We broadly divide games in three categories, according to the cognitive skills they mostly involve: observation games, reflection games, and action videogames. Observation games. These games privilege the sight as a sense to investigate and explore the surrounding environment. In general, these games tend to exploit the “knowledge in the world” in order to develop the cognition activity [7, 10]. They aim to stimulate spatial processing skills. Such skills are important in cognitive development since they allow an individual to create meaning by manipulating visual images [7,8]. Reflection games. These games tend to favor reflection, discussion among team members, analysis of questions and possible answers considering clues available in the neighborhood and concepts learned previously during the game. Action videogames. These games stimulate similar skills as observation games. Their specificity lies in the animated graphics and engaging interaction, which helps to create a convincing and pleasant experience. They stimulate fantasy and evoke images and atmospheres that can be used to convey educational messages which are easily memorized by players. We will now describe 3 different kinds of games (one per each of the above mentioned categories), showing our design choices and implementations and briefly discussing how technology can be exploited to support knowledge acquisition in-the-field by the general public and in a pleasant and entertaining fashion. 2.1
RightPlace
This game involves dragging some objects - represented by small icons – onto their right position on a map (Fig. 3a). Samples of objects can be details of a picture, prod-
Designing Mobile Games for a Challenging Experience of the Urban Heritage
(a)
(b)
1133
(c)
Fig. 3. Sample games. (a) RightPlace: put the details in the right place on the picture. The picture represents the XVth century trptych of Bartolomeo Vivarini at Santa Maria Formosa. (b) HistoricalQuiz: What is the name of the store palaces that used to host the communities of foreign merchants in Venice? (c) Casanova. The lady is trying to grab the rose cast by Casanova.
ucts, names of cities, etc. The map could be a real geographic map – for instance, the player is asked to associate the names of XIV century’s Venetian naval basis to their right position in the Mediterranean area – or a picture, a palace’s façade, etc. Depending on the actual implementations, the RightPlace game typology solicits different types of skills. In general, it aims at supporting creation of a mental model of the surrounding environment (e.g. the neighborhood of a church or of a square), possibly favoring visualization and localization of historical events and situations (e.g. historical maps), and stimulates the ability to understand a map and find correspondences between its parts. In the case of pictures and other elements of the heritage, RightPlace favors an attentive analysis of the details and the relationships among them and with the bigger picture (e.g. a painting, a façade, etc.). 2.2
HistoricalQuiz
HistoricalQuiz is typically a question concerning the story of Venice (Fig. 3b). The question is generally tied with the place where it is made. For instance, the question at the Arsenal concerns interpretation of a Dante’s piece of the Divina Commedia where the work at the Venice’s Arsenal is described. This game tends to stimulate critical reasononing and evaluation of alternatives (the HistoricalQuiz is often a multiple choice question). Typically, the quiz does not rely on previous knowledge, but it stimulates reasoning on clues available in the territory and/or on concepts addressed in the introduction and/or other previous games. The first user tests have shown that players appreciate this approach since it does not rely on scholastic knowledge, and rewards the actual field-activity of the tourist.
1134
2.3
F. Bellotti et al.
Casanova
This game can be played by two teams connected through a real-time Bluetooth channel. One team is Casanova - the famous libertine gentleman of the XVIII century – who has to collect roses and throw them to his lover. The second team plays the lady’s role, who lies in a gondola and has to catch the roses (Fig. 3c).In the single player version (in case there is no team available in the neighborhood) the lady’s role is played by the computer. The Casanova game evokes the customs and the atmosphere of Venice in the XVIII century. In general, videogames are important to create interest and deliver images useful to represent meaning and content which can be recalled with ease. From early tests with users and experts we discovered that it is important to complement such images with explicit verbal knowledge in order to convey an educational message clearly understandable by the players. For instance, Casanova evokes the XVIII century Venice’s atmosphere, which is really different from XIII - XVI centuries - introduced in the previous stage at the Arsenal - when Venice was a marine super-power. The Casanova game’s conclusion text explicates the concept by saying that Casanova’s Venice was important from the cultural viewpoint, but had limited international power. Casanova explicitly fosters cooperation between teams: the total score, in fact, is shared among the two players. This favors interaction between teams and fun, since team players typically communicate among each other in order to help the partner to take the right decision on when to cast the rose or how to grab it. We observed in field tests that Casanova – as the other videogames – is played by just one player at a time, for every team. Anyway, the others are not excluded: they gather around the team-mate supporting her/him and giving suggestions. 2.4
Inter-team Communication
According to the educational principles of VeGame – to acquire cultural knowledge and have an entertaining experience of the heritage – inter-team communication should not disturb the player visitors from their main educational task. Thus, interteam communication involves (i) simple two-player games, where two teams have to meet in a physical space (the Bluetooth range is about 10 meters) and agree to play together, and (ii) a chat and messaging system – managed by a server connected via the GPRS cellular network. These two systems do not require real-time interaction and are not invasive nor strictly necessary to complete the game. But they support an important type of interaction: teams can freely help each other, exchange opinions to improve their field experience, comment on their activity, exchange personal information, and use these communication channels with extreme freedom.
Designing Mobile Games for a Challenging Experience of the Urban Heritage
3
1135
Related Works
Recent research has combined wireless and ubiquitous computing technologies to implement a transparent linkage between the physical world and the resources available on the web [11, 12], enhancing the quantity and quality of information available to visitors and field workers. In fact, this model has been successfully implemented to enhance the tourist’s experience in contexts such as museums and parks [1, 12, 13]. Recent studies have also highlighted the suitability of handheld devices for education in particular for children from the ages 5 to 18 [14, 15]. In parallel, Ubiquitous Computing Game (UCG) research [16] has implemented games which incorporate live data from the real world and process players’ location and movement. Our work does not involve innovative sensing technologies. Instead, we rely on standard, commercial hardware and middleware systems to study how mobile gaming can enhance the visitor’s experience and support field learning. The UCG research also involves games with have an educational focus. For instance, Environmental Detectives [17] implements scenarios – ranging from airborne particles to radioactive spills – which allow educators to investigate a wide range of environmental issues, leveraging their pedagogical and entertainment potential. Inspiring to tour guides, VeGame features an in-depth linkage of the game with the territory - which is an indispensable context, rich of local information and people. VeGame exploits mobile computing technologies in order to provide users two major added-values: the possibility of providing an enjoying experience – similar to that of a compelling videogame - and the support to explore autonomously the territory in order to accomplish challenging tasks. These factors are important to support learning, in particular attracting a category of public which is less prone to cultural activities.
4
Conclusions
In the VeGame project we are exploring how mobile technologies can enable new services and applications in nontraditional settings to enhance people’s education and understanding of their cultural heritage. Despite the proposed games typologies are relatively simple – they are to be understood and played with ease by the general public - we believe that these games are good samples of ubiquitous computing games. They exploit the computer as a constant presence [18] that accompanies the user as a friendly travel-mate, favoring cognition development, and stimulating interaction with the real-world, in particular with the heritage, the other players and the local people (e.g. the Dialect game makes questions about the meaning of Venetian words and phrases). While we still miss quantitative data from extensive field user tests - they will start in June 2003 – it is important that our initial field tests show an enthusiastic user acceptance, essentially because the system is able to assist the visit privileging cognitive approaches based on field observation, social interaction and fun, which give the visitor a challenging and engaging experience of the heritage.
1136
F. Bellotti et al.
Acknowledgments. We would like to thank the Telecom Italia Lab, the Future Center, and, in particular, Roberto Saracco and Fabio Carati for their precious contribution to VeGame project. We also thank the anonymous reviewers for their insightful and constructive comments, which have helped us to improve the quality and readability of the paper.
References 1. 2.
3. 4.
5. 6. 7. 8. 9. 10. 11. 12. 13.
14. 15. 16. 17.
18.
Bellotti F., Berta R., De Gloria A., and Margarone M., User Testing a Hypermedia Tour Guide, IEEE Pervasive Computing, Volume 1, Issue 2, April-June 2002. Bellotti F., Berta R., De Gloria A., Gabrieli A. and Margarone M., E-Tour: Multimedia Mobile Guides to Enhance Fruition of the Heritage, in E-work and E-commerce, ed. Brian Stanford-Smith and Enrica Chiozza, IOS Press, 2001. Weiser M., The computer for the 21st century, Scientific Am., September 1991, pp. 94104. Mühlhäuser, M., Trompler, Ch.: Learning in the Digital Age: Paving a Smooth Path with Digital Lecture Halls. Proc. Hicss 35th Hawaii Intl. Conference, Waikola, HI, Jan 7–12, 2001. IEEE CS press, Los Alamitos, CA, 2001. T.M. Duffy and D. H. Jonassen, Constructivism and the Technology of Instruction: A Conversation. Lawrence Erlbaum, Hillsdale, New Jersey, 1992. B. Wilson (Ed.): Constructivist Learning Environments: Case Studies in Instructional Design. Englewood Cliffs, New Jersey. Educational Technology Publications, Inc., 1996. Gardener H., Multiple intelligences, the theory in practice. N.Y.: Basic Books, 1993. Abowd G. D., Mynatt E. D., and Rodden T., The Human Experience, IEEE Pervasive Computing, January-March 2002, Vol. 1, No. 1, pp. 48–57. Natale M. J., The Effect of a Male-Oriented Computer Gaming Culture on Careers in The Computer Industry, Computers and Society, June 2002, pp. 24–31. Hutchins E., Cognition in the Wild, MIT Press, Cambridge, Mass., 1995. S. Pradhan, C. Brignone, J-H. Cui, A. McReynolds, M. T. Smith, “Websigns: Hypelinking Physical Locations to the Web”, IEEE Computer, August 2001, pp. 42–48. N. Davies, K. Cheverst, K. Mitchell, A. Efrat, “Using and Determining Location in a Context-Sensitive Tour Guide”, IEEE Computer, August 2001, pp. 35–41. F. Kusunoku, “Toward an Interactive Museun Guide System with Sensing and Wireless Network Technologies”, IEEE Int.l Workshop on Wireless and Mobile Technologies in Education (WMTE), August 2002, Vaexjoe, Sweden. E. Soloway, C. Norris, P. Blumenfeld, B. Fishman, J. Krajcik, and R. Marx, Handheld Devices are Ready-at-Hand, Comm. of the ACM, June 2001, Vol. 44, No. 6, pp. 15–20. C. Norris, E. Soloway, and T. Sullivan, Examining 25 Years of Technology in Education, Communications of the ACM, August 2002, Vol. 45, No. 8, pp. 15–18. S. Björk, J. Holopainen, P. Ljungstrand, and R. Mandryk, Editors, Special Issue on “Ubiquitous Gaming”, Personal and Ubiquitous Computing, December 2002, Vol. 6. E. Klopfer, K. Squire, H. Jenkins, “Environmental Detectives: PDAs as a Window into a Virtual Simulated World“, IEEE Int.l Workshop on Wireless and Mobile Technologies in Education (WMTE), August 2002, Vaexjoe, Sweden. Abowd G. D., Mynatt E. D., Charting Past, Present, and Future Research in Ubiquitous Computing, ACM Transaction on Computer-Human Interaction, Vol. 7, No. 1, March 2000, pp. 29–58.
QoS Provision in IP Based Mobile Networks Vilmos Simon, Árpád Huszák, Sándor Szabó, and Sándor Imre Budapest University of Technology and Economics, Department of Telecommunications Mobile Communications and Computing Laboratory Magyar Tudósok krt.2, H-1117. Budapest, HUNGARY
[email protected],
[email protected] Abstract. While IP is declared as the key technology of the future’s wired and mobile communication, the currently used version of IP, IPv4 itself is not suitable to be used in mobile scenarios. Next generation mobile users require special support to provide connectivity, although they change their place of attachment to the network frequently [1]. In our work, we have created a network design algorithm and an agent (GMA/MAP) router selection algorithm in Regional Registration and Hierarchical Mobile IPv6 to optimise the handover management in IP based next generation mobile networks. Our research is supported by ETIK (Inter-University Center for Telecomunications and Informatics).
1 Introduction Reduced radio cell sizes (increasing the number of handovers) and more complex protocols and services in next generation mobile networks increase the signalling overhead, causing significant signalling delay. This is critical in the case of timingsensitive real-time media applications that call for mobile QoS [2]. Mobile IPv6 [3] is not capable of supporting real-time handovers. A solution is to make Mobile IPv6 responsible for macro-mobility, and to have a separate protocol to manage local handovers inside micro-mobility domains. Other solutions are Hierarchical Mobile IPv6 [4], or Regional Registration [5]. The basic idea of these hierarchical approaches is to use domains organised in the hierarchical architecture with a mobility agent on the top of the domain hierarchy. The standard does not address the realisation considerations in detail of the hierarchical tree during the network design. The implementation details of the hierarchy are entrusted to the engineer. In our work we present a method, showing how to configure these hierarchy levels in order to reduce signalling traffic. We propose a novel graphtheory algorithm, which takes into consideration the mobile node’s mobility model.
2 Protocol Optimisation 2.1 The Domain Forming Algorithm The main concept of the algorithm is to connect adjacent radio cells with high mutual handover probabilities to the same access router. As a result, most handovers will take place between base stations belonging to the same access router, so the number of care-of H. Kosch, L. Böszörményi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 1137–1140, 2003. © Springer-Verlag Berlin Heidelberg 2003
1138
V. Simon et al.
address changes – and the amount of related signalling messages, handover delays, etc. – are reduced. The algorithm is described mathematically as follows: in a graph-modelled network, the cells are the nodes of the graph ( C ) and the possible directions of the cell
c1 and c2 are adjacent points (cells), if {c1 , c2 }∈ E . We must divide the G = (C , E ) graph into G ’= (C ’, E ’) subgraphs, so that the subgraphs contain the maximal weight spanning boundary crossing between adjacent cells will be the edges ( E ).
tree. We define weights to the graph’s edges, not negative real numbers in the range [0,1], based on the direction probability vector (the handover vector), equivalent to the probabilities of the movement directions. We choose the edges one by one in the following manner: first we choose one of the edges with the biggest weight: emax = p max . The two nodes (cell), which are connected
{c1 , c2 }∈ emax , can be joined, and after that we manage them as if they are one, C ’= {c1 , c2 }. From the six edges outgoing from the two joined nodes, we must choose one with the biggest weight value, max(E ’) , and the belonging node becomes the member of the common set, C ’= {c1 , c2 , c3 }. From the eight edges belonging to the to this edge,
nodes, which are in the common set, we choose the next one so as to make a circle in the graph, namely if we have an (c1 , e1 , c2 , e2 ,..., c k , ek , c k +1 ) edge series, then c1 = c k +1 . In this way, we can avoid the domains becoming too entangled and far-reaching. We can continue this algorithm, until the element number of C ’ reaches M , N C ’ ≤ M , , so
( )
one domain will consist of M cells. When we cover the entire G graph with notconnected
G ’1 , G ’2 ,...G N subgraphs, the algorithm of forming domains is finished.
2.2 Relation between the Mobile Node’s Speed and the Hierarchical Levels There are several potential GMA (Gateway Mobility Agent) routers inside a HMIPv6 capable domain. The MN chooses one GMA from the list attached to the Router Advertisement. In the list there are Regional Care-of-Addresses . While the MN is moving in the domain area, it can choose another GMA router. Every GMA change must be reported to the Home Agent. For the best result the selection of the GMA must depend on the movement speed of the mobile node. If the MN often changes its Access Router, a high level GMA must be chosen while a GMA on low hierarchical levels advertises Binding Updates too often. For a slow moving MN the situation is the opposite. It is advisable to choose a GMA near the Access Router. Because of the slow moving, the change of the Regional CoA is rare, and the incoming packets get to the MN in a shorter way. We give an algorithm, which selects the optimal Regional CoA (GMA) from the list. It is noticeable that increasing the movement speed, the probability of the GMA changing is increasing too, if the MN selects the GMA randomly. The effect is the same, if the number of the hierarchy levels is increasing. Using the algorithm, in a router hierarchy, which has three levels, up to 50% gain can be achieved (section 3.2.).
QoS Provision in IP Based Mobile Networks
1139
3 Analytical Examination 3.1 Analytical Examination of Domain Forming Algorithm We have examined analytically, how the number of signalling messages changes using a randomly formed lowest hierarchy level and the one, which is formed by our algorithm, to see if the algorithm really decreases the administrative overload generated by the handover. In our examination we have modelled the cover of a busy avenue and its surroundings. We adjusted probabilities to the handover directions, modelling a realistic traffic on the given system of roads. The probabilities gave us the information in which direction (north/south/west/east) and with what relative frequency the mobile users will cross the cell boundary, based on online measurements. The required bandwidth, if the mobile node changes a domain, is:
B = K ⋅ N ⋅ L + 2 ⋅ L = L ⋅ ( K ⋅ N + 2),
(1)
where N is the number of the foreign Correspondent Nodes, L is the size of the Binding Update message, and K is the number of Binding Updates which was sent to one foreign Correspondent Node. In the first case, we made a random arrangement of four cell groups and we calculated how many domain handovers it causes when the mobile node is travelling on significant routes on our system of roads. One change of domain requires K ⋅ N + 2 number of Binding Updates, and the bandwidth is defined by (1). In the second case we used our domain-forming algorithm to see whether it really reduces the number of domain changes for the same significant routes. The results are given in Table I.. Table 1. Number of domain changes
Route (a) (b)
Random
Algorithmic
5⋅L⋅(K⋅N+2) 3⋅L⋅(K⋅N+2)
3⋅L⋅(K⋅N+2) 2⋅L⋅(K⋅N+2)
Based on the results obtained with the help of the domain forming algorithm the handover signalling load can be reduced by 30-40% on average (see Table I.), which makes it possible to improve the QoS parameters (for example in real time applications). The domain changing update traffic was calculated for other significant routes, too. 3.2 Analytical Examination of GMA/MAP Selection We have examined how the number of re-registrations at the Home Agent changes using a random GMA/MAP selection and our selection algorithm. For simplicity in the three-layer hierarchy network, all of the routers are GMA/MAP capable routers. In this network, the MN can always choose from three different GMA/MAP routers. In this case, the MN receives a Regional Care-of-Address Extension message. In this message there are three different GMA/MAP router addresses. In this sample the whole speed interval is partitioned in four smaller intervals (va, vb, vc, vd). If we know in which interval the speed-rate is, we can choose the optimal
1140
V. Simon et al.
GMA/MAP router. We have calculated the probability of GMA/MAP changes in case of random GMA/MAP selection. As the results show, the probabilities that the MN changes its GMA/MAP router once, twice, and so on are given in Table II. Table 2. Probability of GMA changes in random selection GMA change 1 2 3 4 Σ M(ξ)
va
vb
vc
vd
1 1 1
2/3 1/3 1 1,33
1/3 4/9 2/9 1 1,88
1/3 1/3 7/27 2/27 1 2,07
The expected number of GMA/MAP changes can be calculated with (2) using the results from Table II: n
M (ξ ) = ∑ pi ⋅ xi
(2)
i =1
It is noticeable that in case of random agent selection, as the speed of the terminals increases (or when the hierarchical tree has more levels), the number of GMA/MAP changes also rises. As a result, the amount of signalling overhead is increasing significantly compared to the optimised scenario.
4 Conclusions Comparing the signalling traffic load of a random network and the optimised one, the results show a significant reduce of the signalling traffic. An agent selection algorithm is also proposed, and its advantages are studied in comparison with a random agent choosing method. These results help to support global QoS in next generation networks. Our future plan is to use computer simulations (like NS2 or OmNet++) to analyse the algorithms, which will help us to get more results. Even if further studies necessary, next generation IP based mobile networks will surely benefit of the enhanced IP mobility management solutions.
References [1] I.F. Akyildiz, J. McNair, J. Ho, H. Uzunalioglu, W. Wang, ”Mobility Management in Current and Future Communications Networks”, IEEE Network, July/August 1998 [2] Hemant Chaskar, ” Requirements of a QoS Solution for Mobile IP”, draft-ietf-mobileipqos-requirements-02.txt, 10 February 2002 [3] S. Deering, R. Hinden, ”Internet Protocol, Version 6 Specification”, RFC 2460, 1998 [4] Claude Casteluccia, K.Malki, Hesham Soliman, Ludovic Bellier, ”Hierarchical Mobile MIPv6 mobility management”, draft-ietf-mobileip-hmipv6-05.txt, 2001 [5] J. Malinen, C. Perkins, ”Mobile IPv6 Regional Registrations”, draft-malinen-mobileipregreg6-00.txt
Design of a Management System for Wireless Home Area Networking 1
1
1
Tapio Rantanen , Janne Sikiö , Marko Hännikäinen , Timo Vanhatupa 2 1 Olavi Karasti , and Timo Hämäläinen 1
1
Tampere University of Technology Institute of Digital and Computer Systems Korkeakoulunkatu 1, FIN-33720 Tampere, Finland {tapio.a.rantanen, marko.hannikainen}@tut.fi 2 Elisa Corporation, POB 81, FIN-33211 Tampere, FINLAND
[email protected] Abstract. A management system for home area networks has been developed. The main design targets have been to support heterogeneous network technologies, automate the network configuration and management, and to enable application based Quality of Service support. The resulting system architecture has a five-layer functional architecture and a centralised topology with a management server. Also, the support of new proprietary management functionality has been designed for wireless access points and client terminals. A wireless LAN and Java based prototype of the management system is being implemented and its architecture presented.
1 Introduction Both the number and variety of home applications and projected network technologies for home environment are growing. While office type of data transfer is still required, interactive gaming and consumer electronics are placing new demands on network services [1]. Broadband wireless technologies can replace wired networks because of the comfort and mobility provided [2]. Wireless Local Area Networks (WLAN) and Wireless Personal Area Networks (WPAN) are the main wireless technology category for home area. Currently, the most important standard WLAN technology used in home networking is the IEEE 802.11 [3]. No single management protocol has been established for the management of various equipment of the home network environment. Therefore, flexible management systems are required with the support for multiple management protocols. The Simple Network Management Protocol (SNMP) standard has been most widely adopted by network device manufacturers [4]. It uses a Management Information Base (MIB) to define attributes managed in a network device. Quality of Service (QoS) support for enabling the different applications to operate and coexists is seen as a key functional requirement. The application QoS distribution can be divided into two main implementation approaches: bandwidth reservation and differentiated traffic handling. Bandwidth reservation systems use control messages to H. Kosch, L. Böszörményi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 1141–1147, 2003. © Springer-Verlag Berlin Heidelberg 2003
1142
T. Rantanen et al.
allocate bandwidth for a flow before the actual data transfer. This approach is taken by the Internet Engineering Task Force (IETF) for Integrated Services (IntServ) [5]. Few new management specifications targeting at management of heterogeneous networks are emerging. IETF SNMPConf working group has been developing methods for using SNMP framework for policy management [6]. The term policy management means the practise of applying management operations globally on all managed elements with similar characteristics. The Distributed Management Task Force (DMTF) is developing the Web-Based Enterprise Management (WBEM) initiative with high level of interoperability support with existing management standards [7]. Sun Microsystems has extended the Java 2 platform with a Federated Management Architecture (FMA) standard. The FMA implementation is called Jiro, which enables interoperability of different management systems, and provides tools for easy management application programming. [8] The home area network management is a new challenge. The management challenges in home environment have been identified and a high level architecture proposed in [9]. However, no technical architecture has been provided and the operation of the system is left open. Home network service and management architecture is provided in [10]. The architecture is flexible and takes advance of mobile code. However, it requires that the agents situated in home network appliances to be capable of running the mobile code. In heterogeneous network environment not all devices support running environment required for mobile code. In this paper, an architecture and design for a QoS management system for home area wireless networks are presented. The system is targeted for managing of heterogeneous networks and to provide abstraction and automation to the home management tasks. Section 2 presents the architecture and functionality of the designed management system, while Section 3 gives an overview of the prototype implementation. Conclusions and future work is discussed in the final section.
2 Wireless Home Area Management System (WHAMS) The conceptual layer architecture of the system is depicted in Fig. 1. First, physical devices represent the actual managed network devices, such as APs, LAN bridges, and terminals. The devices contain a set of managed attributes (e.g. variables, management procedures), which are organised into a MIB. Attribute adaptors have been defined for hiding the details of management access protocols and the physical device parameters from the higher layers of WHAMS. Thus, the adaptors are a uniform interface layer to the needed attributes of devices. Table 1 presents the designed attribute adaptors. Functions use the attribute adaptors to perform automated management tasks. Functions can be adjusted to be in observation or automation states. In an automation state, a function observes, generates notifications, and automatically reacts to changes according to the operation and configuration of the function. The target is to improve the network operation and performance that is related with the function.
Design of a Management System for Wireless Home Area Networking
1143
Profile Profile Function Function
Function Function Attribute AttributeAdaptor Adaptor
Attribute AttributeAdaptor Adaptor
Attribute AttributeAdaptor Adaptor
Management Management Access AccessProtocol Protocol
Management Management Access AccessProtocol Protocol
Management Management Access AccessProtocol Protocol
Device
Device Attribute
Device Attribute
Agent
Attribute
Fig. 1. WHAMS concept architecture with different layers
The following functions are defined: traffic function, frequency function, media function, security function, and auto configuration function. The traffic function observes traffic on all managed sub-networks, and balances the traffic loading of wireless networks by assigning client terminals to less loaded APs. The function provides a view of network traffic loading to a network manager. The frequency function observes or automates frequency allocations using radio attribute adaptors. When automated this function tries to balance used frequencies so that interference from other networks is minimal. The media function observes or automates bandwidth allocations of traffic flows. The function uses media connection attribute adaptors. When automated this function provides media connections with the requested bandwidth according to the connection priorities. This is established by limiting other traffic in the network when required. The security function manages the security policy of the whole network. When automated this function observes changes in the wireless networks (new devices), and ensures that the requested security level is achieved at all times. The auto configuration function enables automatic configuration of devices in the network. The configuration requires that the client is running auto configuration software that performs the configuration of the wireless client. A profile defines the network management configuration, defining the operation environment and its characteristics. Thus, a profile contains all parameters that are required by the functions for automated network management. A common policy is needed, as the operation of the different functions can be contradictory. A profile can emphasise the importance of certain applications and devices. In the current design, the profile rules are stated as priorities assigned for functions, terminals, and applications. The resulting system architecture consists of a centralised management server, wireless APs and optional client terminals. The server will contain the management functionality. A client terminal can have added WHAMS specific functions for
1144
T. Rantanen et al. Table 1. Attribute adaptors of the WHAMS system
Adaptor Status Radio Security Media connection Access point Internet gateway Traffic control
Examples of attributes Connection status, data transmission rate Radio type, used frequency, signal strength, radio usage Encryption, authentication, failed authentications Start address, end address, bandwidth control, delay, jitter Number of connected devices, loading Loading, configuration, access type Number of retries, number of errors, duplicate count
Description Generic adaptor that can be used to any type of device Abstracts the management of wireless media. Abstracts the security issues of a device. Abstracts the QoS management of an application flow. Abstracts the management of a access point. Abstracts the management of Internet gateway. Abstracts traffic control management
enabling more efficient management. Each device is connected to the WHAMS management system via a management access protocol. As discussed, there are several different standard protocols available, while proprietary protocols can also be supported. The efficient and full scale implementation of advanced management functions defined by WHAMS, such as application based QoS management, requires means to measure and analyse application traffic in network nodes. Adding of special measurement agents (WHAMS APs) can fulfil these functional requirements [11]. Still, the implantation of application QoS can be partly supported at the endpoints of a flow, where WHAMS specific functionality is easier to add. WHAMS is not platform specific but it can adapt to provide functionality that is possible with current technology.
3 WHAMS Prototype Implementation A prototype of the WHAMS is being implemented for verifying the functionality. The prototype is presented in Fig. 2. The server software contains implementations of layer architecture components: management protocols, attribute adaptors, functions, and profiles. The WHAMS server is implemented as Java. The server also contains a WWW-server for loading the user interface implemented as Java applet. The server platform is PC computer with Windows 2000 operating system. The WHAMS AP is capable of measuring flows and flow attributes such as throughput, delay, and delay variance. It provides bandwidth control in form of bandwidth reservation by queuing and access control for defined application traffic flows. The WHAMS AP software can be divided into four modules: bandwidth control, measurements, packet analyser, and management access. The management access module contains sniffer functionality. The packet analyser examines packet headers
Design of a Management System for Wireless Home Area Networking
TCP/IP Stack
Bandwidth Control Measurements Packet Analyser
Management Protocols WWW RMI WBEM WBEM Server Server SNMP SNMP TUTMP TUTMP HTTP
WLAN Driver
TCP/IP Stack
Ethernet Adapter Standard WLAN AP
WHAMS Server
TCP/IP stack
WLAN Bridge
Ethernet Driver
WLAN Adapter WHAMS Client
WHAMS AP
WLAN Management Access (DLL)
WLAN Management Access (DLL)
Client UI Local Configuration Auto Configuration
WHAMS Server Profiles Functions Adaptors
Internet Applications
WHAMS Java Client
1145
WLAN Driver
Ethernet Driver
WLAN Adapter
Ethernet Adapter
WHAMS AP
Fig. 2. WHAMS prototype topology and node architectures
for identifying flows. The measurements function performs actual flow measurements and stores the acquired data. The bandwidth control performs bandwidth reservations and control. It follows that the current reservations are fulfilled, and monitors the amount of unreserved bandwidth.
Fig. 3. Main view with floor plan (right) and the media function view (left) of the prototype
1146
T. Rantanen et al.
The user interface applet loads automatically to an Internet browser and establishes a communication to the WHAMS Remote Method Invocation (RMI) server. The manager UI consists of system views, function views, and adaptor views. Each function and adaptor has its own view displaying specific attributes and measurement results in graphics. The main window of the manager client shows the floor plan and all known devices and gives an access to managing devices, radio environment, function views, and placing new devices on the floor plan. The device window shows device properties and available adaptor views. The main windows and the media function views are presented in Fig. 3. In prototype implementation, the basic functions and adaptors use SNMP queries to several different standard and commercial MIBs. The media connection and adaptor implementations use a custom protocol to connect to a WHAMS AP and to obtain/adjust application QoS measurements and settings. The auto configuration function in WHAMS client listens to a selected TCP port and adjusts the client networks settings according to the configuration received from WHAMS server.
4 Conclusions and Future Work The WHAMS is being developed for the management of heterogeneous wireless and wired networks in home environments. The design targets have been to facilitate the co-existence of wireless networks, support QoS for application flows, and enable more automatic management. The resulting architecture has been a centralised management server, and add-on functionality for client terminals and APs. The hierarchical functional architecture enables flexibility. A prototype system based on Java implementation is currently under construction, and will be used to evaluate and verify the performance of the developed management system.
References 1. 2. 3. 4. 5. 6. 7. 8.
EURESCOM: LUPA: Local provision of 3G and 3G+ services, available at http://www.eurescom.de/ (May 2003) M. Hännikäinen et al.: Trends in personal wireless data communications, Computer Communications 25 (2002) 84 – 99 IEEE Sdt 802.11: Wireless Medium Access Control (MAC) and Physical layer (PHY) specifications (1999) J.D. Case, et al.: Simple Network Management Protocol (SNMP), IETF RFC 1157 (1990) P. P. White: RSVP and Integrated Services in the Internet: A Tutorial, IEEE Communications, (1997) 100-106 S. Waldbusser et al.,: Policy Based Management MIB, IETF SNMPConf working group, Internet-draft (2002) Homepage of DMTF WBEM: http://www.dmtf.org/standards/standard_wbem.php (May 2002) Sun Microsystems Inc.: Federated Management Architecture (FMA) specification version 1.0 revision 0.4, available at http://jcp.org/en/home/index (May 2003)
Design of a Management System for Wireless Home Area Networking 9.
1147
S. Moyer, et al.: Home Network Configuration Management & Service Assurance, IEEE 4th International Workshop on Networked Appliances, Gaithersburg, USA (2002) 77–86 10. C. Tsai, et al.: MASSIHN: a Multi-Agent Architecture for Intelligent Home Network Service, IEEE Transactions on Consumer Electronics 48, issue3 (2002) 509–514 11. T Rantanen et. al.: Design of a Quality of Service Management System for Wireless Local Access Networks, The International Conference on Telecommunications (ICT’2001), Bucharest, Romania (2001) Vol.3 107–114
Short Message Service in a Grid-Enabled Computing Environment Fenglian Xu, Hakki Eres, and Simon Cox School of Engineering Sciences, University of Southampton, Highfield, Southampton, SO17 1BJ, UK {F.Xu, Hakki.Eres, sjc}@soton.ac.uk http://www.geodise.org/
Abstract. Mobile computing devices such as mobile phones together with a land-based and wireless communication network infrastructure are the existing technical prerequisites for continuous access to networked services. The security of the system is a high concern in this environment, as well as its usability. This paper presents a network infrastructure for using the Short Message Service (SMS) to communicate with mobile phones via a grid-enabled service. The network system includes: a messenger server, a messenger client, Globus Servers and the Short Message Service Centre (SMSC) tied together with XML-RPC, TCP/IP and GRAM protocols for communication. A Matlab tool to use the gridenabled SMS has been implemented and we demonstrate its use in a grid-enabled application.
1
Introduction
Pervasive computing can deliver access to information with no limits. The Geodise project [1] aims to aid engineers in the design process by making available a suite of design optimisation and search tools and Computational Fluent Dynamics (CFD) analysis packages integrated with distributed grid-enabled computing, data, and knowledge resources. Here we use the mobile phone short message service (SMS) to communicate to a service or an application in a grid-enabled environment results of, for example, progress in an Engineering Design Calculation. The SMS should provide a two-way channel to schedule messages crossing most mobile network operators immediately and it also should be robust, reusable and platform independent. This paper demonstrates a case of how to use a mobile text message function in a grid-enabled computing environment [2] with a simple usage. The process is done remotely via Globus 2.2 [3] middleware software integrated into Matlab [4] which we use as a scripting/hosting environment. A full lifecycle of the SMS service software from design, implementation and test is presented. A usage of a tool is demonstrated to show how the SMS service is used in the entire gridenabled environment [5]. Related work is discussed in Sect. 2. The architecture and the implementation of the message system are described in Sect. 3. An H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 1148–1152, 2003. c Springer-Verlag Berlin Heidelberg 2003
Short Message Service in a Grid-Enabled Computing Environment
1149
exemplar of using the SMS is depicted in section Sect. 4. Conclusions and future work are summarised in Sect. 5.
2
Related Work
Several recent efforts have been proposed to address the issue of incorporating SMS capabilities for the remote monitoring and control of automation systems. The OPC-SMS gateway [6] is a service which enables mobile users to send a SMS message to request specific information from OPC servers and to be notified by OPC servers. A similar mechanism has been used for sending notifications to a cellular phone in a grid environment [7]. IBM has focused its resource on new technologies that connect to any application, from any device over any network, using any style of interface [8]. With recent advances in mobile computing technology, it is possible to support wireless messaging services which enable mobile users to access their messages at anytime and anywhere [9]. However, they all have a complicated usages to send/access the messages and the mobile phones play roles same as the computers. Several commercial SMS service providers allow users to send messages to mobiles via either a web application or their own applications which use APIs from the provider. Redcoal SMS [10] uses Simple Mail Transfer Protocol (SMTP) protocol so that users have to access the Internet to read and send messages. However, this service only works with Vodafone networks. Lucin SMS [11] has a web service with an engine to enable users to send text messages, ringtones, logos and retrieve a list of messages. Level 9 [12] provides a Short Message Service Centre (SMSC) which enables a two-way communication at a low cost. They also provide APIs in Perl, C/C++ and Java programming environment. Java programming is object-oriented and platform independent which meets the system requirements mentioned earlier.
3
System Architecture and Implementation
The infrastructure of the SMS system is shown in Fig. 1. The Messenger Service (MS) is the main entity which enables computer applications to send/receive messages to/from mobile phones. The MS includes a stack of services: Globus 2.2 Server, Messenger Client and Messenger Server. They provide the functions of security, sending and receiving the messages over the Internet crossing firewalls. The Globus server enables the message client to be run remotely and safely in a grid-enabled environment using the Globus grid computing toolkit. The protocol between the Globus server and the remote application is via Globus Resource Allocation Management (GRAM) and the protocol between the messenger client and the messenger server is TCP/IP. The messenger server waits for connections from clients and it then forwards the messages from the clients to the SMSC provided by Level 9. The SMSC enables communications between GSM networks and the Internet for sending and delivering text messages/binary images to current/3G mobile phones. The protocol between the messenger server
1150
F. Xu, H. Eres, and S. Cox
Fig. 1. System Architecture
and the SMSC software is via XML-RPC [13] over the Internet. XML-RPC is a simple, portable way to make remote procedure calls over HTTP. We have integrated the functionality into Matlab which is used as a hosting/scripting environment in Geodise. Security is an important issue in a network system due to the data transfer over the Internet and over the wireless network. A combination of authentication and authorisation is used for controlling who can use the software and what permissions are granted to the users. Globus Toolkit 2.2 Grid Security Infrastructure (GSI) [14] is used for enabling authentication and communication over an open network. Users can benefit from a single sign-on and delegation to run a job remotely via GRAM with a trusted authority, such as the Globus organisation or the UK-eScience Certificate Authority [15]. The SMS system is based on the client-server design pattern and is implemented with Java socket programming. The messenger client is responsible for requesting a connection to the messenger server, sending a short text message and closing the connection. The messenger server creates a server socket for incoming connections and listens for clients constantly for accepting a new client connection and starting a message received thread. The interface between the Matlab tool and the messenger client is via Resource Specification Language (RSL) [16] which is understood by GRAM. The message server acts as an SMSC client to connect with the SMSC and makes a remote procedure call with arguments which include the following fields: user name, password, mobile number and text message.
4
Application Exemplar
Design optimisation and search is a long and repetitive process. The process begins with a problem definition or a tool to generate a meshed geometry; a
Short Message Service in a Grid-Enabled Computing Environment
1151
CFD analysis is then performed from which an objective function is calculated. Design Search is the process by which the geometry is systematically modified to improve the value of the chosen objective function. This process often takes a long time (up to weeks) and may be fraught with problems caused by the compute infrastructure, meshing, or the actual CFD runs, or with the optimisation failing to make much progress in improving the design. It is therefore important to monitor the ongoing design run to ensure that it is progressing and hasn’t stalled or terminated anomalously. This section demonstrates how to use the gd_sendtext tool to send messages to mobile users in the Matlab environment. This tool checks the security before sending a message and returns a job status to see if the message is sent successfully. The Geodise toolkit [17] provides a set of Matlab tools, such as gd_proxyquery, gd_createproxy and gd_jobsubmit, to perform these tasks. The job is submitted by giving an RSL string and a remote host name as arguments. The RSL string combines the remote directory, the executable program name and the arguments to execute the program. An example of using gd_sendtext is shown in Fig. 2. As it can be seen from Fig. 2(a), a message is sent by the tool gd_sendtext in the Matlab environment and the job handle can be obtained for checking the job status by using gd_jobstatus. Fig. 2(b) shows that the message is received from a mobile phone in a second.
(a) Using gd sendtext to send a message
(b) Message received from gd sendtext
Fig. 2. Example of using gd sendtext
5
Conclusions and Future Work
The power of this application has enabled users to be notified beyond geographic boundaries in a grid-enabled environment. The messenger service can be plugged into any grid-enabled environment and it is easy to use. We have demonstrated how the short message service has been integrated with a grid-enabled computation of design search and optimisation system to allow monitoring of a job.
1152
F. Xu, H. Eres, and S. Cox
The advantage of using the short message service instead of the email protocol SMTP is that it does not require an Internet connection and is fast and easy to use. We have also developed a two way process in which a message from a phone can be used to steer or terminate calculations. Future work will include use of third generation (3G) technology to send output, e.g. graphics, from a code to a 3G mobile. We will also move to using the Open Grid Services Architecture [18] as reference implementations become available as an open standard for connecting our service to grid-enabled applications. Acknowledgements. We are grateful to Level 9 networks for providing SMS service capability for this project and EPSRC grant GR/R67705/01.
References 1. 2. 3. 4. 5.
6. 7. 8. 9. 10. 11. 12. 13. 14.
15. 16. 17.
18.
The Geodise Project. (2002) http://www.geodise.org/ Global Grid Forum. (2002) http://www.gridforum.org/ The Globus Project. (2002) http://www.globus.org/ Matlab 6.5. (2002) http://www.mathworks.com/ Pound, G.E., Eres, M.H., Wason, J.L.and Jiao, Z., Keane, A.J., Cox, S.J.: A Grid-Enabled Problem Solving Environment (PSE) for Design Optimisation within Matlab. Proceedings of 17th IPDPS (2003) Kapsalis, V., Koubias, S., Papadopoulos, G.: OPC-SMS: a Wireless Gateway to OPC-based Data Sources. Computer Standards & Interfaces 24 (2002) 437–451 Kranzlmueller, D.: Dewiz – Event-based Debugging on the Grid. EUROMICROPDP’02, IEEE (2002) IBM Resource. (2002) http://www.research.ibm.com/thinkresearch/pervasive.shtml Tan, D.H.M., Hui, S.C., Lau, C.T.: Wireless Messaging Services for Mobile Users. Journal of Network and Computer Applications 24 (2001) 151–166 (Redcoal Website) http://www.redcoal.com (Lucin Website) http://www.soapengine.com/lucin/soapenginex/smsx.asmx (Level9 Website) http://www.level9.net:2048/RPC2 (XMLRPC Website) http://xmlrpc-c.sourceforge.net/xmlrpc-howto/xmlrpc-howto.html Foster, I., Kesselman, C., Tsudik, G., Tuecke, S.: A Security Architecture for Computational Grids. In: 5th ACM Conference on Computers and Communications Security, San Francisco, California (1998) UK Grid-Support Centre. (2002) http://www.grid-support.ac.uk/ The Globus Resource Specification Language (RSL) v1.0. (2002) http://www-fp.globus.org/gram/rsl_spec1.html Eres, M.H., Pound, G.E., Jiao, Z., Wason, J., Xu, F., Keane, A.J., Cox, S.J.: Implementation of a grid-enabled Problem Solving Environment in Matlab. Accepted by the Workshop on Complex Problem-Solving Environments for Grid Computing (Held in conjunction with the ICCS) (2003) Foster, I., Kesselman, C., Nick, J.M., Tuecke, S.: The Physiology of the Grid - an open grid services architecture for distributed systems integration. http://www.globus.org/ogsa/ (2002)
Service Migration Mechanism Using Mobile Sensor Network* Kyungsoo Lim, Woojin Park, Sinam Woo, Sunshin An Department of Electronics Engineering, Korea University, 1, 5-Ga, Anam-dong Sungbuk-ku, Seoul, 136-701 Republic of Korea {angus, progress, niceguy, sunshin}@dsys.korea.ac.kr
Abstract. The recent advancement of wireless communication technology provides us with new opportunities, but it initiates new challenges and demands as well. Especially, there is an increasing demand for supporting user mobility and service mobility transparently. In this paper, we propose a new mechanism for service migration based on sensor networks of wireless sensor nodes. This mechanism is composed of a component-based server-side model for service mobility and Mobile IP technology for supporting user mobility among sensor networks.
1
Introduction
The technologies of wireless communications and electronics have made a remarkable progress recently. These technological advancements have enabled us to develop a new mechanism based on the sensor networks. A sensor network consists of a large number of sensor nodes which collect information in their coverage regions, and a sink node[1]. In this paper, we propose a new mechanism for service migration using the networks of wireless sensor nodes[2]. This mechanism can support user mobility and service mobility independent of the current location of users, and is composed of two parts: a component-based server-side model for service mobility[3,4] and Mobile IP technology for supporting user mobility among sensor networks. The remainder of this paper is organized as follows. In section 2, we illustrate our service migration model using mobile sensor network to support user mobility. Section 3 describes the service management server mechanism to provide service migration transparently. Also, in section 4, the overall scenario is proposed. Finally, section 5 summarizes our work. _____________________________ * An unabridged version of this paper is available at the following URL : “http://nlab.korea.ac.kr/~angus/euro-par2003_paper.pdf”.
H. Kosch, L. Böszörményi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 1153–1158, 2003. © Springer-Verlag Berlin Heidelberg 2003
1154
2
K. Lim et al.
Service Migration Model Using Mobile Sensor Network
The term “service” means the set of information that is provided from a computer user’s application programs. Also, the term “mobile sensor node” means the sensor node that provides the mobile node’s function in Mobile IP. The users are identified and authenticated by using the mobile sensor node. Service migration means that the working environment of a computer user moving into other networks is supported and maintained transparently by using a mobile sensor node[2]. In this paper, we propose a new mechanism to support service migration effectively in wireless communication networks. To configure a sensor network, sensor detection, sensor auto-configuration, and sensor handoff are required. Also, we utilize Mobile IP for constituting these sensor networks. Mobile IP is the technology that can maintain its connection even though the mobile host is moving. Mobile IP has route discovery, tunneling, and handoff mechanism for mobility support, and it is used for the global network[5]. By utilizing these characteristics of Mobile IP, sensor nodes can be automatically detected and configured. Therefore, sensor mobility is ensured. In addition, the architecture of a sensor network has the advantages of flexibility and extensibility that both mobile devices and sensors can be applied and integrated in the same way. A sensor node is configured by Mobile IP and ad-hoc routing protocol[2], and a SMS maintains and manages the service profile and authentication information for service migration. Sensor Network
Sensor Network Interface
Sensor Node VC-1 Visited Agent
Visited Network
Visited Client
Moved User with Sensor Node form Home Network
Sensor Node
VC-2
Service Management Server
moving service
Home Agent
Home Client
Home Network HC-1
HC-2
Fig. 1. Service migration model
Figure 1 shows the overall model for service migration. It illustrates the conceptual model of a sensor network for providing service migration effectively. A sensor network is composed of sensor nodes and a sink node. A sink node is the external interface to communicate with agents and SMS(Service Management Server). Also, home agent and foreign agent in Mobile IP scheme are executed in the sink node to communicate with other sensor networks and external networks. Also, sensor nodes can be intermediate nodes to configure the sensor networks and the routing mechanism necessary for mobility support. In this paper, an agent operates as a sink
Service Migration Mechanism Using Mobile Sensor Network
1155
node, manages its sensor network, and configures sensor nodes automatically. Because the communication range of a sensor node is limited, it can function as a router for neighbor nodes to interact with its agent. Agents are entities that process the transmission of messages between a server and the clients. A visited client has only to send request messages to receive a service object regardless of its server location and its current network. This provides the transparency of request messages. A VC(Visited Client) can communicate with a SMS in a home network through VA(Visited Agent) only. Because the secure tunneling mechanism of Mobile IP is used, service object has the characteristics of the integrity and the safety. A HA(Home Agent) sends messages to a VA by encapsulating the messages and the VA decapsulates them. Then, the VA transfers the messages to the visited client based on its sensor ID and its IP address.
3
Service Management Server
In this paper, we propose a component-based model that processes and transports service objects to support sensor-based service mobility. A SMS consists of a serverside component, which processes the request of users and stores its result. Also, SMS administers and manages the user passport, the client information, and the application information. The server-side component is composed of a session bean and three entity beans[6] as shown in figure 2. DAO (Data Access Object)
Client Context
Entity Bean
Session Bean User Passport Authentication of Originator Sensor ID
Client
Client Information Address Goals and Status Information Capability Set
ServiceManage Agent UserPro
Client
ClientInfo
Client DB AppInfo EJB Server
Container
Home Interface
Application Information Application List Data version & file information Application Information
Local Home Interface
Fig. 2. System architecture of a server
A service management agent(SMA) bean takes charge of overall service logic of a server. This bean operates as follows. A SMA is a session bean, which loads the overall service logic and processes messages from a HA. When a SMA receives the home client’s request message from a HA, it processes and classifies the request, and
1156
K. Lim et al.
then sends its result to a HA. Entity beans are object expressions of a persistence data process[7], and they each have their own database and manage it. These entities are made up of UserPro, ClientInfo, and AppInfo. Messages are classified according to characteristics of each entity bean, and it is stored in a pertinent table of each database[7]. UserPro, ClientInfo, and AppInfo bean are mapped to User Passport, Client Information, and Application Information in order. When a SMA receives the request related to service object from a HA, it collects the stored information of the user in the database through DAO(Data Access Object) and send it to a HA. Agents send the requests of the clients to a SMS and the responses of the SMS to the clients. Therefore, agents are entities that process the transmission of messages between a server and the clients. SN move
HC
SMS
HA
VA
VC
SN
Advertisement Service Registration SN Registration SN Notification Authentification Request
Authentification Response
Service Object & CS Request Service Object & CS Response
Moving Service
Fig. 3. Service scenario
4
Service Migration Mechanism
Figure 3 shows the overall Moving service scenario flow. A sensor node has only to communicate with an agent of its current network. Also, a visited client has only to communicate with its SMS through agents. Authorization and information for service migration are provided through a SMS. When a sensor node moves into a VN, it is configured by its VA. The VA broadcasts the sensor node’s registration information and then this information is displayed in its local hosts. When the user confirms the information, its client application requests its authentication and waits for its response to be received. After that, the visited client sends the user’s service object request and capability set to its VA. The VA sends these messages to the sensor node’s HA and the HA forwards the information to a SMS. The SMS reply by sending the service object and some additional information to the HA. Finally, the HA transmits the above messages to the visited client. This scenario ensures our sensor-based service migration effectively.
Service Migration Mechanism Using Mobile Sensor Network
1157
The method to maintain the user’s working environment should be considered. There are two ways to maintain the user’s working environment in our network model that supports sensor-based service migration. First, the user stores its working environment information and registers to its SMS through home client application. The second, if the user moves into another VN without storing its information, a HA can conceive the sensor node’s mobility through sensor node detection process. In the home network, a HA asks home client application to store and register the user’s working environment in its SMS. As a sensor node moves into the other VN, the above two methods are repeated. Although a sensor node roams around the world, service migration can be provided through its VA and HA in our proposed model. Our component-based SMS has advantages in interoperability and compatibility. Also, it can back-up service profile. These service profiles in a SMS are maintained and managed by individual version information.
5
Conclusion and Future Work
In this paper, we have proposed a new mechanism to support service migration transparently in wireless sensor networks. This mechanism consists of a componentbased server-side model to accomplish service mobility and Mobile IP technology between sensor networks to ensure user mobility. With these two new features, service migration can be guaranteed. Our model is interoperable and compatible with any server in other networks, and this system can be implemented on any ad hoc network. We have showed that the proposed mechanism can be applied to a wireless sensor network, and that it can be a key technology of future wireless sensor networks for supporting user mobility and service mobility. Future work of this research includes implementation and experimentation of our proposed model. In particular, we will focus on the following: -
Proposing a method for a sensor node to find and select the optimal client that satisfies CS(capability-set)[8] in mobile sensor networks. Resolution of the problem of interference between adjacent sensor nodes. Reduction of the useless packets transmitted when a handoff is occurred. Providing an enhanced model to support service mobility and user mobility in various network environments.
References 1. 2.
Ian F. Akyildiz, Weilian su, Yogesh Sankarasubramanism, and Erdal Cayirci “A Survey on Sensor Networks”, IEEE Communications Magazine, August, 2002. Jin Zhu, Symeon Papavassiliou, and Sheng Xu, “Modeling and analyzing the dynamics of mobile wireless sensor networking infrastructures” Vehicular Technology Conference (VTC 2002-Fall), IEEE, 2002 56th.
1158 3.
K. Lim et al.
C. BRENT HIRSCHMAN, "Service Mobility/Transparency For Personal Communications", Universal Personal Communication (ICUPC ’92) 1992. 4. Rong N. Chang, “Realizing Service Mobility for Personal Communications Applications”, Universal Personal Communications 1993 ‘Personal Communicaiton : Gateway to the 21st Century(ICUPC’93), 1993. 5. James D. Solomon “Mobile IP”, PRENTICE HALL 1998 . 6. Richard Monson-Haefel, “Enterprise Java Beans”, O’REILLY, 1999. 7. Ted Neward, “Server-Based Java Programming”, MANNING, 2000. 8. Pearce, P.R. “CS-2 enhancements for user interaction“, Telecommunications, 1998. 6th IEE Conference on (Conf. Publ. No. 451) , Mar-1 1998. 9. Sasha Slijepcevic, Vlasios Tsiatsis, Scott Zimbeck, “On Communication Security in Wireless Ad-Hoc Sensor Networks”, International Workshops on Enabling Technologies: Infrastructure for Collaborative Enterprises(WETICE’02), 2002. 10. Charles. E. Perkins, “Mobile IP”, International Journal of communication system, May, 1998.
Topic 16 Distributed Systems and Distributed Multimedia Fernando Pereira, Abdulmotaleb El Saddik, Roy Friedman, and L´ aszl´o B¨osz¨orm´enyi Topic Chairs
Topic 16 (Distributed Systems and Distributed Multimedia) has received 23 papers. 9 of them were redirected to Topic 12 (Architectures and Algorithms for Multimedia Applications). From the remaining 14 papers, the programm committee has accepted 7 regular and 1 short papers. The majority of the papers in this chapter cover issues of distributed systems in general. Two papers deal with questions of distributed multimedia systems. The chapter contains papers on – – – – –
Cluster computing; Mobile agent systems; Distributed shared memory systems; Dynamic node placement in adaptive video servers; Multimedia bitstream description in the context of the emerging MPEG-21 standard.
H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, p. 1159, 2003. c Springer-Verlag Berlin Heidelberg 2003
Nswap: A Network Swapping Module for Linux Clusters Tia Newhall, Sean Finney, Kuzman Ganchev, and Michael Spiegel Swarthmore College, Swarthmore, PA 10981, USA
Abstract. Cluster applications that process large amounts of data, such as parallel scientific or multimedia applications, are likely to cause swapping on individual cluster nodes. These applications will perform better on clusters with network swapping support. Network swapping allows any cluster node with over-committed memory to use idle memory of a remote node as its backing store and to “swap” its pages over the network. As the disparity between network speeds and disk speeds continues to grow, network swapping will be faster than traditional swapping to local disk. We present Nswap, a network swapping system for heterogeneous Linux clusters and networks of Linux machines. Nswap is implemented as a loadable kernel module for version 2.4 of the Linux kernel. It is a spaceefficient and time-efficient implementation that transparently performs network swapping. Nswap scales to larger clusters, supports migration of remotely swapped pages, and supports dynamic growing and shrinking of Nswap cache (the amount of RAM available to store remote pages) in response to a node’s local memory needs. Results comparing Nswap running on an eight node Linux cluster with 100BaseT Ethernet interconnect and faster disk show that Nswap is comparable to swapping to local, faster disk; depending on the workload, Nswap’s performance is up to 1.7 times faster than disk to between 1.3 and 4.6 times slower than disk for most workloads. We show that with faster networking technology, Nswap will outperform swapping to disk.
1
Introduction
Using remote idle memory as backing store for networked and cluster systems is motivated by the observation that network speeds are getting faster more quickly than are disk speeds [9]. In addition, because disk speeds are limited by mechanical disk arm movements and rotational latencies, this disparity will likely grow. As a result, swapping to local disk will be slower than using remote idle memory as a “swap device” and transferring pages over the faster network. Further motivation for network swapping is supported by several studies [2,4,10] showing that large amounts of idle cluster memory are almost always available for remote swapping. We present Nswap, a network swapping system for heterogeneous Linux clusters and networks of Linux machines. Nswap transparently provides network swapping to cluster applications. Nswap is implemented as a loadable kernel H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 1160–1169, 2003. c Springer-Verlag Berlin Heidelberg 2003
Nswap: A Network Swapping Module for Linux Clusters
1161
module that is easily added as a swap device to cluster nodes and runs entirely in kernel space on an unmodified 1 Linux kernel; applications can take advantage of network swapping without having to re-compile or link with special libraries. Nswap is designed to scale to large clusters using an approach similar to Mosix’s design for scalability [1]. In Nswap there is no centralized server that chooses a remote node to which to swap. In addition, Nswap does not rely on global state, nor does it rely on complete or completely accurate information about the state of all cluster nodes to make swapping decisions. Thus, Nswap will scale to larger clusters because each node independently can make swapping decisions based on partial and not necessarily accurate information about the state of the cluster. Nswap supports dynamic growing and shrinking of each node’s Nswap cache in response to the node’s local memory use, allowing a node to reclaim some of its Nswap cache space for local paging or file I/O when it needs it, and allowing Nswap to reclaim some local memory for Nswap cache when the memory is no longer needed for local processing. Growing and shrinking of the Nswap cache is done in large chunks of memory pages to better match the bursty behavior of memory usage on a node [2]. Another feature of Nswap is that it avoids issuing writes to disk when swapping; it does not write-through to disk on swap-outs, and it supports migration of remotely swapped pages between the cluster nodes that cache them in response to changes in a remote node’s local memory use. Only when there is no available idle memory in the cluster for remote swapping does Nswap revert to swapping to disk. Because Nswap does not do write-through to disk, swapping activity on a node does not interfere with simultaneous file system disk I/O on the node. In addition, as long as there is idle remote memory in the cluster, Nswap can be used to provide disk-less cluster nodes the ability to swap. In section 2 we discuss related work in network swapping. In section 3 we present the details of Nswap’s implementation, including a presentation of the swapping protocols. In section 4 we present the results of measurements of Nswap running on an eight node cluster. Our results show that with 100 BaseT technology we are comparable to high-end workstation disks, but that with faster interconnect, Nswap will outperform swapping to disk. In section 5 we discuss future directions for our work.
2
Related Work
There have been several previous projects that examine using remote idle memory as backing store for nodes in networks of workstations [3,6,10,8,5,11,7]. For example, Feeley et. al.[6] implement a network cache for swapped pages by modifying the memory management system of the DEC OSF/1 kernel. Their system views remote memory as a cache of network swapped pages that also are written through to disk; remote servers only cache clean pages and can arbitrarily drop 1
Currently, we require a re-compile of the 2.4.18 kernel to export two kernel symbols to our module, but we have not modified Linux kernel code in any way.
1162
T. Newhall et al.
a page when their memory resources become scarce. Each node’s memory is partitioned into local space and network cache space. When a page fault occurs, a node’s local space may grow by one page at the expense of a page of its network cache space. Our growing and shrinking policies do something similar, but we grow and shrink in larger units to better match bursty memory use patterns. In addition, our system does not do a write-through to disk on every swap-out. Thus, our system will not be slowed down by disk writes and will not interfere with a node’s local file I/O. Markatos and Dramitinos [10] describe reliability schemes for a remote memory pager for the DEC OSF/1 operating system. Their system is implemented as a client block device driver, and a user-level server for storing pages from remote nodes. Both dirty and clean pages can be stored at remote servers. When a server is full, or when it needs more memory for local processes, remote pages are written to the server’s disk, resulting in a significant slow down to a subsequent swap-in of the page. Our system avoids writing pages to disk in similar circumstances by migrating the page to another server. In addition, our system allows a server’s cache size to change, and it allows nodes to dynamically change roles between clients and servers based on their local memory use. Bernard and Hamma’s work [5] focuses on policies for balancing a node’s resource usage between remote paging servers and local processes in a network of workstations. Their policy uses local and remote paging activity and local memory use to determine when to enable or disable remote paging servers on a node. Our growing and shrinking policies for Nswap cache sizes have to solve a similar problem. However, on a cluster system, where there is no notion of a user “owning” an individual cluster node, so we expect that we can keep Nswap cache space on nodes with under-committed memory but higher cpu loads.
3
Nswap Implementation
Nswap consists of two components running entirely in kernel space (shown in Figure 1). The first is a multi-threaded client that receives swap-in and swap-out requests from the kernel. The second is a multi-threaded server that manages a node’s Nswap cache and handles swap-in and swap-out requests from remote clients. Each node in the Nswap cluster runs an Nswap client and an Nswap server. At any given time, a cluster node is acting either as a client or a server, but typically not as both simultaneously; a node adjusts it role based on its local memory use. The goals of Nswap’s design are to provide efficient, transparent network swapping to cluster applications, to dynamically adjust to individual node’s memory needs, and to scale to large, heterogeneous clusters. 3.1
Nswap Client and Server
The Nswap client is implemented as a block pseudo-device. This abstraction allows an easy and portable interface to the Linux kernel’s swapping mechanism, which communicates with Nswap exactly as it would any physical block device. Idle memory across the cluster is accessed through read and write requests from
Nswap: A Network Swapping Module for Linux Clusters Node B
Node A User Space Kernel Space
11 00
swap in page i IP Table
Nswap Cache
11 00 00 11
A’s page j
sock
...
...
...
host amt B 44 C 17 D 20
Nswap Server
swap out page j
Nswap client
Nswap Server
1163
shadow swap map
B
B
client threads
Nswap client A’s page i
1 0 0 1 0 1
IP Table
server threads Nswap Comm. Layer
Nswap Communication Layer
PUTPAGE GETPAGE
Fig. 1. Nswap System Architecture. Node A shows the details of the client including the shadow slot map used to store information about which remote servers store A’s pages. Node B shows the details of the server including the Nswap cache of remotely swapped pages. In response to the kernel swapping in (out) a page to our Nswap device, a client thread issues a PUTPAGE (GETPAGE) to write (read) the page from a remote server.
the kernel to the Nswap client device. The client uses multiple kernel threads to simultaneously service several swap-in or swap-out requests. The Nswap client needs to add additional state to the kernel’s swap map so it can keep track of the location of its remotely swapped pages. We add a shadow swap map (Figure 1) that stores the following information (in 16 bits) for each slot in the kernel’s swap map of our device: the serverID, designating which remote server stores the page; the hop count, counting the number of instances a page has been migrated (used to avoid race-conditions that can occur during page migration and to limit the number of times a page can be migrated); the time stamp, identifying an instance of the slot being used by the kernel (used to identify ”dead” pages in the system that can be dropped by the server caching them); and the in use bit to control conflicting simultaneous operations to the same swap slot. Additionally, these fields allow for a communication protocol that does not require the client’s state to be synchronously updated during page migrations. The Nswap server is implemented as a kernel-level daemon that makes local memory available for Nswap clients. When the Nswap module is loaded, the server starts three kernel threads: a listener thread, a memory thread, and a status thread. The listener thread listens for connections from Nswap clients, and starts new threads to handle the communication. The memory thread monitors the local load, communicating its load to other hosts, and if necessary, triggering growing and shrinking its Nswap cache. The status thread accepts UDP broadcast messages from memory threads on other servers. These messages contain changes in a server’s available Nswap cache size. The status thread updates its IPTable with this information. The IPTable on each node contains a potentially incomplete list of other server’s state. Each entry contains an Nswap server’s IP, the amount of Nswap cache it has available, and a cache of open sockets to the server so that new
1164
T. Newhall et al.
connections do not have to be created on every page transfer. The Nswap client uses information in the IPTable to select a remote server to send its pages, and to obtain connections to remote servers. The information in the IPTable about each server’s Nswap cache availability does not have to be accurate, nor does there need to be an IPTable entry for every node in the cluster, for a client to choose a remote server on a swap-out. This design will help Nswap scale better to larger clusters, but at the expense of clients perhaps not making the best possible server choice. Typically, the information is accurate enough to make a good server choice. Nswap recovers from a bad server choice by migrating pages to a better server. When the client makes a request for a page, it sends page meta-data (clientID, slot number, hop count, timestamp) that the server uses to locate the page in its Nswap cache. Page meta-data is also used to determine when cached pages are no longer needed. Since the Linux kernel does not inform a swap device when it no longer needs a page, “dead” pages can accumulate. A garbage collector thread in the Nswap client periodically runs when the client has not swapped recently. It finds and cleans dead slots in the swap slot map, sending the server a message indicating that it can drop the ”dead” page from its cache. In addition, “dead” pages are identified and dropped during page migration, and when a client re-uses an old slot during a swap-out operation. A predictive monitoring policy is used to dynamically grow or shrink the size of Nswap cache. A status thread on each node periodically polls the behavior of the Linux memory management subsystem. If the machine is showing symptoms of high memory usage, Nswap cache will shrink, possibly triggering migration of some of the remote pages it caches to other Nswap servers. When local memory is underutilized, Nswap cache size is increased. 3.2
Nswap Communication Protocol
The communication protocol between different nodes in an Nswap cluster is defined by five types of requests: PUTPAGE, GETPAGE, PUNTPAGE, UPDATE and INVALIDATE. When Nswap is used as a swap device, the kernel writes pages to it. The client receives a write request and initiates a PUTPAGE to send the page to a remote server. At some later time the kernel may require the data on the page and issue a read request to the Nswap device. The client uses the GETPAGE request to retrieve the page (see Figure 1). If the workload distribution changes, it may become necessary for a server to reduce its Nswap cache size, using the PUNTPAGE request to offload the page to another Nswap server. Moving a page from one server to another involves an UPDATE request to alert the client to the new location of the page and an INVALIDATE request from the client to the old server to inform the old server that it can drop its copy of the page (see Figure 2). GETPAGE and PUTPAGE are designed to be as fast as possible for the client who is currently swapping. If a client makes a bad server choice for a PUTPAGE, the Nswap servers handle it through page migration rather than forcing the client to make a better server choice, which would slow down the client’s swap-outs. In addition, the protocol is designed to limit synchronous activities. As a result,
Nswap: A Network Swapping Module for Linux Clusters Node A
00 (0) 11 00 11 00 11
Node A
(1)
Node A
Node B PUTPAGE
11 00 00 11 00 11
Node B
00 11 11 00 00 11
Node C
0 1 1 0 0 1 0 1
0 1 1 0 0 1 UPDATE
Node A
Node C PUNTPAGE
Node B
(2)
1165
11 00 00 11 00 11
Node B
x 1 0 0 1 0 1
(3)
Node C
1 0 0 1 0 1
INVALIDATE
Fig. 2. Page Migration in Nswap. Node A acts as an Nswap client, and Nodes B and C act as servers. A PUNTPAGE from server B to server C (1), triggers an UPDATE from server C to client A (2), which in turn triggers an INVALIDATE from client A to server C (3). At this point, B can drop its copy of A’s page.
extra state (page meta-data) is passed with requests so that a receiver can detect out-of-order and old requests and handle them appropriately. The following is an overview of the communication protocol: PUTPAGE is used to ask a remote host to store a local page. The client picks a remote server using its IPTable information, and sends the server a PUTPAGE command and the page’s meta-data. The server almost always responds with an OK PUTPAGE, and then the client sends the page data. Even a server with a full Nswap cache typically will accept the page from the client. In these cases, the server starts to remove some of its least recently cached pages by issuing PUNTPAGEs after completing the PUTPAGE transaction. GETPAGE is a request to retrieve a page that is stored remotely. The client sends the GETPAGE command and the page meta-data. The server uses the meta-data to find the page in its Nswap cache, and sends it to the client. INVALIDATE is used by a client to inform a server that it may drop a page from its Nswap cache. An INVALIDATE can result from page migration (see PUNTPAGE below) or from garbage collection on the client. The client sends an INVALIDATE command and the page meta-data. The server compares all the meta-data of the page and frees the page if these match. Otherwise, the INVALIDATE request is for an old page and is ignored. PUNTPAGE is used to implement page migration. When a server becomes overcommitted, it attempts to get rid of the least recently cached foreign pages by punting them to other servers. The over-committed server sends a PUNTPAGE command and the page data to another remote server. The original server cannot drop the page until it receives an INVALIDATE from the client as described above. Once the page has been transferred to the new server, the new server initiates an UPDATE to the owner of the page. If there is no available Nswap cache space in the cluster, the server punts the page back to its owner who writes it to its local disk. UPDATE is used to inform a client that one of its pages has moved. The new server sends an UPDATE command to the client with the page meta-data. The page meta-data is used to detect whether the UPDATE is for an old page, in which case the client sends an INVALIDATE to the sender of the UPDATE; otherwise, the client sends an INVALIDATE to the previous server caching the page so that it can drop its copy of the page.
1166
4
T. Newhall et al.
Results
We present results comparing swapping to disk and Nswap for several workloads. Our experiments were run on a cluster of 8 nodes 2 running version 2.4.18 of the Linux kernel connected by 100BaseT Ethernet. Four of the nodes have Intel Pentium II processors, 128 MB of RAM, and Maxtor DiamondMax 4320 disks with a data rate of up to 176 Mb/sec. The other machines have Pentium III processors, 512 MB of RAM, and IBM Deskstar disk with a sustained data rate of 167-326 Mb/sec and a max rate of 494 Mb/sec. In both machine types, disk transfer rates are faster than network transfer rates, so we expect to be slower than swapping to disk. However, our results show that for several workloads swapping across a slower network is faster than swapping to faster local disk. In addition, we calculate that on 1 and 10 Gbit Ethernet, Nswap will outperform swapping to disk for almost all workloads. 4.1
Workload Results
Table 1 shows run times of several workloads comparing Nswap with swapping to disk. On the PIII’s we ran two versions of Nswap, one using TCP/IP to transfer pages (column 5), the other using UDP/IP (column 6). We ran one and four process versions of each workload with one node acting as the client and three nodes acting as servers. We also disabled Nswap cache growing and shrinking to reduce the amount of variation between timed runs. Workload 1 consists of a process that performs a large sequential write to memory followed by a large sequential read. It is designed to be the best case for swapping to disk because there will be a minimal amount of disk head movement when swapping due to the way in which Linux allocates space on the swap partition. Workload 2 consists of a process that performs random writes followed by random reads to a large chunk of memory. This workload stresses disk head movement within the swap partition only. Workload 3 consists of two processes. The first runs a Workload 1 application and the second is a process that performs a large sequential write to a file. Workload 3 further stresses disk head movement when swapping to disk because file I/O and swap I/O are concurrently taking place in different disk partitions. Workload 4 consists of two processes; the first runs the Workload 2 application and the second runs the large sequential file write application. The four process versions of each workload are designed to represent a more realistic cluster workload. The size of each Workload varied from platform to platform based on the size of RAM on each machine, and they varied between the Random and Sequential tests. As a result, for each Workload (row), only the values in columns 2 and 3 can be compared, and the values in columns 4, 5, and 6 can be compared. We expect that disk will perform much better than Nswap on the single process versions of Workload 1 and Workload 2 because the disk arm only moves 2
Although our cluster currently consists of only Pentium architectures, Nswap will run on any architecture supported by Debian Linux.
Nswap: A Network Swapping Module for Linux Clusters
1167
Table 1. Swapping to fast disk vs. TCP Nswap and UDP Nswap on 100BaseT for PII and PIII nodes. The rows are 1 and 4 process runs of each of the four Workloads. Time is measured in seconds and the values are the mean of 5 runs of each benchmark. Bold values indicate runs for which Nswap is faster than disk. (Workload) # procs (1), 1 (1), 4 (2), 1 (2), 4 (3), 1 (3), 4 (4), 1 (4), 4
DISK (PII) 98.0 652.9 874.6 996.7 632.9 1312.1 1971.2 1453.0
NSWAP (PII, TCP) 450.9 (4.6x slower) 630.6 (1.04x fast) 1937.0 (2.2x slow) 617.0 (1.6x fast) 737.7 (1.7x slow) 1127.2 (1.2x fast) 2111.0 (1.07x slow) 1094.6 (1.3x fast)
DISK (PIII) 13.1 551.4 266.8 68.6 770.2 727.1 923.9 502.5
NSWAP (PIII, TCP) 154.3 (11.7x slow) 1429.7 (2.6x slow) 1071.8 (4.0x slow) 189.3 (2.8x slow) 1111.1 (1.4x slow) 1430.5 (1.9x slow) 1529.3 (1.7x slow) 498.7 (1.01xfast)
NSWAP (PIII, UDP) 61.3 (4.6x slow) 614.4 (1.1x slow) 153.5 (1.7x fast) 50.3 (1.4x fast) 811.0 (1.1x slow) 619.5 (1.2x fast) 821.7 (1.1x fast) 429.2 (1.2x fast)
within the swap partition when these workloads run, and our results confirm this; the worst slow downs (4.6 and 11.7) are for the single process runs of Workload 1. The differences between the performance of Workloads 1 and 2 show how disk head movement within the same disk partition affects the performance of swapping to disk. For the four process runs of Workloads 1 and 2, Nswap is much closer in performance to disk, and is faster in some cases; the TCP version on the PIIIs is 2.6 and 2.8 times slower than disk, the UDP version on the PIIIs is 1.4 times faster for Workload 2, and on the PIIs both workloads are faster than disk (1.04 and 1.6 times faster). This is due to a slow down in disk swapping caused by an increase in disk arm movement because multiple processes are simultaneously swapping to disk. The results from Workloads 3 and 4, where more than one process is running and where there is concurrent file I/O with swapping, show further potential advantages of Nswap; Nswap is faster than disk for the four process version of all Workloads on the PIIs, for three of the four Workloads under UDP Nswap, and for Workload 4 under TCP Nswap on the PIIIs. The differences in TCP and UDP Nswap results indicate that TCP latency is preventing us from getting better network swapping performance. Nswap performs better for Workloads 3 and 4 because network swapping doesn’t interfere with file system I/O, and because there is no increase in per-swap overhead when multiple process are simultaneously swapping. 4.2
Results for Nswap on Faster Networks
Currently we do not have access to faster network technology than 100BaseT Ethernet, so we are unable to run experiments of Nswap on a cluster with a network that is actually faster than our cluster’s disks. However, based on measurements of our workloads running Nswap with 10BaseT and Nswap with 100BaseT Ethernet, we estimate run times on faster networks. The rows of Table 2 show execution times of workloads run on 100BaseT and 10BaseT, and our estimates of their run times on 1 and 10 Gigabit Ethernet.
1168
T. Newhall et al.
Table 2. Time (and Speedups) for TCP & UDP Nswap on faster networks. All Workloads were run on the PIIIs. Bold values indicate cases when Nswap is faster than disk. The rows show for each workload calculated 1 and 10 Gbit run times (col. 5 & 6) based on measured speedup values between runs on 10 and 100 BaseT (col. 3 & 4). (Workload) Disk 10BaseT 100BaseT 1 Gbit 10 Gbit (1) TCP 580.10 5719.00 1518.34 (speedup 3.8) 1075.00 (5.3) 1034.17 (5.5) (1) UDP 12.27 306.69 56.80 (speedup 5.4) 28.90 (10.6) 26.30 (11.6) (2) UDP 226.79 847.74 153.54 (speedup 5.5) 77.30 (10.9) 70.30 (12.1) (4) UDP 6265.39 9605.91 1733.93 (speedup 5.54) 866.18 (11.1) 786.72 (12.2)
We use the speedup results from our 10 and 100 BaseT measurements of each benchmark and apply Amdahl’s Law to get estimates of speedups for 1 and 10 Gigabit Ethernet using the following equation: T otalSpeedup =
1 1−F ractionBandwith+F ractionBandwith/SpeedupBandwidth .
For each Workload, we compute F ractionBandwith based on T otalSpeedup values from measured 10 and 100 BaseT runs. Using this value, we compute T otalSpeedup values for 1 and 10 Gbit Ethernet (columns 5 and 6 in Table 2). The measured speedups between 10 and 100 BaseT of TCP version of Nswap are lower than we expected (one example is shown in row 1 of Table 2); timed runs of these workloads when they completely fit into memory compared to when they don’t, show that over 99% of their total execution time is is due to swapping overhead. We also measured our GETPAGE and PUTPAGE implementations and found that over 99% of their time is due to transferring page data and metadata. However, the UDP version of Nswap results in speedup values closer to what we expected. In fact, UDP Nswap on Gigabit Ethernet outperforms swapping to disk for all Workloads except the single process version of Workload 1 on the PIIIs. Our speedup results further indicate that TCP latency is preventing Nswap from taking full advantage of improvements in network bandwidth.
5
Conclusions and Future Work
Nswap is a network swapping system for heterogeneous Linux clusters. Because Nswap is implemented as a loadable kernel module that runs entirely in kernel space, it efficiently and transparently provides network swapping to cluster applications. Results from experiments on our initial implementation of Nswap on an eight node cluster with 100BaseT Ethernet show that Nswap is comparable to swapping to faster disk. Furthermore, since it is likely that network technology will continue to get faster more quickly than disk technology, Nswap will be an even better alternative to disk swapping in the future. One area of future work involves examining reliability schemes for Nswap. We are also investigating reliable UDP implementations for the Nswap communication layer so that we can avoid the latency of TCP to better take advantage
Nswap: A Network Swapping Module for Linux Clusters
1169
of faster networks, but to still get reliable page transfers over the network. Other areas for future work involve testing Nswap on faster networks and larger clusters, and further investigating predictive schemes for determining when to grow or shrink Nswap caches sizes. In addition, we may want to examine implementing an adaptive scheme into Nswap. Based on the results from the single process version of Workload 1 in Table 1, there may be some workloads for which swapping to disk will be faster. Nswap could be designed to identify these cases, and switch to disk swapping while the workload favors it. Acknowledgments. We thank Matti Klock, Gabriel Rosenkoetter, and Rafael Hinojosa for their participation in Nswap’s early development.
References 1. Barak A., La’adan O., and Shiloh A. Scalable cluster computing with MOSIX for Linux. In Proceedings of Linux Expo ’99, pages 95–100, Raleigh, N.C., May 1999. 2. Anurag Acharya and Sanjeev Setia. Availability and Utility of Idle Memory on Workstation Clusters. In ACM SIGMETRICS Conference on Measuring and Modeling of Computer Systems, pages 35–46, May 1999. 3. T. Anderson, D. E. Culler, D. A. Patterson, and the NOW Team. A case for NOW (Networks of Workstations). IEEE Micro, Febuary 1999. 4. Remzi H. Arpaci, Andrea C. Dusseau, Amin M. Vahdat, Lok T. Liu, Thomas E. Anderson, and David A. Patterson. The Interaction of Parallel and Sequential Workloads on a Network of Workstations. In ACM SIGMETRICS Conference on Measurement and Modeling of Computer Systems, pages 267–278, 1995. 5. G. Bernard and S. Hamma. Remote Memory Paging in Networks of Workstations. In SUUG’94 Conference, April 1994. 6. Michael J. Feeley, William E. Morgan, Frederic H. Pighin, Anna R. Karlin, Henry M. Levy, and Chandramohan A. Thekkath. Implementing Global Memory Management in a Workstation Cluster. In 15th ACM Symposium on Operating Systems Principles, December 1995. 7. Michail D. Flouris and Evangelos P. Markatos. Network RAM, in High Performance Cluster Computing: Architectures and Systems, Chapt. 16. Prentice Hall, 1999. 8. Liviu Iftode, Karin Petersen, and Kai Li. Memory Servers for Multicomputers. In IEEE COMPCON’93 Conference, Febuary 1993. 9. John L. Hennessy and David A. Patterson. Computer Architectures A Quantitative Approach, 3rd Edition. Morgan Kaufman, 2002. 10. Evangelos P. Markatos and George Dramitinos. Implementation of a Reliable Remote Memory Pager. In USENIX 1996 Annual Technical Conference, 1996. 11. Li Xiao, Xiaodong Zhang, and Stefan A. Kubricht. Incorporating Job Migration and Network RAM to Share Cluster Memory Resources. In Ninth IEEE International Symposium on High Performance Distributed Computing (HPDC’00), 2000.
Low Overhead Agent Replication for the Reliable Mobile Agent System Taesoon Park and Ilsoo Byun Department of Computer Engineering Sejong University Seoul 143-737 KOREA {tspark,widepis}@sejong.ac.kr
Abstract. Fault-tolerance is an important design issue to build a mobile agent system satisfying the exactly-once execution property. Two fault tolerance schemes have been proposed. The replication scheme provides a higher degree of fault-tolerance, while the checkpointing scheme requires the lower overhead. In this paper, we analyze the performance of three optimization approaches to the agent replication and compare their overhead with the one of the checkpointing scheme. For the performance evaluation, optimized replication schemes and the checkpointing scheme have been implemented on top of the Aglet system. The experimental results show that the optimization can achieve up to 37% reduction on the agent migration time.
1
Introduction
A mobile agent moves over the system sites connected by a network, in order to perform an assigned task. During the execution or the migration, the agent may get lost if the execution site or the migration site fails. Also, it may get duplicated, if the related sites are not careful in recovery from the failure. To guarantee the correct execution of a mobile agent, it should not be lost or duplicated. For the exactly-once execution [7] of a mobile agent, many fault-tolerance schemes have been proposed. One approach is to use checkpointing [1,6,8]. On arrival of a new site, the agent saves its current states into a stable storage. In case of a site failure, the agent is re-activated from the checkpointed states, as if it just arrives to the site. The checkpointing scheme is usually implemented with two-phased agent migration scheme [3] to properly handle the failure during migration. The checkpointing scheme provides the fault-tolerance with a low cost. However, it may suffer from the long blocking time, since in this scheme, an agent can resume the execution only after the site recovers from the failure. To cope with the blocking problem, the replication scheme [2,5,7] is used. A mobile agent in this scheme is replicated and sent to 2k + 1 sites to tolerate k failures. Among 2k + 1 agents, one called a primary is responsible for the initial execution. When the primary fails, one of the other replicas takes over the execution. To detect failures of other replicas and prevent duplicate execution, H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 1170–1179, 2003. c Springer-Verlag Berlin Heidelberg 2003
Low Overhead Agent Replication for the Reliable Mobile Agent System
1171
the replicas perform the consensus at the end of each site execution. When the majority of the replicas agree on an execution site, one stage of the execution is successfully completed. Considering the degree of fault-tolerance, the replication scheme is more desirable. However, the cost of agent replication and consensus is considerable compared to the checkpointing cost. The performance of two fault-tolence schemes have been compared in [4,5,6]. Especially, in [4], we have suggested two optimization approaches to the agent replication and compared their performance with that of checkpointing. One approach is the asynchronous agent replication, in which the replicas are migrated to the designated sites in an asynchronous manner so that the primary can begin execution without waiting for the migration of other replicas. The other optimization is the usage of a consensus agent. In this approach, to tolerate k failures, k + 1 agent replicas for the possible execution and k consensus agent for the agreement are used. In this paper, we further analyze the effect of each optimization approach to the agent replication. We also suggest another optimization approach, the pipelined scheme, in which the replica migration sites are selected among the ones previously visited by the agent. Most of the agent replication time is taken by the agent migration time, which includes the time to move agent’s current states and the time to move agent’s class codes. By reusing the previously visited sites, migration time for class codes can be eliminated. To measure the effect of various optimization schemes, we implement the schemes on top of the mobile agent system, Aglets [3]. The performance results show that the various optimization schemes together can achieve up to 37% reduction on the replication overhead.
2
The Aglets System
The Aglets [3] is a Java-based mobile agent system. To support execution, migration and communication of agents, the system provides the AgletContext environment. The agents in the Aglets system is created inheriting properties and methods from the AgletClass and perform the event-driven activities. For the inter-agent communication, the message-passing mechanism is provided and the AgletProxy is used to support the location transparency of the mobile agent. The AgletProxy is an interface to the Aglet object and every message is sent to the Aglet object through the AgletProxy, regardless of its location. Two types of agent migration can be considered. One is the strong migration, in which agent codes, intermediate data and agent execution states are migrated. The other is the weak migration, in which only the agent codes and the intermediate data are migrated. As most of the Java-based mobile agent systems, the Aglets system supports the weak migration. For the migration, an agent is first serialized and then transferred. On the destination site, the agent is de-serialized and then activated. For the performance reason, the entire class codes for the agent are not migrated with the agent and the class codes are transferred only when the destination site does not have them.
1172
T. Park and I. Byun
For the reliable agent migration, ATP(Agent Transfer Protocol) is provided. The ATP is for the two-phased agent migration. The current execution site in the first phase sends the ATP request to the destination site, which sends back the ATP reply to notify the acceptance of the agent migration. The agent is then migrated within the ATP transfer message in the second phase. The destination site notifies the successful agent migration by sending the ATP Ack. The failstop failure model is assumed; that is, once a system component fails, it stops its execution and does not perform any malicious actions.
3 3.1
The Experimental System Agent Replication and Consensus
An agent in the mobile agent system migrates from a site to another site, in order to perform a task assigned by a user. The execution of a subtask at one system site is called a stage. In other words, the mobile agent execution consists of a sequence of stages and the agent migrates between two stages. In the replication scheme, an agent is replicated and the replicas are migrated to the 2k + 1 sites to execute one stage. With the 2k + 1 replicas, the system can tolerate up to k failures. Among the 2k + 1 replicas, one called a primary is responsible for the initial execution of a stage. The others are called observers, which may take over the stage execution in case that the primary fails. Fig. 1 describes the agent replication process when three replicas are used (k = 1). As shown in the figure, a primary residing in a site Pi−1 first sends the observers to the designated sites, Ri,1 and Ri,2 ; and then migrates to the execution site, Pi . In our experimental system, ATP protocol is used for the reliable agent migration. The AT P Begin and AT P Complete in the figure denote the initiation and the completion of the ATP protocol.
Ri,1 ATP_Begin
Ri,2 ATP_Begin
Pi ATP_Begin
Pi-1
ATP_Complete
ATP_Complete
xxxxxxxxxxx xxxxxxxx xxxxxxxxxxx xxxxxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxxxxx xxxxxxxxxxx xxxxxxxx xxxxxxxxxxx xxxxxxxx xxxxxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxxxxx xxxxxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxxxxx xxxxxxxxxxx xxxxxxxx xxxxxxxxxxx xxxxxxxx xxxxxxxxxxx xxxxxxxx xxxxxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxxxxx xxxxxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxxxxx xxxxxxxxxxx xxxxxxx xxxxxxxx xxxxxxxxxxx xxxxxxx xxxxxxxx xxxxxxxxxxx xxxxxxxx xxxxxxxxxxx xxxxxxx xxxxxxxx xxxxxxxx xxxxxxx xxxxxxxx xxxxxxxxxxx xxxxxxxx xxxxxxxxxxx xxxxxxx xxxxxxxx xxxxxxxx Consensus_Begin xxxxxxx xxxxxxxx xxxxxxxxxxx xxxxxxxx xxxxxxxxxxx xx xxxxxxxxxx xxxxxxx xxxxxxxx xxxxxxx xxxxxxxx xxxxxxxxxxx xxxxxxxx xxxxxxxxxxx xx xxxxxxxxxx xxxxxxx xxxxxxxx xxxxxxx xxxxxxxx xxxxxxxxxxx xxxxxxxx xxxxxxxxxxx Agent Execution xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxx xxxxxxx xxxxxxxx xxxxxxx xxxxxxxx xxxxxxxxxxx xxxxxxxx xxxxxxxxxxx Consensus_Ack xx xxxxxxxxxx xxxxxxx xxxxxxxx xxxxxxx xxxxxxxx xxxxxxxxxxx xxxxxxxx xxxxxxxxxxx x xxxxxxxxx xxxxxxx xxxxxxxx xxxxxxx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxx xxxxxxxxxxx xxxxxxxx xxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xx xxx x xxxxxxxxxxxxxxxxxxxxxxxxxxx xx Consensus xxx xx
ATP_Complete
xx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xx Agent Replication and Migration xx x
xx xx xx xx x
Fig. 1. Agent Replication and Consensus
Low Overhead Agent Replication for the Reliable Mobile Agent System
1173
On the completion of the execution, the primary at the site Pi begins the consensus process by sending the Consensus Begin messages to the observers. Each observer replies with the Consensus Ack message. The primary decides the successful completion when it receives the Consensus Ack messages from k observers. It then prepares for the agent replication for the next stage. When the primary fails, one of the observers takes over the execution. To decide which observer takes over the execution first, our experimental system follows the priority-based scheme proposed in [5]. In this scheme, the priority of each observer is predetermined and an observer may begin the execution only when all of the observers with the higher priority are known to fail.
3.2
Optimized Agent Replication
The first approach to optimize agent replication is the use of asynchronous agent replication. Fig. 2 describes the asynchronous agent replication process for three replicas. As shown in the figure, the primary migrates to the next site, Pi , first. And then, the other replicas are sent to the sites, Ri,1 and Ri,2 , by the previous primary at site Pi−1 . The new primary begins execution as soon as it arrives at Pi , and hence the agent execution at Pi is overlapped with the migration of the observers. Therefore, the migration time of observers can be compensated, if the agent execution time at Pi is long enough.
Ri,1
xxxxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxxxx xxxxxxxxxx xxxxxxxx xxxxxxxxxx xxxxxxxx xxxxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxxxx xxxxxxxxxx xxxxxxxx xxxxxxxxxx xxxxxxxx xxxxxxxxxx xxxxxxxx xxxxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxxxx xxxxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxxxx xxxxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxxxx xxxxxxxxxx xxxxxxxx xxxxxxxxxx xxxxxxxx xxxxxxxxxx xxxxxxx xxxxxxxx xxxxxxxx xxxxxxxxxx xxxxxxx xxxxxxxx xxxxxxxx xxxxxxxxxx xxxxxxx Consensus_Begin xxxxxxxx xxxxxxxx xxxxxxx ATP_Begin xxxxxxxx xxxxxxxxxxxxxxxxxx xxxxxxxxxx xxxxxxx xxxxxxxx xxxxxxxx xxxxxxx xxxxxxxx xxxxxxxxxxxxxxxxxx xxxxxxxxxx xxxxxxx x x xxxxxxxx xxxxxxxx xxxxxxxxxx xxxxxxx x xxxxxxxxx xxxxxxxxxxxxxxxxxx xxxxxxxxxx xxxxxxx x x xxxxxxxx xxxxxxxx xxxxxxx x xxxxxxxxx xxxxxxxxxxxxxxxxxx xxxxxxxxxx xxxxxxx Agent Execution x x xxxxxxxx xxxxxxxx xxxxxxx x xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxx xxxxxxxxxxxxxxxxxx xxxxxxxxxx xxxxxxx Consensus_Ack x xxxxxxxx xxxxxxxx xxxxxxx x xxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxx xxxxxxxxxx xxxxxxx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxx xxxxxxx xxxxxxxx xxxxxxxxxx xxxxxxx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx x x xxxxxxxxxxxxxxxxxxxxxxxx x x x x x Consensus xxx x x x x
ATP_Begin
Ri,2
Pi ATP_Begin
Pi-1
ATP_Complete
ATP_Complete
ATP_Complete
x x x xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx x x x x x x x Agent Replication and Migration x x x x x x
Fig. 2. Asynchronous Agent Replication
The implementation of asynchronous replication is however somewhat different from the one of synchronous replication. In the synchronous replication, the primary at Pi−1 first makes replicas by using the clone() method and sends them to the sites Ri,1 and Ri,2 . During this transfer, the primary obtains the AgletProxys for the replicas so that the primary can use them for the consensus, after it migrates to Pi and completes the execution.
1174
T. Park and I. Byun
However, if the primary migrates to Pi first in the asynchronous replication, all the threads for the primary at Pi−1 should be terminated and hence the transfer of the other replicas becomes impossible. Therefore, in the asynchronous replication, the first replica sent by the primary at Pi−1 becomes a new primary at Pi . The previous primary at Pi−1 continues copying two more replicas. After successfully transferring them, the primary at Pi−1 terminates itself. One more possible problem in the asynchronous scheme is that the new primary does not have AgletProxys for the observers. Hence, the previous primary has to transfer the AgletProxys of the observers to the new primary at Pi , before its termination. Another approach to optimize the agent replication is to use the consensus agent instead of copying the primary agent. In the replication scheme, 2k + 1 replicas are used to tolerate up to k failures. Among these, k + 1 replicas are for the possible execution which includes the alternative agent execution or the exception handling routines. However, the other k replicas are for attending the consensus only. Therefore, instead of copying 2k + 1 replicas, using the k agents carrying the small consensus codes can reduce the migration time, especially when the size of the original agent is large. The consensus agents have the same agent identifier and the same stage number as the other replicas, so that they can be treated as the replicas of the same agent. The pipelined replication is another approach to optimize the agent replication. Usually, the migration of an agent includes copying and transferring of the source codes of the agent and its intermediate data states. However, if an agent revisits a site which has been visited before, the site may have the source codes for the agent and hence it is enough to transfer the current data states. Using this property, in the pipelined replication, the set of migration sites for the observers are selected from the 2k recently visited sites. As a result, replication time and migration time for the source codes of an agent can partially be eliminated. To compare the optimized replication schemes with the checkpointing scheme, our experimental system also implements the checkpointing. To checkpoint the intermediate states of a primary agent, the snapshot() method is called when a primary arrives on a new site. The snapshot() method is to save the current states of an agent into the secondary storage. If the system detects the abnormal termination of an agent, the agent image saved in the secondary storage is reactivated. The saved snapshot is safely eliminated when the agent is successfully migrated to the next site or completed by the user.
4 4.1
Performance Study Experimental Setup
To evaluate each of the optimization approaches, replication schemes and the checkpointing scheme have been implemented on top of the Aglets system. For our experimental system, Aglet DSK 1.1b2 was used. For the agents to access
Low Overhead Agent Replication for the Reliable Mobile Agent System
1175
the appropriate replication methods, the Replication class was created under the AgletClass. Three replication classes were created under the Replication class. One is the AsynRep class to support the asynchronous agent replication. Another is the ConsRep class under which the consensus agents can be used. The PipeRep class is used for the pipelined selection of the execution sites of the observers. A cluster of six Pentium IV 1 GHz PCs connected through a 100 Mbps Ethernet was used for the experiments. Each machine supported an AgletContext and an agent traversed the sites in a predetermined order. The sites to execute the observers are randomly selected unless the pipelined replication is used. A task assigned to an agent consists of twenty stages and at each stage, the agent sleeps for one second instead of performing any action. For the stable performance, ten runs of the agent task were measured and then eight measured values were averaged out, excluding the lowest and the highest values. Also, to observe the influence of the agent size, we implemented a Sizer class, which iteratively inserts garbage values in the Vector class so that the size of the serialized agent object can be controlled. 4.2
Experimental Results
Fig. 3 first shows the portion of replication time and the consensus time out of the total execution time. This performance was obtained when no optimization was applied. In the figure, the replication time and the consensus time are denoted by Replication and Consensus, respectively. The actual execution time for the task is denoted by Sleep. Also, the notation, αP βk, denotes the agent execution with α replicas when the size of the agent is βKBytes.
60000
' ). 50000 s m (e 40000 im T no30000 it uc ex 20000 E tn eg A10000 0
Replication Consensus Sleep
3P1k
3P50k
3P100k
5P1k
5P50k
Fig. 3. Replication Time vs. Consensus Time
5P100k
1176
T. Park and I. Byun
As shown in the figure, the consensus time depends on the number of replicas. Because the consensus time consists of the time for broadcast the consensus message and the time for obtaining the majority of the replies, it is obvious that a large number replicas impose more consensus time. It is also obvious that there is not much changes in the consensus time, as the agent size increases. However, the replication time is greatly influenced by the number of replicas as well as the size of the agent. Since for the synchronous replication, the primary begins a new stage after the whole replica migration has been completed, the waiting time of the primary should be longer as the number of replicas or the agent size increases. As it is also noticed from the figure, the replication time takes larger portion of the total execution time compared to the consensus time. This can be the main motivation of the replication optimization. Fig. 4 shows the effect of each optimization approach to the agent replication. Fig. 4.(a) first shows the migration time of the agent when the asynchronous replication was applied. Besides the asynchronous replication, no other optimization was applied. The asynchronous replication achieves 16%–24% reduction of the migration time compared with the synchronous replication, when the number of replicas are three. In this case, the amount of reduction becomes larger as the agent size is increased. The possible reason is that most of the agent migration time can be overlapped with the execution time of the next stage. However, when the number of replicas is five, at most 12% reduction was achieved for the small agent. The reduction of the migration time for the large agent is negligible. Maybe, the agent migration in this case cannot be fully overlapped with the next stage execution since the number of agent and the agent size are too large. Such undesirable performance of the ”5P100k” case can be solved with the consensus agent. Fig. 4.(b) shows the effect of consensus agents. Besides the use of consensus agents, no other optimization was applied. Basically, using the consensus agent reduces the size of k replicas among 2k + 1 replicas. Hence, the performance gain must be larger as the value of k increases and also the agent size becomes increased. As it is expected, we can observe up to 29% reduction of the migration time when the number of replicas is five and the agent size is 100 KBytes. For the five replicas, two replicas are substituted by the consensus agents, which causes the large reduction. However, for the three replicas, there can be a reduction of one replica. As a result, there was not much benefit for the case ”3P1k”, especially when the agent size is small. For this case, the reduction is only 14%, which is even smaller than that of the asynchronous replication. Fig. 4.(c) shows another effect of the pipelined selection of the observer sites. Besides the use of pipelined optimization, no other optimization was applied. By choosing the previously visited sites for the observers, the migration time of class codes for the observer can be eliminated. Hence, the pipelined approach can benefit the agent which is computation intensive or carries small data. However, in our experimental setup, we intentionally inserted more data to increase the agent size and hence the large-sized agents can have only marginal benefits from this approach. The small agent, a large portion of which consists of source codes, can have the benefit from this approach. Fig. 4.(c) confirms this logic. As shown
Low Overhead Agent Replication for the Reliable Mobile Agent System
(a)
35000
' ). 30000 s 25000 (m e m iT 20000 no it 15000 rag i M tn 10000 eg A 5000 0
AsynchronousReplication Replication
3P1k
3P50k
3P100k
0
5P50k
5P100k
5P1k
5P50k
5P100k
5P1k
5P50k
5P100k
(b)
35000
' ). 30000 s m (e 25000 m iT 20000 no tia 15000 rg i M tn 10000 eg A 5000
5P1k
ConsensusAgent Replication
3P1k
3P50k
3P100k
(c)
35000
' ). 30000 s m (e 25000 m iT 20000 no tia 15000 rg i M tn 10000 eg A 5000 0
PipelinedReplication Replication
3P1k
3P50k
3P100k
Fig. 4. The Effect of Optimization Approach
1177
1178
T. Park and I. Byun
in the figure, the 100KBytes-sized agents achieve only 8% and 9% reduction of the migration time when the number of replicas is three and five, respectively. However, for the 1KBytes agents, the reduction of the migration time becomes 19% and 25% when the number of replicas is three and five, respectively. This is the most reduction for the 1KByte agent.
90000 80000
' ). 70000 s m (e 60000 im T 50000 no it ar 40000 ig M tn 30000 eg20000 A 10000 0
1k
50k
100k
200k
Agent Size(byte)
NoFaultTolerance Replication(k=3) OptimizedReplication(k=5)
300k
400k
Checkpointing OptimizedReplication(k=3) Replication(k=5)
Fig. 5. The Performance of Optimized Replication
Now, Fig. 5 shows the effects of three optimization approaches. When all three optimization schemes were implemented, the system at least achieved 19% and 24% reduction of the migration time for three and five replicas, respectively. The reduction becomes larger as the size of the agent is increased. For 100KBytes agents, 30% and 37% reduction of migration time was achieved for three and five replicas. When the agent size becomes 400KBytes, 59% and 61% of migration time reduction was achieved. By applying three optimization approaches together, the system becomes possible to have the best performance provided by each approach. However, as it is noticed in the figure, there is still the performance gap between the replication scheme and the checkpointing scheme. Consider the case that agent size is 1KByte and three replicas are used. The difference in the migration time of two fault-tolerance schemes is 7103 ms. However, among this time, the inter-agent consensus takes 3805 ms. In other words, the replication itself incurs only 3298 ms. extra time, which is almost 64% reduction on the replication time. The reduction becomes larger as the number of replicas and
Low Overhead Agent Replication for the Reliable Mobile Agent System
1179
the agent size are increased. Therefore, we can conclude that using three optimization approaches, the system may achieve possibly the best replication time.
5
Conclusions
In this paper, we have presented three optimization approaches to the agent replication. In the presented schemes, the migration and the execution of the primary agent proceeds asynchronously with the agent replication. Also, to reduce the replica migration time, the use of small-sized consensus agents was considered and the pipelined selection of the observer sites was used to reduce the time for the source code migration. To evaluate the performance of the optimization approaches, we have implemented the various replication schemes on top of the Aglets system and measured the performance. The performance results show that the optimized replication achieves 37% reduction on the replication overhead, when the agent size was 100KBytes and five replicas were used. Also, the cost reduction can be enlarged as the agent size and the number of replicas are increased. Acknowledgments. This work was supported by grant No. (R04-2002-00020102-0) from the Basic Research Program of the Korea Science & Engineering Foundation.
References 1. Gendelman, E., Bic, L.F., Dillencourt, M.B.: An Application-Transparent, PlatformIndependent Approach to Rollback-recovery for Mobile Agent Systems. Proc. of the 20th Int’l Conf. on Distributed Computing Systems (2000). 2. Johansen, D., Marzullo, K., Schneider, F.B., Jacobsen, K.: NAP: Practical FaultTolerance for Itinerant Computations. Proc. of the 10th Int’l Conf. on Distributed Computing Systems (1999). 3. Karjoth, G., Lange, D.B., Oshima, M.: A Security Model for Aglets. IEEE Internet Computing (1997). 4. Park, T., Byun, I., Kim, H., Yeom, H.Y.: The Performance of Checkpointing and Replication Schemes for Fault Tolerant Mobile Agent Systems. Proc. of the 21st Symp. on Reliable Distributed Systems (2002) 256–261. 5. Pleisch, S., Schiper, A.: FATOMAS - A Fault-Tolerant Mobile Agent System Based on the Agent-Dependent Approach. Proc. of the Int’l Conf. on Dependable Systems and Networks (2001) 215–224. 6. Silva, L., Batista, V., Silva, J.G.: Fault-Tolerant Execution of Mobile Agents. Proc. of the Int’l Conf. on Dependable Systems and Networks (2000). 7. Strasser, M., Rothermel, K.: Reliability Concepts for Mobile Agents. International Journal of Cooperative Information Systems, Vol. 7, No. 4 (1998) 355-382. 8. Strasser, M., Rothermel, K.: System Mechanism for Partial Rollback of Mobile Agent Execution. Proc. of the 20th Int’l Conf. on Distributed Computing Systems (2000).
A Transparent Software Distributed Shared Memory Emil-Dan Kohn and Assaf Schuster Computer Science Department, Technion {emild,assaf}@cs.technion.ac.il
Abstract. Software Distributed Shared Memory (SDSM) systems use clusters to provide yet another level of scalability to multi-threaded shared-memory applications. However, linking with SDSM libraries usually requires adaptation of the program’s system calls to the SDSM specific APIs, alignment of program variables to page boundary, in-depth verification of the program against the SDSM memory model, transforming global variables to be dynamically allocated, and more of the like. In this work we present the transparent SDSM - an SDSM that can efficiently execute any multi-threaded program (given in binary compiled form). The memory model of the transparent SDSM correctly supports any shared-memory application, whether programmed using relaxed or strict consistency in mind. By presenting a prototype and measurements, we show that the performance of the transparent SDSM is not compromised, essentially matching that of a non-transparent high-performance SDSM.
1
Introduction
The programming interface offered by most software distributed shared memory (SDSM) systems consists of a header file and a library. In order to use the services provided by a standard SDSM system, an application must link with the SDSM system library. At runtime, the application calls a special function which initializes the SDSM system. After that, the application accesses the services offered by the SDSM system via the application programming interface (API) exported by the SDSM system library. Unfortunately, the APIs provided by SDSM systems are not standard. Many SDSM systems present incompatible APIs and different models for the behavior of the shared memory. Porting an existing application to an SDSM system, or from one SDSM system to another is thus a tedious and error-prone process. Moreover, such an approach assumes that the source code of the application is available, which is not always the case. On the other hand many modern operating systems support the multithreading programming paradigm. A multi-threaded application consists of several execution threads which run in parallel and are scheduled independently on the available processors. Each thread has its own stack and CPU register set, but threads belonging to the same process share a common address space. H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 1180–1189, 2003. c Springer-Verlag Berlin Heidelberg 2003
A Transparent Software Distributed Shared Memory
1181
Thus several threads from the same process can communicate efficiently through global variables. The multi-threaded programming model is intuitive and easy to use. Multi-threading is available on most operating systems via portable, standardized APIs (POSIX on UNIX-like systems, Win32 on Windows systems) or is directly supported by modern high-level languages such as Java. The standard execution environments for parallel multi-threaded applications are symmetric multi-processor (SMP) machines. The major disadvantage of SMP machines is that they are not easily extendible, and that their maximum number of processors is fairly limited. This poses a serious limitation on the maximum scalability that can be achieved by these machines. On the other hand SDSM systems and computing clusters in general can be easily extended, and are far less limited in the maximum number of computing nodes than SMP machines. Thus, their scalability can be much better than that of the SMP machines. In this paper we show the design and implementation of an SDSM system which combines the convenience of the multi-threaded programming model with the extra level of scalability offered by computing clusters. Our SDSM system allows running standard multi-threaded applications on a computing cluster and lets them take advantage of the computation power of all the nodes in the cluster. We assume that the applications are available only in compiled (binary) form; no assumptions whatsoever are made about the availability of the applications source code. The SDSM system is completely transparent to the applications. In order to achieve transparency, the SDSM system must emulate the execution environment of standard multi-threaded applications. Our approach for emulating the execution environment of multi-threaded applications is based on a fine granularity page-based SDSM system, which uses strict consistency. We use binary interception for intercepting the calls the application makes to the operating system/C standard library and redirecting them to the SDSM system library. However, intercepting OS calls is not sufficient to achieve full transparency. First, multi-threaded applications share data via global variables, and thus the global variables must be managed by the SDSM as well. Second, while for most SDSM systems initialization is a trivial task which is accomplished by calling a special function at the start of the application execution, this simple solution cannot be used for a transparent SDSM system running standard multi-threaded binary applications. The initialization code must be injected somehow into the application after it has been loaded. Finally, memory allocation, commonly optimized for on-the-fly calls in multi-threaded applications, is traditionally triggered at the beginning of execution (out of the critical path for “speedup measurements”) for SDSM benchmarks. Clearly, the transparent SDSM must match the light-weight allocation capabilities assumed by multi-threaded applications. It turned out not easy for a base SDSM system to meet the performance, correctness and portability attributes challenged by the transparent SDSM: it must tolerate false sharing, it must support the standard multi-threaded memory model, and it must allow for efficient intra-node multi-threading. Many SDSM systems do not have good answers to this combination: page-based systems tend
1182
E.-D. Kohn and A. Schuster
to use relaxed consistency to avoid false sharing, and relaxed consistency does not preserve the multi-threading memory model; page-based systems must share protection between threads thus corrupting complex, carefully-optimized SDSM protocols; non page-based systems, on the other hand, do not fully avoid false sharing, impose instrumentation overhead, and tend to be machine-architecture dependent. We have thus implemented a prototype of the transparent SDSM system, by adapting the Millipede 4.0 SDSM [6,8]. Millipede 4.0 can share memory in application-tailored granules, thus efficiently avoiding false sharing. In this way, Millipede 4.0 matches the performance of relaxed consistency SDSMs, despite using the OS protection mechanism, and despite providing strict consistency with an exceptionally thin-protocol. The results show that the transparent SDSM matches the performance of Millipede 4.0, with a negligible (non-measurable) overhead.
2
Intercepting the Operating System API
One of the key requirements for making our system fully transparent is the ability to intercept all the relevant calls the user application makes to the operating system and/or to the standard C library and redirect these calls to the SDSM system library. Unlike many SDSM systems, we assume nothing about the availability of the source code of the original application. Therefore we have to intercept the calls made by the application at the binary code level. Binary Interception. We start by setting up a common terminology: Target: the original operating system call/library function that we want to intercept; Detour: The function that should be invoked whenever the application attempts to invoke the target. Trampoline: All the techniques described below allow calling the original system call/C library function. Often the detour invokes the original target as a subroutine and performs some extra processing. The mechanism which makes this possible is called a trampoline. The main difference between the different interception techniques is in the mechanism to construct the the trampoline. The idea behind the technique that we use, called Target Function Rewriting [5], is to replace the first bytes of the target by a jump instruction to the detour. This way whenever the application code tries to invoke the target, control will be transferred to the detour because of the jump instruction. Creating the trampoline is a little bit more complicated. It involves saving aside the bytes that are going to be overwritten by the jump instruction and appending a new jump instruction to the rest of the original target code. This technique has several advantages over other techniques. First, it does not assume that the target resides in a DLL (as do IAT-rewrite techniques). Second, it does not require knowing all functions in the target DLL (as do proxy DLL techniques) Third, it is portable, as it has no special requirements from the underlying platform (such as import tables). Finally, the required disassembler
A Transparent Software Distributed Shared Memory
1183
is very simple (much simpler than that required by naive code instrumentation techniques). The Intercepted Operating System API Subset. The operating system API subset that is intercepted includes thread management, dynamic memory management, synchronization, and I/O. Using binary interception, these functions are replaced by their counterparts from the SDSM system library. Threads are created using the SDSM system thread creation API. The SDSM system makes sure that a handle to a thread object is valid on all the nodes of the SDSM system. Dynamic memory is allocated from the SDSM area. A pointer returned by the memory allocation function is valid on all the nodes. The synchronization primitives are also global in that they involve the threads on all the nodes of the SDSM system, not just the threads on a single node. For example an SDSM lock guarantees mutual exclusion for all the threads running in the SDSM system, regardless of their location. Virtual Handles. When a program running on a SDSM system requests a resource, ultimately the request will propagate via the SDSM system library to the local operating system running on a certain node, which will return a handle to that resource. If we aim for full transparency, this resource should be usable – via its handle – by threads running on nodes other than the node on which the resource has been allocated, in the same way as a resource allocated by one thread in a multi-threaded environment can be used by all the other threads belonging to the same process. However, the standard resource handle returned by the operating system is valid only on the node that made the request. Moreover, it is possible for two handles to two different resources which have been allocated on two different nodes to have an identical bit pattern, because each node runs its own copy of operating system. In this case, there is no way for the SDSM system to distinguish between these two resources. In order to overcome these problems, we have devised the virtual handles mechanism. Resource handles returned by the SDSM system library functions (which replace the operating system API functions using binary interception) have a special bit pattern which makes them distinct from standard operating system resource handles, and are guaranteed to be unique across all the SDSM nodes. These handles are called virtual handles. The bit pattern of the virtual handles is chosen such that it is straightforward for functions from the SDSM system to locate the actual operating system resource handle, as well as the node on which the handle is valid. Since the application programs do not care about the bit pattern of the resource handles, the virtual handles mechanism is completely transparent to them.
3
Moving the Global Variables to the SDSM
Most SDSM systems provide shared memory to applications only via a malloc()-like API function. However, for full transparency, this is not sufficient. The threads of an ordinary multi-threaded application can share data through global variables, whereas typical SDSM systems allow sharing only via dynami-
1184
E.-D. Kohn and A. Schuster
cally allocated variables. Therefore we have to provide a means for sharing the global variables among the nodes of our system. Since we want to run our applications without any re-compilation, the only reasonable solution is to map the global variables to the shared memory. The Executable Loading Process. Most modern operating systems use memory mapping for loading executable files into memory. The executable file mirrors the in-memory image of a process. An executable file consists of several parts, called sections. When loading an executable, the operating system loader maps each section separately and sets the protection attributes accordingly. For example the .text section (also known as .code section on other systems) which contains the code of the application is typically mapped with read +execute permissions. The sections which are of more interest to us are .data, which contains all the initialized global variables, and .bss which contains all the uninitialized global variables. Both sections are mapped with read+write permission by the loader. Typically, the .bss section does not occupy space in the executable. The loader allocates the necessary space only when the executable file is loaded. Moving the Global Variables to the Shared Memory. The SDSM system on which we base our implementation uses the MultiView technique in order to achieve high performance and eliminate false sharing [6]. A brief introduction to MultiView can be found in section 6. In MultiView, each memory page is mapped several times to the address space of the process using it. Each such mapping is called a view. The SDSM system uses one special view, called the privileged view for updating the contents of the shared memory. All the other views, called user views are used by the application. The SDSM system adjusts the access permissions of the user views according to the consistency protocol. The crux in the implementation is to exploit the fact that both the operating system loader and the MultiView technique use memory mapping. The memory mappings performed by the operating system on the .data and .bss sections of the executable are used as user views. The privileged view is allocated by the SDSM system at initialization. Unfortunately, because of the Copy On Write mechanism (COW), which is applied by the OS on the global variable sections (.bss and .data), mapping the privileged view is not easy. Simply setting up a privileged view, and using it for accessing the global variables will result in their copy to a different location, which would invalidate the OS-created view. Our solution is to copy the sections containing global variables to a location that is not COW-protected, and then remap the OS view. The details of this mechanism can be found in the full version.
4
System Initialization
Most SDSM systems require some sort of initialization to be performed before a user application can use their services. These SDSM systems provide an initialization function in their API. User programs have to call this function before
A Transparent Software Distributed Shared Memory
1185
calling any other function from the SDSM system API. Typically the call to the SDSM initialization function is placed at the beginning of the main() function in the application source. Unfortunately this simple initialization procedure cannot be used by fully transparent SDSM systems, because these systems run ordinary multi-threaded applications that have already been compiled for the underlying platform. These applications are aware neither of the SDSM system nor of its initialization function, because these applications have not been linked against the SDSM system library. Therefore we have to devise a more sophisticated technique for the transparent SDSM system initialization, which is described in this section. For all C/C++ programs, main() is considered the program entry point. However, main() is not the function that is invoked immediately after the operating system loader has loaded the executable file into memory. Before main() gets invoked, the program has to perform several initialization tasks. After an executable has been loaded into memory, control is passed to the entry point of the executable. The address of the entry point function is written in the header of the executable file, so it can be easily located by the loader. The entry point performs the following tasks: It initializes the C/C++ runtime library. This includes opening the standard I/O streams and initializing the dynamic memory management data structures. It retrieves the command line arguments, parses them if necessary and invokes the main() function. It invokes the exit() function, passing it the return value of main() as a parameter. The code of the entry point and the initialization function resides in the C/C++ runtime library which any C/C++ program is linked with. Fortunately, the code between the entry point and the invocation of main() is fixed and does not depend at all on the application being executed. While the address of main() may change, the relative address of the jump to subroutine instruction which invokes main() is fixed. In other words, the instruction invoking main() lies at a fixed offset after the program entry point. Based on this observation, we can devise the following procedure for the transparent SDSM system initialization: (1) Using a platform specific mechanism, we arrange that the program main execution thread gets suspended immediately after the executable has been loaded into memory. (2) Using a platform specific mechanism, we “forcefully” link the executable with the DLL that contains the code of the SDSM system library. This will cause the entry point of the DLL to be executed, which will perform the following actions: (2.a) Obtains the address of the entry point by examining the executable header. (2.b) From the address of the entry point, it obtains the address of the jump to subroutine instruction invoking main(). This address is located at a fixed offset from the address of the entry point. The address of the main() function is also obtained during this step from the target of the jump to subroutine instruction. (2.c) Overwrites the target address of the jump to subroutine instruction invoking main() with the address of a new function which does the following: (2.c.i) Invokes the SDSM initialization function. (2.c.ii) If the id of the current node is zero, creates a thread using the SDSM system API with main() being its
1186
E.-D. Kohn and A. Schuster
start function; otherwise, the current thread is put to sleep. (3) We resume the execution of the main thread that has been suspended during step (1) again by using a platform specific mechanism. The platform specific mechanisms will be detailed in the full version. After Step (3) completes the application will execute the code from substeps (2.c.i) and (2.c.ii) instead of the usual main(). The SDSM system initialization function will be invoked on each node. This procedure performs the interception of the Win32/standard C library calls, in addition to the usual initialization tasks of a standard SDSM system. On node zero, a thread will be created starting at the main() function. Note that this thread is started on node zero only, and not on all nodes as with non-transparent SDSM systems, since the transparent SDSM must emulate the behavior of standard multi-threaded applications that begin with only a single main thread. The main thread must be created with the SDSM system thread creation API and not with the operating system API, because we want the main thread to behave like any other thread from the SDSM system (for example, its page fault handlers must be designated the SDSM fault handlers). In turn, the threads created from within the main thread will be created using the SDSM system thread creation API, due to the interception of the operating system thread creation API during the initialization of the SDSM system.
5
Optimizing Memory Allocation for Small Objects
Many SDSM systems use simple, sometimes even stop-the-world like memory allocators. Such allocators are easier to design, but their performance can be very poor. A typical simple memory allocator works like this: When a node wants to allocate memory, it sends a message to the memory allocation server, indicating the size of the memory area it wishes to allocate. The memory allocation server performs the allocation locally and broadcasts the result of the allocation (start address and size) to all the nodes in the SDSM system. Upon receiving the message from the allocation server, each node updates its memory management data structures accordingly and acks. In addition to this, the node that requested the allocation also returns the result to the caller. This kind of solution is unacceptable for a transparent SDSM system. While the benchmarks for most SDSM systems are written such that they allocate the whole memory at the beginning of their execution and distribute it to all nodes, typical multi-threaded object-oriented programs perform many relatively smallsized allocations throughout their execution. Since transparent SDSM systems do not run applications specifically written for them, but rather they run ordinary multi-threaded programs, the memory allocator should be adapted to the behavior of such programs. In order to satisfy the performance demands of a transparent SDSM system, we have devised a new memory allocator which is better suited to the memory behavior of real object-oriented programs. This new allocator minimizes the
A Transparent Software Distributed Shared Memory
1187
communication that takes place during memory allocation, thereby significantly improving its performance. In fact, no communication takes place at all during memory allocation. While the implementation of this scheme seems straightforward on other SDSM systems, this is not the case for MultiView. An initial allocation of views should be computed, and the virtual address space should be split according to the allocation. However, because this complication is specific to the MultiView method, its details are out of the scope of this paper. The interested reader can find them in [4].
6
Performance Evaluation
In this section we give an overall evaluation of the transparent SDSM system runtime performance. Our test-bed consists of a cluster of Pentium PCs, interconnected by a Myrinet LAN. Each node runs Windows NT 4.0 as the underlying operating system. We have based the implementation on the Millipede 4.0 SDSM system, using the MultiView technique. The MultiView Technique and the Millipede 4.0 SDSM System. MultiView [6] is a technique to eliminate false sharing completely, despite using the memory page protection mechanism. The basic idea is to use the underlying operating system’s virtual address mapping mechanism to map each shared variable to its own virtual page. This way, the access permissions of a variable do not affect the access permissions of other variables. A physical page is mapped at several different locations in the virtual address space, one for each variable. These mappings are called views. User programs access each variable v through its separate view, using a designated ptrv pointer which points to the corresponding view. The SDSM system can set the protection of each view according to the consistency protocol, without affecting the protection of other views. Using this technique, Millipede 4.0 can achieve runtime performance comparable to systems employing relaxed consistency, but without posing any restrictions to the application programmer. Performance Evaluation. In order to run the benchmarks on the transparent SDSM, we had to modify their code slightly. We had to completely decouple the application code from the SDSM library with whom the original benchmarks had to be linked, and convert the benchmarks into ordinary multi-threaded Win32 applications. This involves the following main steps: (1) Calls to dsm malloc() – the shared memory allocation function – are replaced with calls to the standard C malloc() function. (2) SDSM locks are replaced by Win32 CRITICAL SECTION synchronization primitives. (3) Since the barrier synchronization primitive has no direct Win32 equivalent, we had to emulate it using Win32 API functions. We have experimented with two implementations, one based on a combination of Win32 events and critical sections, and another one based on a server thread and Win32 messages. (4) The calls to the dsm distribute() function, which used to scatter data to all the nodes of
1188
E.-D. Kohn and A. Schuster
the Millipede 4.0 system, are no longer necessary, because global variables are shared among all the nodes, so these calls are removed. (5) The explicit call to dsm init() – Millipede 4.0’s initialization function – is no longer necessary, because the transparent SDSM initializes itself automatically before the execution of main() as described in section 4. After these transformations, the benchmarks become stand-alone Win32 multi-threaded applications which no longer require linking with the Millipede 4.0 library. The results are shown in figure 1. Executing the stand-alone Win32 versions of the benchmarks on the transparent SDSM system yields speed-ups comparable to those obtained by the original Millipede 4.0 system. The differences in speed-ups between the transparent SDSM system and the classical Millipede 4.0 SDSM system are less than 10%.
IS Speedup
SOR speedup
7
8 7
6 Speedup
Transparent SDSM Millipede 4.0 Transparent+Barrier SMP (2 processors)
4 3 2
Speedup
6
5
Transparent SDSM Millipede 4.0 Transparent+Barrier SMP (2 processors)
5 4 3 2
1
1
0
0 0
2
4
6
8
10
Number of threads
(a) IS speed-up comparison
0
2
4
6
8
10
Number of threads
(b) SOR speed-up comparison
Fig. 1. Speed-up comparisons. Other benchmarks show similar behavior (see full version).
The main sources of overhead are the implementation of the barriers and the extra code executed by the API interception layer, though the latter is negligible. Since the barriers are implemented using ordinary Win32 function calls (Win32 has no native support for barrier synchronization primitives), the number of messages passed around is much higher than in the Millipede 4.0 implementation. In order to illustrate this, we have written a version of the barriers library which has the same interface as the library used by the transparent SDSM system, but instead of being based on the Win32 API only, invokes the original Millipede 4.0 implementation behind the scenes. The results of running the benchmarks with this version of the barrier library are shown by the “Transparent+Barrier” graphs in figure 1. We notice that the run-time performance of this version is extremely close to the original Millipede 4.0.
7
Related Work
Relaxed consistency SDSM systems, such as Munin[1,3], TreadMarks[7], and Midway[2], place restrictions on application programs to ensure their correct
A Transparent Software Distributed Shared Memory
1189
execution (e.g., no intentional data races). In contrast, our transparent SDSM supports a strict memory model. Thus, the transparent SDSM can execute correctly any program, regardless of the memory model that the programmer had in mind. In particular, any multithreaded program can be executed correctly. Instrumentation-based SDSMs, such as Blizzard[10] and Shasta[9], modify the executable by wrapping each load and store instruction with code that implements the consistency protocol. This approach has two main disadvantages. First, the overhead incurred by the extra code to check all memory accesses can be quite significant, and requires aggressive optimization techniques for its minimization. Second, the technique is architecture-specific and thus not portable. In contrast, our page-based SDSM system achieves fine sharing granularity without the overhead associated with code instrumentation. Furthermore, despite using instrumentation, our use of the target rewriting technique is relatively portable. As proven by our results, applying it on the operating system calls incurs only negligible overhead.
References 1. J. K. Bennett, J. B. Carter, and W. Zwaenepoel. Munin: Distributed shared memory based on type-specific memory coherence. In Proc. of the Sixth ACM SIGPLAN Symp. on Principles and Practice of Parallel Programming (PPOPP’97), pages 90–99, June 1997. 2. B. N. Bershad, M. J. Zekauskas, and W. A Sawdon. The midway distributed system. In Proc. of 38th IEEE Int’l Computer Conf. (COMPCON Spring ’93), pages 528–537, February 1993. 3. J. B. Carter. Design of the munin distributed shared memory system. Journal of Parallel and Distributed Computing, 29(2):219–227, September 1995. 4. E.-D. Kohn. A Transparent Software Distributed Shared Memory, 2002. M.Sc. thesis, Computer Science Department, Technion. 5. G. Hunt and D. Brubacher. Detours: Binary interception of win32 functions. In 3rd USENIX Windows NT Symposium. Microsoft Research, July 1999. 6. A. Itzkovitz and A. Schuster. Multiview and millipage – fine-grain sharing in pagebased dsms. In Proc. of the 1999 Intl’ Conf. on Parallel Processing (ICPP ’99), pages 220–227, September 1999. 7. P. Kelehrer, S. Dwarkadas, A. L. Cox, and W. Zwaenepoel. Treadmarks: Distributed shared memory on standard workstations and operating systems. In Proc. of the Winter 1994 USENIX Conference, pages 115–131, January 1994. 8. N. Nitzann and A. Schuster. Transparent Adaptation of Sharing Granularity in MultiView-based DSM Systems. Software, Practice & Experience, 31:1439–1459, October 2001. (Prelim. version (Best Paper Award) in proc. IPDPS, San Francisco, April 2001). 9. D. J. Scales, K. Gharachorloo, and C. A. Thekkath. Shasta: A low overhead, software-only approach for supporting fine-grain shared memory. In Proc. of the 7th Symp. on Architectural Support for Programming Languages and Operating Systems (ASPLOS VII), pages 174–185, October 1996. 10. I. Schoinas, B. Falsafi, A. R. Lebeck, S. K. Reinhardt, J. R. Larus, and D. A. Wood. Fine-grain access control for distributed shared memory. In Proc. of the 6th Symp. on Architectural Support for Programming Languages and Operating Systems (ASPLOS VI), pages 297–307, October 1994.
On the Characterization of Distributed Virtual Environment Systems Pedro Morillo1 , Juan M. Ordu˜na1 , M. Fern´andez1 , and J. Duato2 1
Departamento de Inform´atica. Universidad de Valencia. SPAIN 2 DISCA. Universidad Polit´ecnica de Valencia. SPAIN {Pedro.Morillo,Juan.Orduna}@uv.es,
[email protected] Abstract. Distributed Virtual Environment systems have experienced a spectacular growth last years. One of the key issues in the design of scalable and costeffective DVE systems is the partitioning problem. This problem consists of efficiently assigning clients (3-D avatars) to the servers in the system, and some techniques have been already proposed for solving it. In this paper, we propose the correlation of the quality function proposed in the literature for solving the partitioning problem with the performance of DVE systems. Since the results show an absence of correlation, we also propose the experimental characterization of DVE systems. The results show that the reason for that absence of correlation is the non-linear behavior of DVE systems with the number of avatars in the system. Also, the results show that workload balancing mainly has an effect on system throughput, while minimizing the amount of inter-server messages mainly has an effect on system latency.
1
Introduction
Professional high performance graphic cards currently offer a very good frame-rate for rendering complex 3D scenes in real time. On other hand, fast Internet connections have become worldwide available at a relatively low cost. These two factors have made possible the current growth of Distributed Virtual Environment(DVE) Systems. These systems allow multiple users, working on different computers that are interconnected through different networks (and even through Internet) to interact in a shared virtual world. This is achieved by rendering images of the environment as if they were perceived by the user. Each user is represented in the shared virtual environment by an entity called avatar, whose state is controlled by the user input. Since DVE systems support visual interactions between multiple avatars, every change in each avatar must be propagated to the rest of the avatars in the shared virtual environment. DVE systems are currently used in many different applications, such as collaborative design, civil and military distributed training, e-learning or multi-player games. One of the key issues in the design of a scalable DVE system is the partitioning problem [6]. It consists of efficiently assigning the workload (avatars) among different servers in the system. The partitioning problem determines the overall performance of the DVE system, since it has an effect not only on the workload each server in the
Supported by the Spanish MCYT under Grant TIC2000-1151-C07-04
H. Kosch, L. B¨osz¨orm´enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 1190–1198, 2003. c Springer-Verlag Berlin Heidelberg 2003
On the Characterization of Distributed Virtual Environment Systems
1191
system is assigned to, but also on the inter-server communications (and therefore on the network traffic). Some methods for solving the partitioning problem have been already proposed [6,7]. However, there are still some features in the proposed methods that can be improved. For example, the quality function proposed in the literature must be correlated with system performance, in order to design actually scalable and efficient partitioning strategies. In this paper, we present the experimental correlation of the quality function proposed in the literature with the performance of DVE systems. Since the results show an absence of correlation, we also propose the experimental characterization of DVE systems, in order to analyze the reasons for that absence of correlation. This characterization study measures the impact of different parameters of the quality function on the performance of DVE systems, and it shows that the behavior of DVE systems is non-linear with the number of avatars in the system. Therefore, in order to design an actually scalable partitioning method for DVE systems, a different (non-linear) partitioning strategy must be proposed. The rest of the paper is organized as follows: Section 2 describes the partitioning problem and the existing proposals for solving it. Section 3 details the proposed characterization setup that allows to experimentally study the behavior DVE systems. Next, Section 4 presents the correlation and evaluation results. Finally, Section 5 presents some concluding remarks and future work to be done.
2 The Partitioning Problem in DVE Systems Architectures based on networked servers are becoming a de-facto standard for DVE systems [7,6]. In these architectures, the control of the simulation relies on several interconnected servers. Multi-platform client computers join the DVE system when they are connected to one of these servers. When a client modifies an avatar, it also sends an updating message to its server, that in turn must propagate this message to other servers and clients. Servers must render different 3D models, perform positional updates of avatars and transfer control information among different clients. Thus, each new avatar represents an increasing in both the computational requirements of the application and also in the amount of network traffic. When the number of connected clients increases, the number of updating messages must be limited in order to avoid a message outburst. In this sense, concepts like areas of influence (AOI) [7], locales [1] or auras [4] have been proposed for limiting the number of neighboring avatars that a given avatar must communicate with. Lui and Chan have shown the key role of finding a good assignment of clients to servers in order to ensure both a good frame rate and a minimum network traffic in DVE systems [6]. They propose a quality function, denoted as Cp , for evaluating each assignment of clients to servers. This quality function takes into account two parameters. One of them consists of the computing workload generated by clients in the DVE system, denoted as CpW . In order to minimize this parameter, the computing workload should be proportionally shared among all the servers in the DVE system, according to the computing resources of each server. The other parameter of the quality function consists of the overall inter-server communication requirements, denoted as CpL . In order to
1192
P. Morillo et al.
minimize this parameter, avatars sharing the same AOI should be assigned to the same server. Quality function Cp is defined as Cp = W1 CpW + W2 CpL
(1)
where W1 + W2 = 1. W1 and W2 are two coefficients that weight the relative importance of the computational and communication workload, respectively. These coefficients should be tuned according to the specific features of each DVE system. Using this quality function (and assuming W1 = W2 = 0.5) Lui and Chan propose a partitioning algorithm that re-assigns clients to servers [6]. The partitioning algorithm should be periodically executed for adapting the partition to the current state of the DVE system (avatars can join or leave the DVE system at any time, and they can also move everywhere within the simulated virtual world). Lui and Chan also have proposed a testing platform for the performance evaluation of DVE systems, as well as a parallelization of the partitioning algorithm [6]. The partitioning method proposed by Lui and Chan currently provides the best results for DVE systems. However, the correlation of the quality function Cp with DVE system performance should be studied, and parameters W1 and W2 must be properly tuned. In this sense, the characterization of DVE systems is crucial in order to design partitioning strategies that actually improves the scalability of DVE systems.
3
Characterization Setup
We propose the characterization of generic DVE systems by simulation. The evaluation methodology used is based on the main standards for modeling collaborative virtual environments, FIPA [3], DIS [2] and HLA [5]. We have developed a simulation tool (a program written in C++) that models the behavior of a generic DVE system with a network-server architecture. Concretely, we have implemented a set of multi-threaded servers. Each thread in a server uses blocking sockets for communicating with a client. Each client simulates the behavior of a single avatar, and it is also implemented as a multi-threaded application. One of the threads of the client manages the communication with the server assigned to that client, and another thread manages user information (current position, network latency, etc.). Our simulator model is composed of a set of S interconnected servers and n avatars. Following the approach specified in FIPA and HLA standards, one of the servers acts as the main server (called Agent Name Service [3] or Federation Manager [5]) and manages the whole system. The main server also maintains a partitioning file for assigning a given server to each new avatar. Avatars can join the simulation through this main server, that assigns each new avatar to one of the servers in the system. In each simulation, all avatars sharing the same AOI must communicate between them for notifying their position in the 3D virtual world.The message structure used for notifying avatar movements is the Avatar Data Unit (ADU) specified by DIS [2]. A simulation consists of each avatar performing 100 movements, at a rate of one movement every 2 seconds. Each time an avatar performs a movement, he notifies that movement to his server by sending a message with a timestamp. That server must then notify that movement to all the avatars in the same AOI of the sender avatar. When
On the Characterization of Distributed Virtual Environment Systems
1193
that notification arrives to these avatars, they return an ACK message to the server, that in turn propagates that ACK messages to the sender avatar. When an ACK message arrives, the sender avatar computes the round-trip delay for communicating with each neighbor avatar. We have denoted this round-trip delay (measured in real-time) as the system response. When a simulation ends, each avatar has computed the average system response for the avatars in its AOI. At this point, all avatars send these average system responses to their respective servers, and each server then computes its own average system response. Finally, starting from the average value provided by each server the main server computes the average system response for the entire system, denoted as the average system response (ASR). An ASR value is computed after each simulation. An actually scalable DVE system must keep this measure as low as possible as the number of avatars in the system increases. On other hand, the system throughput is given by the maximum number of avatars that the system can manage while keeping the ASR below a certain threshold. Therefore, we have considered ASR as the main performance measure for characterizing the behavior of DVE systems. In order to evaluate the performance of each partitioning method, usually 3 different distributions of avatars in the virtual world are proposed in the literature: uniform, skewed and clustered distributions of avatars. However, both the movement rate and also the AOI of avatars can also be adjusted in order to make that workload to be independent of the distribution of avatars in the virtual world. Therefore, for the sake of simplicity we have only considered a uniform distribution of avatars, with the same AOI and the rate of movements for all the avatars in the system.
4
Simulation and Correlation Results
In this section, we present the correlation and simulation results obtained for the DVE model described in the previous section. We have tested a great number of different DVE configurations, ranging from small virtual worlds (3 servers and 180 avatars) to large virtual worlds (900 avatars and 6 servers). Since we have obtained very similar results in both of them, we present the results for a small DVE configuration. Figure 1 shows the performance results for a small DVE system composed of 3 servers and 180 avatars when different partitions (showing different values of the quality function Cp ) are simulated. This Figure shows on the X-axis the values of Cp obtained for different partitions (assignments of avatars to the servers in the system). The Y-axis shows ASR values for the simulations performed with these partitions. Each point in the plot represents the average value of the ASR obtained after 30 simulations of the same DVE system. The standard deviation for any of the points shown in the plot was not higher than 25 ms. in any case. This Figure clearly shows that Cp does not correlate with DVE system performance. ASR ranges in a 5% of its initial value (sometimes decreasing) as Cp values are greatly increased. Thus, for example, the ASR value obtained for a partition with a value of Cp equal to 800 (the worst value of Cp ) is even lower (better) than the ASR values obtained for partitions with Cp values ranging from 400 to 700. There not exists a linear correspondence between the values of the quality function Cp and ASR. Therefore, a characterization study is needed in order to capture the behavior of DVE systems.
1194
P. Morillo et al.
Fig. 1. Correlation of quality function Cp with average system response
In order to make a methodical characterization study, we have studied the behavior of DVE systems as the two terms of the sum in equation 1 vary. Given a DVE system composed of 3 servers, first we have optimized the value of CpW (workload balancing) and we have simulated both a partition with optimum (minimized) value of CpL (interserver communication requirements) and also a partition with the worst value of CpL as possible. Also, for the same DVE system we have unbalanced the partitions to obtain the worst value of CpW as possible, performing the same simulations with the partitions showing both the best and the worst values of the term CpL . Figure 2 shows the obtained ASR values for a DVE configuration of 3 servers as the number of avatars in the system increases. The value of Cp associated to the simulated partitions was zero in all the cases (perfect workload balancing and absence of interserver messages, that is, CpL and CpW equal to zero). ASR seems to be invariant with the number of avatars (ASR plot has the shape of a flat line) until it reaches a saturation point. From that saturation point, ASR values greatly increase as new avatars are added to the simulation. These results clearly show that the behavior of DVE system is non linear with the number of avatars in the system. Since Cp is defined (equation 1) as a linear function of both workload balancing and inter-server communications, this non-linear behavior can explain the absence of correlation shown in Figure 1. In order to determine the reason for the non-linear behavior of DVE systems, Table 1 shows the CPU utilization and the average system response in milliseconds (SR-Sx) obtained for each server corresponding to the simulation results shown in Figure 2, as well as the corresponding global ASR value. This table shows that the DVE system reaches the saturation point when any of the servers in the system reaches a CPU utilization of 100%. All the SR-Sx values increase as the respective CPU utilization so does, and there are not significant differences between the three SR-Sx values while the CPU utilization remains under 99% (simulation with 500 avatars). When CPU reaches more than 99% in any of the servers then the system response for that server greatly increases,
On the Characterization of Distributed Virtual Environment Systems
1195
Fig. 2. Average system response as the number of avatars per server increases
also increasing global ASR accordingly. Therefore, we can conclude that the non-linear behavior of DVE systems shown in Figure 2 is due to the limit of 100% in any CPU utilization. Since quality function Cp does not take into account CPU utilization in order to measure the quality of a partition, it cannot take into account the non-linear behavior of DVE systems as the number of avatar increases. Table 1. Average system responses and server CPU utilization in simulations shown in Figure 2 No. of avatars %CPU1 %CPU2 %CPU3 SR-S1 SR-S2 SR-S3 Average SR 350 (116,117,117) 51 54 52 193.21 191.81 196.43 193.82 400 (133,133,134) 67 69 71 249.29 245.60 247.36 247.17 450 (150,150,150) 78 76 78 279.43 281.12 280.91 280.49 500 (166,167,167) 96 99 100 545.73 892.54 1775.85 1138.04 550 (183,183,184) 100 100 100 3021.44 2811.45 2982.12 2938.34 600 (200,200,200) 100 100 100 4356.91 3645.21 3409.20 3803.77 650 (216,217,217) 100 100 100 5320.65 5114.32 5112.22 5182.40
These results means that although a partition can provide perfect workload balancing and minimum inter-server communication requirements (thus providing a optimum value of Cp , as the partitions in Figure 2 do), ASR will be very poor if the amount of avatars assigned to any server requires a CPU utilization of 99% or more. Also, given a DVE system and given a certain amount of avatars in the system, the performance of that DVE system will remain practically invariant with Cp if none of the servers reaches a CPU utilization of 99%. This is the case for the simulations whose results are shown in Figure 1, where only 90% of CPU utilization was reached in the worst case.
1196
P. Morillo et al.
Figure 2 shows the behavior of the DVE with the best partitions as possible. Table 2 shows the same measurements of Table 1 taken on the same DVE when simulating partitions with the worst CpL as possible (the maximum number of inter-server messages). When comparing these two tables in a line-by-line analysis, we can see that a reduction in the amount of inter-server messages (Table 1) results in a fixed reduction in ASR while the system is not saturated (the difference between ASR values of both tables in the first 3 lines remains constant, about 40 ms.). However, the effect of inter-server messages in ASR becomes more important as servers approach saturation. Also, we can see that for the same number of avatars Table 2 shows higher CPU utilization. Therefore, in order to maximize the throughput of a DVE system (the amount of avatars a given DVE system can manage without enter saturation) inter-server messages must also be minimized.
Table 2. Average system responses and server CPU utilization in simulations shown in Figure 2 No. of avatars %CPU1 %CPU2 %CPU3 SR-S1 SR-S2 SR-S3 Average SR 350 (116,117,117) 61 64 65 221.45 214.62 217.81 217.96 400 (133,133,134) 76 75 79 281.41 291.31 289.91 287.94 450 (150,150,150) 80 82 83 350.13 333.73 342.39 342.08 500 (166,167,167) 98 100 100 942.45 1792.54 1675.85 1470.28 550 (183,183,184) 100 100 100 3674.56 3844.17 3364.22 3627.68 600 (200,200,200) 100 100 100 4991.34 4565.71 4398.83 4651.96 650 (216,217,217) 100 100 100 6541.34 6199.45 5853.37 6198.05
Additionally, Tables 3 and 4 show CPU utilization and server system responses for the same DVE system (composed of 3 servers) when the term CpW is maximized (worsened). The partitioning strategy whose results are shown in Table 3 minimizes interserver communication requirements (the term CpL ), while the partition whose results are shown in Table 4 maximizes this term. In a line-by-line comparison of both tables, we can see that CPU utilization are very similar in both tables. This result shows that the term CpL does not have any effect on the system throughput when the term CpW is worsened (the system reaches saturation with 425 avatars in both tables). However, the system responses obtained for each server greatly differs between the two tables.
Table 3. Average system response and CPU utilization for each server when CpL is minimized No. of avatars %CPU1 %CPU2 %CPU3 SR-S1 SR-S2 SR-S3 Average SR 375 (125,125,125) 72 74 71 221.23 233.57 225.71 226.84 400 (125,150,125) 71 81 73 219.36 285.21 218.77 241.11 425 (125,175,125) 73 100 70 228.74 1725.84 235.54 730.04 450 (125,200,125) 72 100 69 225.67 6308.87 218.99 2251.18 475 (125,225,125) 70 100 72 226.54 11554.86 221.42 4000.94
On the Characterization of Distributed Virtual Environment Systems
1197
A comparison of Tables 3 and 4 with Tables 1 and 2 shows that minimizing the term CpW results in improving the throughput (the scalability) of DVE systems. Effectively, the same DVE system reaches saturation with 500 avatars in Tables 1 and 2, while reaching saturation with only 425 avatars in Tables 3 and 4. However, there not exists any significant difference in ASR values between Tables 1 and Tables 3 in the first two lines, neither between Tables 2 and 4 on the first two lines. These results show that the term CpW does not have an important effect on ASR. On the contrary, the term CpL mainly affects to ASR values (system latency), and this effect is amplified when the term CpW is worsened. However, if the term CpW is optimized then the impact of CpL in ASR is minimized. Table 4. Average system response and CPU utilization for each server when CpL is maximized No. of avatars %CPU1 %CPU2 %CPU3 SR-S1 SR-S2 SR-S3 Average SR 375 (125,125,125) 69 72 73 217.89 241.13 233.45 230.82 400 (125,150,125) 71 83 72 251.41 323.22 218.31 264.31 425 (125,175,125) 69 100 73 1290.21 2992.85 1432.80 1905.29 450 (125,200,125) 72 100 70 5678.51 8861.91 6007.82 6849.41 475 (125,225,125) 71 100 72 9998.83 16873.21 10254.31 12375.45
5
Conclusions
In this paper, we have proposed the experimental correlation of the quality function proposed in the literature for solving the partitioning problem with the performance of DVE systems. Since results show an absence of correlation, we have also proposed a characterization study of DVE systems. DVE systems show a non-linear behavior with the number of avatars in the system. Average system response (round trip-delay of messages notifying movements of avatars) remains practically invariant with the number of avatars in the system until the DVE system reaches a saturation point. This saturation point is given by the limit of a CPU utilization of 100% in any of the servers. When this limit is reached, then average system response greatly increases as new avatars are added to the system. We have also studied the effects of the two terms of the quality function proposed in the literature (workload balancing and the amount of inter-server messages) on the performance of DVE systems. The results show that workload balancing mainly has an effect on system throughput, while the amount of inter-server messages mainly has an effect on system latency. However, if none of the servers reaches saturation then the amount of inter-server messages does not have any significant effect on system latency, regardless of the amount of inter-server messages. Therefore, in order to design an efficient and scalable DVE system, the partitioning method should be targeted to balance the workload among the servers in the system in such a way that none of them reaches 100% of CPU utilization.
1198
P. Morillo et al.
References 1. D.B.Anderson, J.W.Barrus, J.H.Howard, “Building multi-user interactive multimedia environments at MERL”, in IEEE Multimedia, 2(4), pp. 77–82, Winter 1995. 2. DIS. 1278.1 IEEE Standard for Distributed Interactive Simulation-Application Protocols (ANSI). DMSO. DoD High Level Architecture. 1997. 3. FIPA Agent Management Specification. Foundation for Intelligent Physical Agents, 2000. Available at http://www.fipa.org/specs/fipa00023/ 4. J.C.Hu, I.Pyarali, D.C.Schmidt, “Measuring the Impact of Event Dispatching and Concurrency Models on Web Server Performance Over High-Speed Networks”, Proc. of the 2nd. IEEE Global Internet Conference, November.1997. 5. Kuhl, F., Weatherly, R, Dahmann, J., “Creating Computer Simulation Systems: An Introduction to the High Level Architecture”, Prentice-Hall PTR, Upper Saddle River, NJ, 1999. 6. Jonh C.S. Lui, M.F. Chan, “An Efficient Partitioning Algorithm for Distributed Virtual Environment Systems”, IEEE Trans. Parallel and Distributed Systems, Vol. 13, March 2002 7. S.Singhal, and M.Zyda, “Networked Virtual Environments”, ACM Press, New York, 1999.
A Proxy Placement Algorithm for the Adaptive Multimedia Server Bal´azs Goldschmidt and Zolt´an L´ aszl´o Budapest University of Technology and Economics {balage|laszlo}@iit.bme.hu
Abstract. Multimedia services are becoming widespread, while the network capacity can not keep up with the growth of user demands. Usually proxies are used to overcome the QoS degradation. One of the fundamental problems of applying proxies is to recognize the relevant set of proxy placement criteria, and to find a good algorithm. In this paper we propose a greedy algorithm variant, ams-greedy, that aims at taking into account the different property preferences of the proxy-server and the proxy-client links. We also analyse the consequences of the initial results. Keywords: Greedy algorithm, optimization, proxies, distributed multimedia.
1
Introduction
There is an increasing demand for on-line multimedia services, like video-ondemand. But as the number of the clients grows, and their distance from the servers is growing, the QoS (bitrate, jitter, etc.) of the video streaming decreases. The Adaptive Multimedia Server (AMS [1]) uses proxies, or data collectors, that collect the video from servers and stream them to the clients. The novelty is that the proxies can be dynamically migrated to new hosts where the provided QoS can be better. This kind of placement of the proxies is a special, dynamic case of the classical load balancing problem, and greedy algorithms have already proved to be a good candidate in similar optimization problems. In this paper we suggest a greedy algorithm variant and examine the experiences. The rest of the paper is organized as follows. In section 2 a short introduction is given into the Adaptive Multimedia Server and the Host Recommender it relies upon, in section 3 our greedy algorithm variant is described, in section 4 we introduce the test environment and analyse the results, and finally in section 5 we conclude our work and propose future steps.
Work partially supported by the European Community under the IST programme – Future and Emerging Technologies, contract IST-1999-14191 – EASYCOMP. The authors are solely responsible for the content of this paper. It does not represent the opinion of the European Community, and the European Community is not responsible for any use that might be made of data appearing therein.
H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 1199–1206, 2003. c Springer-Verlag Berlin Heidelberg 2003
1200
2
B. Goldschmidt and Z. L´ aszl´ o
The AMS and the Host Recommender
The Adaptive Multimedia Server (AMS) is a part of a major project at the University Klagenfurt, that has the aim of creating a distributed virtual videoon-demand service that supports proactive adaptation techniques. [2] The AMS comprises cluster manager, data manager (DM), data collector (DC), and client. When a client wants to see a movie, it contacts the cluster manager, and asks, if the video is available and serviceable to him. If the demand is admitted, the client is redirected to a data collector that will serve him using video streaming. The video is initially stored on the data managers, as stripe units, thus providing a better load balance. The task of the data collector is to collect the stripe units and weave them together. Usually the connection to the data managers is TCP-based, thus correct delivery of data is guaranteed. When the video is collected, the streaming to the clients uses the UDP-based RTP protocol, that does not guarantee correct data delivery. However, for a video stream, proper timing is usually more important than error-free delivery. The novel idea was to place the data collectors dynamically according to (future) client needs. For example when a semester starts the students at the dormitory might want to see some videos displaying courses they want to attend. During this period it would be useful to have the data collectors (or even the data managers) near the dormitory. In other times of the year, however, the data collectors might be put somewhere else, and thus the nodes are available for other tasks. The means of dynamic placement is a mobile agent based infrastructure, Vagabond2 [3,1], that comprises two major modules. One of them is the application module, that enables the migration of CORBA applications written in Java. The other module is the host recommender. The host recommender uses statistical information about the nodes and the network, and also hints on what the client demands will be. Using this information it recommends hosts that can serve as data collectors. The host recommender has to implement such a data collector placement algorithm, that can give nearly optimal results for different user requirements. These requirements, like low network load and low jitter, are usually contradictory. When testing the algorithm we had to check how the QoS varies in different conditions.
3
The ams-greedy Algorithm
Proxy and cache placement is a heavily researched topic with a large number of papers published [4,5,6]. We can not use these results without modifications, because they only investigate static placement. Moreover, the goals of a web proxy and our data collector are also different. In case of the web proxy or a general data replicator the clients download lots of small files, and the aim is to provide the most pages without contacting the web server, and with guaranteed data delivery. In case of the data collector usually one huge media file is transmitted to the clients, while the jitter, network load, etc. has to be minimized. There is usually no harm in losing some smaller parts.
A Proxy Placement Algorithm for the Adaptive Multimedia Server
1201
The basic problem is the following: we have several clients, several data managers, and we have to place the data collectors (the ‘proxies’) in a way that satisfies clients’ demand, such as bitrate’s and jitter’s remaining inside an allowable range. It’s an extra advantage, if we can keep the network load low. Phrasing the problem that way it is obvious, that it is a special type of the NP-hard facility location and k-median problems, a well studied area where lots of papers have been published [7,8,9]. Most of the papers propose some variation of the greedy algorithm [10]. We have developed a new, specialized variant, called ams-greedy. Greedy algorithms usually have a cost function, and a set of elements. The goal is to build such a subset that has enough elements to cover some problem. When building this subset, the greedy algorithm selects the next element with the lowest cost from the original set. When the subset gives a complete cover, the algorithm stops. The cost function of ams-greedy is the sum of the latencies between each client and the DC closest to it (weighted by wcl ), plus the sum of the latencies between each DC and the DM closest to it (weighted by wdm ). We decided to use the latency of the network links, because this is the most easily measurable property, and this property can be modelled the easiest way. Based on the evaluation of the test results, we plan to incorporate other link properties, like jitter and bandwidth, as well. In the cost function the effective parameter is the ratio of the above weights. This parameter is a consequence of the asymmetric kind of the problem: the connection of the DCs to the DMs is TCP-based, the packages can not be lost, while the DCs’ connection to the clients is UDP-based, thus the packages are sent only once, and may be lost. The algorithm starts with an empty set of data collector nodes, with infinite cost. In every iteration the set is extended with that node that would provide the smallest cost if added to the set. The nodes that are not needed by any client are removed from the set. If the new set costs less than the one the iteration started with, a next iteration starts. Otherwise the result is the final set of the previous iteration. (For a formal description see table 1.)
DM Client 1
6
1
1 1
Client 2
7
0
1
2
1
3
1
4
1
1
8
Client 3
1
9
Client 4
5
1
Fig. 1. An example network topology
On figure 1 a small example is shown. On node 0 there is a data manager, on nodes 6, 7, 8, and 9 there are clients. The latency on every link is 1. Let us follow the algorithm choosing node sets for the data collectors. The weights wcl and wdm are both set to 1.
1202
B. Goldschmidt and Z. L´ aszl´ o Table 1. The ams-greedy algorithm
// // // //
nodes is the set of nodes; lat is the latency between any too nodes clients is the set of clients; dms is the set of DMs, dcs is the set of DCs wcl is the weight of the latency between DC and client wdm is the weight of the latency between DC and DM
function cost(nodes, lat, clients, dms, dcs, wcl , wdm ) if (dcs == {}) return ∞ fi ret := 0 foreach c ∈ clients do ret += wcl ∗ mind∈dcs (lat(c, d)) od foreach m ∈ dms do ret += wdm ∗ mind∈dcs (lat(m, d)) od return ret end function select next(nodes, lat, clients, dms, dcs, wcl , wdm ) cand := nodes \ (clients ∪ dms ∪ dcs) let d ∈ cand : ∃c ∈ cand : (cost(nodes, lat, clients, dms, dcs ∪ {c}) < cost(nodes, lat, clients, dms, dcs ∪ {d})) return dcs ∪ {d} end algorithm ams-greedy(nodes, lat, clients, dms, dcs, wcl , wdm ) dcs := {} repeat dcs := select next(nodes, lat, clients, dms, dcs, wcl , wdm ) dcs := {d ∈ dcs | ∃c ∈ clients ∃d ∈ dcs : (lat(d , c) < lat(d, c))} if (cost(nodes, lat, clients, dms, dcs) > cost(nodes, lat, clients, dms, dcs )) dcs := dcs fi until dcs unchanged end
In the first iteration the set of data collectors is empty. The algorithm computes for every free node (1-5) the cost of placing the data collector there. The costs are cost({1}) = wcl × (lat(1, 6) + lat(1, 7) + lat(1, 8) + lat(1, 9)) + wdm × lat(1, 0) = 15 cost({2}) = 14;
cost({3}) = 13;
cost({4}) = 14;
cost({5}) = 15
With the smallest cost, set {3} is selected. In the second iteration the initial set of data collectors is {3}, the free nodes are 1, 2, 4, and 5. The costs of the possible new sets are: cost({1, 3}) = 12;
cost({2, 3}) = 13;
cost({3, 4}) = 13;
cost({3, 5}) = 12
The first and last set have the smallest cost, now the first is selected. Using this set, client 1 and 2 are connected to node 1, client 3 and 4 to node 3, no node
A Proxy Placement Algorithm for the Adaptive Multimedia Server
1203
is removed from the set. The set of DCs at the end of this iteration is {1, 3}, because it costs less then the previous set, {1}. In the third iteration the initial set is {1, 3}. The costs of the possible extensions are cost({1, 2, 3}) = 14;
cost({1, 3, 4}) = 12;
cost({1, 3, 5}) = 11
Using the last set client 1 and 2 are connected to node 1, client 3 and 4 to node 5. Node 3 is not used by any client, therefore it is removed. The new set is {1, 5}, with cost 10. In the fourth iteration the initial set is {1, 5}. The costs of the possible extended sets are cost({1, 2, 5}) = 12;
cost({1, 3, 5}) = 11;
cost({1, 4, 5}) = 12
The one with the smallest cost (11) is ({1, 3, 5}), same as above. But in this set node 3 is deleted again, the result is once more {1, 5}. The cost did not change, thus the algorithm ends. The result is that we have two data collectors: client 1 and 2 are served by a collector on node 1, client 3 and 4 by another on node 5.
4
The Test Results
We have decided to use a network simulator to test our algorithm. We have chosen the freely available NS-2 [11], that provides substantial support for simulation of TCP, UDP, routing, and multicast protocols over wired and wireless (local and satellite) networks. In the simulator we implemented the following model: the system consists of data managers, data collectors, and clients. Every client is connected to exactly one DC, and every DC is connected to exactly one DM. When started, the collector using a TCP connection requests data from the DM it is connected to, and fills up its own buffer. When the buffer is full, it starts to send UDP packets to the clients periodically. When the transmission ends, the result is stored in a log file, and using this log we can analyse the data. The test session contained 10 runs, each with a new network topology that was created randomly using the GT-ITM network topology generator[12]. GTITM created a network of 500 nodes in a transit-stub graph, where the link latencies were also chosen by the generator. Every link had 0.5Mb/sec bandwidth. In each run we had randomly chosen a node for the DM and 20 other nodes for clients. Each run consisted of eight turns; in the first seven the amsgreedy was used with the wdm /wcl ratio ranging from 1/8 to 8. In the eighth turn 20 (the same number as for the clients) random nodes had been chosen for the DCs, and each client had been assigned to the DC closest to it. In each turn the data manager sent a 1 Mbyte video to the clients via the data collectors. When the buffer of the collector had at most 250 kbyte, or the bitrate of the TCP packets from the DM reached 64 kbit/sec, the DC had started to stream at 64 kbit/sec to the clients assigned to it. We measured the number of collectors chosen by the greedy algorithm (table 2), the number of data bytes carried by TCP packets, by UDP packets and by
1204
B. Goldschmidt and Z. L´ aszl´ o
both in the network (fig. 2(a)), and we also measured the standard deviation and the maximum of the delay jitter on the clients (fig. 2(b)). The figures present the average of the result values of turns with the same ratio. In table 2 the improvement of ams-greedy compared to the random algorithm can be seen in percents. Values less than zero mean deterioration. Comparing the ams-greedy algorithm to the random one we can see that when the number of active DCs are near equal (about 6.1) there is a significant improvement both in the jitter and the number of packets used. Table 2. The average number of active data collectors and the improvement of jitter and network load 1 8
wdm /wcl
1 4
1 2
No. of DCs (greedy) 17.1 14.1 No. of DCs (random)
8.5
1
2
4.3 2.5 6.1
4
8
1.5
1.2
jitter (std dev) [%] 89.42 83.33 70.9 54.69 9.39 -60.27 -167.75 jitter (max) [%] 83.90 75.22 55.79 31.07 12.19 -11.34 -17.59 tcp+udp packets [%] -23.1 -3.38 29.31 44.05 43.9 43.55 42.11
180
1.8 TCP UDP UDP+TCP
160 140
1.6 1.2
100
[sec]
[Mbyte]
120 80
1 0.8
60
0.6
40
0.4
20
0.2
0 0.125 0.25
Maximum Standard deviation
1.4
0.5
1
2
4
8
0 0.125 0.25
wdm /wcl
0.5
1
2
4
8
wdm /wcl
(a) Cumulated number of data bytes in different kinds of packets
(b) Delay jitter
Fig. 2. Measurement results
The wdm /wcl ratio can be interpreted as a force that moves the DCs towards the data managers (when the ratio is more than one) or towards the clients (when it is less than one). The average number of the DCs as a function of the ratio can be seen on table 2: ams-greedy can place more DCs near the clients than around a DM. The maximum and the deviation of the delay jitter is getting smaller as we get closer to the clients: the route of the UDP streams is shorter, the delay is smaller, and thus the jitter is decreasing.
A Proxy Placement Algorithm for the Adaptive Multimedia Server
1205
The most important result is the amount of the network packets (network load) in the function of the ratio. As the ratio decreases, the number of TCP packets increases, because the collectors are not only getting closer to the clients, but we have also placed more of them. The inverse function can be seen in the case of UDP packets, although the change is less drastical, because the number of packets is only effected by the length of the DC-client connections. Analysing the total number of packets in the network, we can assume that the optimum is about where the ratio equals one. Getting the DCs closer to the clients the growth of the network load is fast, but moving the DCs to the DM the increase is very moderate. It implies, that if the network load should be low, the ratio has to remain near one, i.e. we have to weight the DC-DM and the DC-client latencies similarly. These result show, that the fine-tuning of the parameters of the algorithm is necessary, and it can lead to significantly reduced network load and delay jitter. Further examinations are needed, however, to provide a more relevant set of network properties that can make the algorithm more effective.
5
Conclusion and Future Plans
In this paper we introduced a greedy-variant algorithm for the host recommender of the Adaptive Multimedia Server. With a simple simulation model we proved that proper parametrization of the algorithm is very important, it may result in such a placement of the data collectors that highly reduces network load and delay jitter. In the future we plan to improve both the algorithm and the model based on the above results. First, the model should be tested with more than one DMs, and we have to introduce network noise, so that the algorithm could be tested in a more realistic network environment. The model should be made ready to handle multicast RTP streaming, and then its results should be compared to those we gathered by unicast streaming. Later the model ought to be tested also in a real network environment. Second, the algorithm has to be improved to support not only latency, but other network link parameters as well, like jitter, bandwidth, or even statistical values like average and maximum jitter, average bandwidth, etc. The algorithm should also be improved to take startup latencies into account. Later on it could support even the migration of the data managers. Acknowledgements. The work related to this paper has been performed in the IKTA-00026/2001, Dynamic broker service for improving quality of database access and resource usage project, which is supported by the Hungarian Ministry of Education in the tender for Information and Communication Technology Applications. The work was partially supported by the European Community under the IST programme – Future and Emerging Technologies, contract IST-1999-14191 – EASYCOMP.
1206
B. Goldschmidt and Z. L´ aszl´ o
The work was also partially supported by the Research and Development Division of the Ministry of Education, Hungary and its Austrian partner the Bundesministerium f¨ ur Ausw¨ artige Angelegenheiten.
References 1. Goldschmidt, B., Tusch, R., B¨ osz¨ orm´enyi, L.: A mobile agent-based infrastructure for an adaptive multimedia server. In: 4th DAPSYS (Austrian-Hungarian Workshop on Distributed and Parallel Systems), Kluwer Academic Publishers (2002) 141–148 2. B¨ osz¨ orm´enyi, L., D¨ oller, M., Hellwagner, H., Kosch, H., Libsie, M., Schojer, P.: Comprehensive treatment of adaptation in distributed multimedia systems in the admits project. In: Proceedings of the 10th ACM International Conference on Multimedia, ACM Press (2002) 3. Goldschmidt, B.: Vagabond2: an infrastructure for the adaptive multimedia server. Technical Report TR/ITEC/02/2.11, Institute of Information Technology University Klagenfurt (2002) 4. Qiu, L., Padmanabhan, V.N., Voelker, G.M.: On the placement of web server replicas. In: INFOCOM. (2001) 1587–1596 5. Barford, P., Cai, J.Y., Gast, J.: Cache placement methods based on client demand clustering. Technical Report TR1437, UW (2001) 6. Xu, J., Li, B., Lee, D.L.: Placement problems for transparent data replication proxy services. IEEE Journal on Selected Areas in Communications 20, No. 7 (2002) 7. Mirchandani, P.B., Francis, R.L., eds.: Discrete Location Theory. John Wiley and Sons (1990) 8. Drezner, Z., ed.: Facility Location: a Survey of Applications and Methods. Springer Verlag (1996) 9. Chudak, F.A., Williamson, D.P.: Improved approximation algorithms for capacitated facility location problems. Lecture Notes in Computer Science 1610 (1999) 99–104 10. Guha, Khuller: Greedy strikes back: Improved facility location algorithms. In: SODA: ACM-SIAM Symposium on Discrete Algorithms (A Conference on Theoretical and Experimental Analysis of Discrete Algorithms). (1998) 11. http://www.isi.edu/nsnam/ns: The network simulator – ns-2 (2003) 12. Zegura, E.: http://www.cc.gatech.edu/fac/ellen.zegura/graphs.html (2003)
A New Distributed JVM for Cluster Computing 1
1
2
Marcelo Lobosco , Anderson Silva , Orlando Loques , and Claudio L. de Amorim
1
1
Laboratório de Computação Paralela, PESC, COPPE, UFRJ Bloco I-2000, Centro de Tecnologia, Cidade Universitária, Rio de Janeiro, Brazil { lobosco, faustino, amorim }@cos.ufrj.br 2 Instituto de Computação, Universidade Federal Fluminense o Rua Passo da Pátria, 156, Bloco E, 3 Andar, Boa Viagem, Niterói, Brazil
[email protected] Abstract. In this work, we introduce CoJVM, a new distributed Java run-time system that enables concurrent Java programs to efficiently execute on clusters of personal computers or workstations. CoJVM implements Java’s shared memo-ry model by enabling multiple standard JVMs to work cooperatively and transpa-rently to support a single distributed shared-memory across the cluster’s nodes. CoJVM requires no change to applications written in standard Java. Our experi-mental results using several Java benchmarks show that CoJVM performance is considerable with speed-ups ranging from 6.1 to 7.8 for an 8node cluster.
1 Introduction One of the most interesting features of Java [1] is its embedded support for concurrent programming. Java provides a native parallel programming model that includes support for multithreading and defines a common memory area, called the heap, which is shared among all threads that the program creates. To treat race conditions during con-current accesses to the shared memory, Java offers to the programmer a set of synchronization primitives, which are based on an adaptation of the classic monitor model. The development of parallel applications using Java’s concurrency model is restricted to shared-memory computers, which are often expensive and do not scale easily. A com-promise solution is the use of clusters of personal computers or workstations. In this case, the programmer has to ignore Java’s support for concurrent programming and instead use a message-passing protocol to establish communication between threads. However, changing to message-passing programming is often less convenient and even more complex to code development and maintenance. To address this problem, new distributed Java environments have been proposed. In common, the basic idea is to extend the Java heap among the nodes of the cluster, using a distributed shared-memory approach. So far, only few proposals have been implemented, and even less are compliant with the Java Virtual Language Specification [2]. Yet, very few reported good performance results [9] and presented detailed performance analysis [17]. In this paper, we introduce the design and present performance results of a new Java environment for high-performance computing, which we called the CoJVM H. Kosch, L. Böszörményi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 1207–1215, 2003. © Springer-Verlag Berlin Heidelberg 2003
1208
M. Lobosco et al.
(Cooperative Java Virtual Machine) [3]. CoJVM’s main objective is to speed up Java applications executing on homogeneous computer clusters. CoJVM relies on two key features to improve application performance: 1) the HLRC software Distributed Shared Memory (DSM) protocol [4] and 2) a new instrumentation mechanism [3] to the Java Virtual Machine (JVM) that enables new latency-tolerance techniques to exploit the application run-time behavior. Most importantly, the syntax and the semantics of the Java language are preserved, allowing programmers to write applications in the same way they write concurrent programs for the single standard JVM. In this work, we evaluate CoJVM performance for five parallel applications: Matrix Multiplication (MM), SOR, LU, FFT, and Radix. The connected figures show that all benchmarks we tested achieved good speedups, which demonstrate CoJVM effectiveness. Our main contributions are: a) to show that CoJVM is an effective alternative to improve performance of parallel Java applications for cluster computing, b) to demonstrate that scalable performance of Java applications for clusters can be achieved without any syntax or semantic change in the language; and c) to present detailed performance analysis of CoJVM for five parallel benchmarks. The remainder of this paper is organized as follows. Section 2 presents the HLRC software DSM system. Section 3 describes Java support for multithreading and synchronization, and the Java memory model. In section 4, we review some key design concepts of CoJVM. In section 5, we analyze performance results of five Java parallel applications executed under CoJVM. In section 6, we describe some related works. Finally, in section 7, we draw our conclusions and outline ongoing works.
2 Software DSM Software DSM systems provide the shared memory abstraction on a cluster of physically distributed computers. This illusion is often achieved through the use of the virtu-al memory protection mechanism [5]. However, using the virtual memory mechanism has two main shortcomings: (a) occurrence of false sharing and fragmentation due to the use of the large virtual page as the unit of coherence, which leads to unnecessary communication traffic; and (b) high OS costs of treating page faults and crossing pro-tection boundaries. Several relaxed memory models, such as LRC [6], have been pro-posed to alleviate false sharing. In LRC, shared pages are write-protected so that when a processor attempts to write to a shared page an interrupt will occur and a clean copy of the page, called the twin, is built and the page is released to write. In this way, modi-fications to the page, called diffs, can be obtained at any time by comparing current co-py with its twin. LRC imposes to the programmer the use of two explicit synchroniza-tion primitives: acquire and release. In LRC, coherence messages are delayed until an acquire is performed by a processor. When an acquire operation is executed the acqui-rer receives from the last acquirer all the write-notices, which correspond to modifica-tions made to the pages that the acquirer has not seen according to the happen-before-1 partial order [6]. HLRC introduced the concept of home node, in which each node is responsible for maintaining an up-to-date copy of its owned pages; then, the acquirer can request copies of modified pages from their home nodes. At release points, diffs are computed and sent to the page’s home node, which reduces memory requirements in homebased DSM protocols and contributes to the scalability of the HLRC protocol.
A New Distributed JVM for Cluster Computing
1209
3 Java In Java, threads programming is simplified since it provides a parallel programming model that includes support for multithreading. The package java.lang offers the Thread class that supports methods to initiate, to execute, to stop, and to verify the state of a running thread. In addition, Java also includes a set of synchronization primitives and the standard semantics of Java allow the methods of a class to execute concurrently. The synchronized reserved word, when associated with methods, specifies that they can only execute in a mutual-exclusion mode. The JVM specifies the interaction model between threads and the main memory, by defining an abstract memory system, a set of memory operations, and a set of rules for these operations [2]. The main memory stores all program variables and is shared by the JVM threads. Each thread operates strictly on its local memory, so that variables have to be copied first from main memory to the thread’s local memory before any computation can be carried out. Similarly, local results become accessible to other threads only after they are copied back to main memory. Variables are referred to as master or working copy depending on whether they are located in main or local memory, respectively. The copying between main and local memory, and vice-versa, adds a specific overhead to thread operation. The replication of variables in local memories introduces a potential memory coherence hazard since different threads can observe different values for the same variable. The JVM offers two synchronization primitives, called monitorenter and monitorexit, to enforce memory consistency. In brief, the model requires that upon a monitorexit operation, the running thread updates the master copies with corresponding working copy values that the thread has modified. After executing a monitorenter operation, a thread should either initialize its work copies or assign the master values to them. The only exceptions are variables declared as volatile, to which JVM imposes the sequential consistency model. The memory management model is transparent to the programmer and is implemented by the compiler, which automatically generates the code that transfers data values between main memory and thread local memory.
4 The Cooperative Java Virtual Machine CoJVM [3] is a DSM implementation of the standard JVM and was designed to efficiently execute parallel Java programs in clusters. In CoJVM, the declaration and synchronization of objects follow the Java model, in which the main memory is shared among all threads running in the JVM [2]. Therefore in CoJVM all declared objects are implicitly and automatically allocated into the Java heap, which is implemented in the DSM space. Our DSM implementation uses the HLRC protocol for two main reasons. First, it tends to consume less memory and its scalable performance is competitive with that of homeless LRC implementations [4]. Second, the HLRC implementation [7] al-ready supports the VIA [8], a high-performance user-level communication protocol. Surdeanu and Moldovan [9] have shown that the LRC model is compliant with the Java Memory Model for data-race-free programs. Although HLRC adopts the page as the unit of granularity, we are not bound to that specific unit.
1210
M. Lobosco et al.
CoJVM benefits from the fact that Java already provides synchronization primitives: synchronized, wait, notify and notifyAll. The programmer with the use of these primitives can easily define a barrier or other synchronization constructs, or invoke a native routine. CoJVM supports only Java standard features, and adds no extra synchronization primitive. Since the declaration and synchronization of objects in the DSM follow the Java model and no extra synchronization primitive is added in our environment, a standard concurrent application can run without any code change. Threads created by the application are automatically moved to a remote host, and data sharing among them are treated following the language specification in a transparent way. Currently, CoJVM allows one single thread per node due to HLRC’s restriction but we plan to solve this problem soon.
5 Performance Evaluation Our hardware platform consists of a cluster of eight 650 MHZ Pentium III PCs running Linux 2.2.14-5.0. Each processor has a 256 KB L2 cache and each node has 512 MB of main memory. Each node has a Giganet cLAN NIC connected to a Giganet cLAN 5300 switch. This switch uses a thin tree topology and has an aggregate throughput of 1.25 Gbps. The point-to-point bandwidth to send 32 KB is 101 MB/s and the latency to send 1 byte is 7.9ms. Table 1. Sequential times of applications, in seconds Program MM LU Radix FFT SOR
Program Size 1000x1000 2048x2048 8388608 Keys 4194304 Complex Doubles 2500x2500
Seq. Time - JVM 485.77 2,345.88 24.27 227.21
Seq. Time - CoJVM 490.48 2,308.62 24.26 228.21
Overhead 0.97 % 0% 0% 0.44 %
304.75
306.46
0.56 %
Table 2. Performance of sequential applications: JVM (first figure) X CoJVM (second figure) Counter % Load/Store Inst % Miss Rate CPI
SOR 75.3 / 73.8 1.9 / 2.3 1.83 / 1.83
LU 86.8 / 90.1 1.2 / 6.1 2.78 / 2.73
Radix 87.4 / 86.0 1.2 / 1.2 1.76 / 1.76
FFT 91.2 / 91.2 1.7 / 1.7 1.85 / 1.85
To quantify the overhead that CoJVM imposes due to modifications of the standard JVM, we ported four benchmarks from the SPLASH-2 benchmark suite [10]: SOR, LU, FFT and Radix, and developed one, MM. The sequential execution times for each application are shown in Table 1. Each application executed 5 times, and the average execution time is presented. The standard deviation for all applications is less than 0,01%. For all applications, the overheads are less than 1%. We also instrumented both machines with the performance counter library PCL [11], which collects at processor level, run-time events of applications executing on commercial microprocessors. Table 2 presents the CPI (cycles per instruction) of sequential executions of SOR, LU, Radix, and FFT on the Sun JDK 1.2.2 and on CoJVM (which
A New Distributed JVM for Cluster Computing
1211
is based on Sun JDK 1.2.2 implementation). The table also presents the percentage of load/store instructions and data cache level 1 miss rates. Table 2 shows that the percentage of load/store instructions does not change for FFT, but increases for LU and decreases slightly for SOR and Radix. However, the increase in the number of memory accesses in LU does not have a negative impact on miss rate. Indeed, the miss rate for CoJVM is half of the miss for the standard JVM. CoJVM modifications to the standard JVM resulted in lower CPI rate for LU while for the other applications, both CPI were equal. Table 3 shows application speedups. We can observe that MM achieved an almost linear speedup while the other applications attained good speedups. Next, we analyze application performance in detail. In particular, we will compare CoJVM scalable performance against that of HLRC running the C version of the same benchmarks. Table 3. Speedups Program MM LU Radix FFT SOR
2 Nodes 2.0 2.0 1.8 1.8 1.8
4 Nodes 4.0 3.8 3.5 3.5 3.3
8 Nodes 7.8 7.0 6.3 6.1 6.8
MM is a coarse grain application that performs a multiplication operation of two square matrixes of size D. Since there is practically no synchronization between threads, MM achieved speedup of 7.8 on 8 nodes. MM spent 99.2% of its execution time doing useful computation and 0.7% waiting for remote pages on page misses. LU is a single-writer application with coarse-grain access that performs blocked LU factorization of a dense matrix. LU sent a total of 83,826 data messages and 164,923 control messages. Data message is related to the information needed during the execution of the application, such as pages that are transmitted during the execution are counted as data messages. Control is related to the information needed for the correct work of the software DSM protocol, such as page invalidations. Compared with the C version, Java sent 28 times more control messages and 20 times more data messages. LU required almost 1 MB/s of bandwidth per CPU and achieved speedup of 7 on 8 nodes. Two components contribute to LU’ slowdown: page and barrier. Page access time increased approximately 23 times, when we compare the execution on 8 nodes with the execution on 2 nodes. Page misses contributed to 3.6% of the total execution time. In the C version of LU, page misses contributed to 2.7% of the execution time. This happened because the number of page misses in Java is 19 times that of C version. Although barrier time increased less than page miss time (11%), it corresponded to 9.6% of the total execution time, against that of 6.4% in C. Radix is a multiple-writer application with coarse-grain access that implements an integer sort algorithm. It sent a total of 1,883 data messages and 2,941 control messages. Compared with the C version, Java sent 1.9 times less control messages and 2.9 times less data message. Radix required 1.9 MB/s of bandwidth per CPU and achieved a good speedup of 6.3 on 8 nodes. Three components contributed to slow down this benchmark: page, barrier and handler. Page misses contributed to 9% of the total execution time, barrier to 6.5% and handler to 4.2%. In the C version of the algorithm, page misses contributed to 8% of the execution, handler to 12.3% and
1212
M. Lobosco et al.
barrier to 45%. We observed that in C implementation there were 4.7 times more diffs created and applied than that of the Java version. The smaller number of diffs also had impact on the total volume of bytes transferred and on the miss rate of cache level 1. The C version transferred almost 2 times more bytes than that of Java, and its cache level 1 miss rate was almost 10 times higher than that of Java. FFT implements the Fast Fourier Transform algorithm. Communication occurs in transpose steps, which require all-to-all thread communication. FFT is a single-writer application with fine-grained access. It sent a total of 68 MB of data, which is 4.7 times more than that of the C version. FFT required 1.8 MB/s of bandwidth per CPU and achieved speedup of 6.1 on 8 nodes. Page misses and barrier contributed to slow down this application. Page misses contributed to 15.8% of the total execution time, while barrier contributed with almost 10.7%. In the C version of the algorithm, page misses contributed to 18% of the execution and barrier to 3.8%. The miss rate on the level 1 cache is almost 7 times higher in C than in Java. Table 4. HLRC statistics on 8 nodes: CoJVM (first figure) versus C (second figure) Statistic Page faults Page misses Lock acquired Lock misses Diff created Diff applied Barrier
SOR 21,652 / 1,056 690 / 532 4/0 4/0 20,654 / 0 20,654 / 0 150 / 150
LU 172,101 / 4,348 81,976 / 4,135 39 / 0 6/0 79,338 / 0 79,337 / 0 257 / 257
Radix 3,818 / 5,169 1,842 / 958 34 / 24 1 / 24 946 / 4,529 946 / 4,529 11 / 11
FFT 20,110 / 9,481 15,263 / 3,591 40 / 0 11 / 0 1840 / 0 1840 / 0 6/6
SOR uses the red-black successive over-relaxation method for solving partial differential equations. Communication occurs across the boundary rows between bands and is synchronized with barriers. This explains why the barrier time was responsible for 13% of execution. In the C version of SOR, barrier time was responsible for just 2.5% of the execution time. The difference between Java and C versions is due to an imbalance in computation caused by the high amount of diffs that Java creates. Because of its optimizations, the C version did not create diffs. Java created more than 20,000 diffs, which caused an imbalance due the deliver of write notices at the barrier. Diff is also responsible for 94% of the total data traffic. Data traffic in Java is 21 times more than in the C version. SOR required 1 MB/s of bandwidth per CPU and achieved speedup of 6.8 on 8 nodes. We can observe in Table 3 that the speedup obtained from 2 and 4 nodes are worse than those obtained with 4 and 8 nodes, respectively. Further investigation revealed that the larger caches were responsible for improving speedups. The above results show that two components are the main responsible for slowing down the benchmarks: page miss and barrier time. The barrier synchronization time is mostly caused by an imbalance in execution times between processors. This imbalance seems to stem mainly from inadequate distribution of pages among nodes, although false sharing could also affect performance. This imbalanced distribution of pages among home nodes occurs because usually one thread is responsible for the initializa-tion of internal objects and fields used during computation. Indeed, the JVM Specifica-tion establishes that all fields of an object must be initialized with their initial or default values during the object’s instantiation. In our implementation,
A New Distributed JVM for Cluster Computing
1213
however, whenever a thread writes to a page for the first time the thread’s node becomes the page’s home for the entire execution, according to the “first touch” rule of the HLRC protocol. This ex-plains the imbalance we observed. One immediate solution is to perform a distributed initialization, which may not be so simple. Actually, we are studying techniques to fix this problem in a transparent fashion. The imbalance in the distribution of home nodes impacts also page faults, page misses, and the creation of diffs - and consequently, the total amount of control and data messages that CoJVM transfers. This is evident when we compare Java against the C implementation of the same algorithm (see Table 4). SOR, LU and FFT did not create diffs in C, while Java created diffs in large amount. Radix is the only exception: CoJVM created 4.7 times less diffs than C. Finally, we verify the small impact of the internal CoJVM synchronization on application performance. In SOR, this overhead was equal to 2.4 ms, i.e., the cost of 4 lock misses. Lock misses occur when the thread needs a lock (due to a monitor enter operation), but the lock must be acquired remotely. FFT presented the highest overhead, with 11 locks misses. Surprisingly, although in Radix CoJVM requested more locks than that of C version, in CoJVM just 1 lock resulted in overhead, against 24 locks in C. However, in Radix lock requests did not have any impact on performance.
6 Related Work In this section we describe some distributed Java systems with implementations based on software DSM. A detailed survey on Java for high performance computing, including systems that adopt message-passing approaches, can be found in [12]. Java/DSM [13] was the first proposal of shared-memory abstraction on top of a heterogeneous network of workstations. In Java/DSM, the heap is allocated in the shared memory area, which is created with the use of TreadMarks [14], a homeless LRC protocol, and classes read by the JVM are allocated automatically into the shared memory. In this regard, CoJVM adopts a similar to approach, but using a home-based protocol. Java/DSM seems to be discontinued, and did not report any experimental result. cJVM [15] supports the idea of single system image (SSI) using the proxy design pattern [16], in contrast to our approach that adopts a software DSM protocol. In cJVM a new object is always created in the node where the request was executed first. Every object has one master copy that is located in the node where the object is created; objects from the other nodes that access this object use a proxy. If an object is heavily accessed, the node where the master copy is located becomes potentially a bottleneck. DISK [9] adopts an update-based, object-based, multiple-writer memory consistency protocol for a distributed JVM. CoJVM differs from DISK in two aspects: a) CoJVM adopts an invalidate-based approach, and b) we currently adopt a page-based approach, although our implementation is sufficient flexible to adopt an object-based or even a word-based approach. DISK detects which objects must be shared by the protocol and uses this information to reduce consistency overheads. DISK presents the speedups of two benchmarks, however without analyzing the results. JESSICA [18] adopts a home-based, object-based, invalidation-based, multiple-writer memory consistency protocol for a distributed JVM. In JESSICA, the node that started the application, called console node, performs all the synchronization operations, which impose a severe performance degradation: for
1214
M. Lobosco et al.
SOR, synchronization is responsible for 68% of the execution time, and the application achieves a speedup of 3.4 on 8 nodes [18]. In CoJVM the lock ownership is distributed equally among all the nodes participating of the computation, which contributed to the better performance achieved by our environment. The modifications made by Jessica in the original JVM impose a great performance slowdown in sequential application: for SOR, this slowdown is equal to 131%, while CoJVM almost do not impose any overhead.
7 Conclusion and Ongoing Works In this work, we introduced and evaluated CoJVM, a cooperative JVM that addresses in a novel way several performance aspects related to the implementation of DSM in Java. CoJVM complies with the Java language specification while supporting the shared memory abstraction as implemented by our customized version of HLRC, a home-based software DSM protocol. Moreover, CoJVM uses VIA as its communication protocol aiming to improve Java application performance even further. Using several benchmarks we showed that CoJVM achieved speedups, ranging from 6.1 to 7.8 on 8 nodes. However, we believe that CoJVM can further improve application speedups. In particular, we noticed that a shared data distribution imbalance can significantly impact barrier times, page faults, page misses and the creation of diffs – and consequently, the total amount of control and data messages that CoJVM transfers unnecessarily. We are studying new solutions to overcome this imbalance while refining CoJVM to take advantage of the application behavior, extracted from the JVM during run-time, in order to reduce the overheads of the coherence protocol. More specifically, a specialized run-time JVM machinery is being developed to create diffs dynamically, to allow the use of smaller units of coherence, and to detect automatically reads and writes to the shared memory, without using the time-expensive virtual-memory protection mechanism.
References 1. 2. 3. 4. 5. 6. 7. 8.
Arnold, K; Gosling, J. The Java Programming Language. Addison-Wesley, 1996. Lindholm, T, Yellin, F. The Java Virtual Machine Specification. Addison-Wesley, 1999. Lobosco, M, Amorim, C, Loques, O. A Java Environment for High-Performance th Computing. 13 Symp. on Computer Architecture and High-Performance Computing, Sep 2001. Zhou, Y, et alli. Performance Evaluation of Two Home-based Lazy Release Consistency Protocols for Shared Virtual Memory Systems. OSDI, Oct 1996. Li, K, Hudak, P. Memory Coherence in Shared Virtual Memory Systems. ACM Transactions on Computer Systems, 7(4):321–359, Nov 1989. Keleher, P, Cox, A, Zwaenepoel, W. Lazy Release Consistency for Software Distributed Shared Memory. Int. Symp. on Computer Architecture, pp. 13–21, May 1992. Rangarajan M, Iftode L. Software Distributed Shared Memory over Virtual Interface Architecture: Implementation and Performance. ALS, Oct 2000. VIA. VIA Specification, Version 1.0. http://www.viarch.org. Accessed on Jan, 29.
A New Distributed JVM for Cluster Computing 9. 10. 11. 12. 13. 14. 15. 16. 17. 18.
1215
Surdeanu, M, Moldovan, D. Design and Performance Analysis of a Distributed Java Virtual Machine. IEEE Trans. on Parallel and Distributed Systems, Vol. 13, No. 6, Jun 2002. Woo, S, et al. The SPLASH-2 Programs: Characterization and Methodological Considerations. Int. Symp. on Computer Architecture, pp. 24–36, Jun 1995. Performance Counter Library. http://www.fz-juelich.de/zam/PCL/. Accessed on January, 29 Lobosco, M, Amorim, C, Loques, O. Java for High-Performance Network-Based Computing: a Survey. Conc. and Computation: Practice and Experience: 2002(14), pp 1– 31. Yu, W.; Cox, A. Java/DSM: a Platform for Heterogeneous Computing. ACM Workshop on Java for Science and Engineering Computation, Jun 1997. Keleher, P, et alli. TreadMarks: Distributed Shared Memory on Standard Workstations and Operating Systems. Winter Usenix Conference, pp.115–131, Jan 1994. Aridor, Y, Factor, M, Teperman, A. cJVM: a Single System Image of a JVM on a Cluster. Int. Conf. on Parallel Processing, Sep 1999. Gamma, E, et alli. Design Patterns: Elements of Reusable Object-Oriented Software. Addison-Wesley, 1995. Fang, W, et alli. Efficient Global Object Space Support for Distributed JVM on Cluster. Int. Conf. on Parallel Processing, Aug 2002. Ma, M, Wang, C, Lau, F. JESSICA: Java-Enabled Single-System-Image Computing Architecture. Journal of Parallel and Distributed Computing (60), pp. 1194–1222, 2000.
An Extension of BSDL for Multimedia Bitstream Syntax Description* Sylvain Devillers IMEC, Kapeldreef 75, B-3001 Leuven, Belgium
[email protected] Abstract. In previous works, a generic framework for multimedia content adaptation has been introduced, where XML is used to describe the high-level structure of a bitstream and the resulting description is first transformed by an XSLT style sheet, and then processed to generate an adapted bitstream. In order to provide full interoperability, a new language named Bitstream Syntax Description Language (BSDL) is built on top of W3C XML Schema for the purpose of this generation process. A schema designed in this language and specific to a given coding format allows a generic processor to parse a description and generate the corresponding adapted bitstream. This paper describes an extension of BSDL to provide the new functionality corresponding to the reverse operation, i.e. allowing a generic software module to parse a bitstream conforming to a given coding format described by a schema, and generate the corresponding description. For this, BSDL introduces a number of language mechanisms on top of XML Schema. This paper details these language extensions and reviews the strengths and limits of this approach.
1
Introduction
The multiplication of new devices gaining access to the Internet makes crucial to be able to propose different versions of the same multimedia content (audio, video or still image) adapted to the client resources in terms of bandwidth, display or computing capabilities. On the other hand, the use of scalable media allows to retrieve different versions of a content from a single file by simple editions and hence saves the burden of generating and handling one file for each required version. However, since each multimedia coding format defines its own data structure, a dedicated software module is required for each offered format in order to properly edit the bitstream and produce the adapted version. To solve this issue, an original and flexible framework was proposed in [1]. In this method, XML [2] is used to describe the high-level structure of a bitstream; the resulting XML document is called a bitstream description. This description is not meant to replace the original binary format, but acts as an additional layer, similar to metadata. In most cases, it does not describe the bitstream on a bit-per-bit basis, but *
This paper was written while the author was with Philips Digital Systems Laboratories, 51, rue Carnot – B.P. 301 92156 Suresnes Cedex, France.
H. Kosch, L. Böszörményi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 1216–1223, 2003. © Springer-Verlag Berlin Heidelberg 2003
An Extension of BSDL for Multimedia Bitstream Syntax Description
1217
rather addresses its high-level structure, e.g. how the bitstream is organized in layers or packets of data. Furthermore, the bitstream description is itself scalable, which means it may describe the bitstream at different syntactic layers, e.g. finer or coarser levels of detail, depending on the application. With such a description, it is then possible for a generic software to transform the XML document, for example with an XSLT style sheet [3], and then generate back an adapted bitstream. This method is directly inspired by the web publishing framework that has been developed to dynamically adapt XML content with XSLT style sheets. It is important to note that while a coding format may be specified as a "flat" structure, i.e. as a sequence of binary symbols, these symbols may be grouped in a hierarchical manner in the description, since the inherent tree structure of an XML document has no impact on the bitstream generation process. Furthermore, the choice of tag names is fully transparent. The design of a bitstream schema for a given coding format is therefore not unique and is fully application dependent. In order to provide full interoperability, it is then necessary that a processor that is not aware of the specific coding format can nevertheless be used to produce a bitstream from its description. For this, a new language based on W3C XML Schema [4] and called Bitstream Syntax Description Language (BSDL) was introduced in [5]. With this language, it is then possible to design specific bitstream schemas describing the syntax of a particular coding format. These schemas can then be used by a generic processor to parse a description and generate the corresponding bitstream. A complementary method named gBSDL was also proposed for the case of highly constrained devices where the specific syntax schema is not available. A combined approach is currently being standardized within the MPEG-21 Digital Item Adaptation framework [6] and is described in [7]. This paper takes the BSDL approach a step further by building a new challenging functionality. In order to achieve a fully generic framework, it is necessary for a format-unaware, generic software module to be able to parse a bitstream based on the information provided by its schema and generate its description. In this respect, BSDL comes close to other formal syntax description languages such as ASN.1 [8] or Flavor [9], also known as Syntactical Description Language (SDL) in the context of MPEG-4 [10]. The section 2 presents a brief overview of BSDL, then section 3 details the language features specific to the bitstream parsing process. Lastly, the strengths and limitations of our approach are discussed in section 4.
2
BSDL Overview
This section briefly describes how the Bitstream Syntax Description Language (BSDL) introduced above is built on top of XML Schema. The primary role of a schema is to validate an XML document, i.e. to check that it follows a set of constraints on its structure and datatypes. On top of this, BSDL adds a twofold, new functionality, which is to specify how to generate an adapted bitstream from its description and vice-versa. For this, a number of restrictions and extensions over XML Schema are required. In the following, we name BSDtoBin Parser the generic software parsing the bitstream syntax description and generating the bitstream, and BintoBSD the software
1218
S. Devillers
performing the reverse operation. BSDtoBin uses a set of XML Schema components for which semantics may be assigned in the context of the bitstream generation. Some components should therefore be ignored because they have no meaning in this context, while other constructs are excluded. Similarly, BintoBSD uses an additional set of extensions and restrictions. In order to remain compatible with XML Schema, these extensions are introduced as application-specific annotations, namely by using the xsd:appinfo schema component and adding attributes with non-schema namespace. In order to distinguish both levels of extensions and restrictions, we name BSDL-1 the set of language features corresponding to the bitstream generation process, and BSDL-2 the set required for the description generation, where BSDL-1 is a subset of BSDL-2. The sets of extensions are declared in two schemas respectively named BSDL-1.xsd and BSDL2.xsd, the latter importing the previous one. The BSDL-1 specification was described in [5] and will not be repeated here. On the other hand, BSDL-2 has not been published yet and is the subject of this paper. The following section details the restrictions and extensions of BSDL-2 in terms of structural and datatypes aspects, as well as their semantics. In this paper, the "xsd:" prefix is used as a convention to refer to the XML Schema namespace, while "bsdl:" refers to the BSDL extensions.
3
BSDL-2: Bitstream Parsing
3.1
Structural Aspects of BSDL-2
A schema or a DTD define a set of constraints on XML documents. In XML Schema, most constraints are static, i.e. they are fully specified by the schema. For example, the minimum and maximum number of occurrences of a particle (xsd:minOccurs and xsd:maxOccurs) can be specified as a constant only and it is not possible to define a number of occurrences depending on the value of some other elements or attributes of the instance. On the other hand, this feature is required for parsing a bitstream since the length of a given field or its number of occurrences may be specified by a parameter found upstream. BSDL-2 thus needs to introduce new dynamic constraints on top of XML Schema, in particular to specify the number of occurrences of a particle or its conditional occurrence. While BSDL-1 declares types and attributes that carry specific semantics in the context of bitstream generation, but are still validated by XML Schema, these constraints are actually new language extensions. In order to remain fully compatible with XML Schema, these new constructs are added as annotations, namely as attributes with non-schema namespace. The three attributes introduced below are used to characterize a particle as an additional property. Firstly, BSDL-2 introduces a new attribute named bsdl:nOccurs to specify a variable number of occurrences. It is declared in the BSDL namespace and is used to characterize a particle in the same way as xsd:minOccurs and xsd:maxOccurs. It contains an XPath [11] expression indicating the relevant value as explained below. While parsing a bitstream, BintoBSD progressively instantiates the output description in the form of a DOM tree [12]. At a given stage in the bitstream parsing process, any parameter found upstream has thus already been instantiated as an element node of
An Extension of BSDL for Multimedia Bitstream Syntax Description
1219
the DOM tree. XPath 1.0 expressions are then used to locate this element relatively to the current element or with an absolute addressing in the DOM tree. Similarly to bsdl:nOccurs, BSDL-2 introduces an attribute named bsdl:if to specify the conditional occurrence of a particle. It contains an XPath expression that should be evaluated as a boolean value. Note that since bsdl:nOccurs is ignored by XML Schema, the allowed range declared by xsd:minOccurs and xsd:maxOccurs should cope with the expected value. Similarly, if a bsdl:if is specified, then xsd:minOccurs should be set to zero in case the condition is evaluated as false. Unlike previous attributes that specify constraints depending on data found upstream, a third attribute named bsdl:ifNext specifies a constraint on data found immediately downstream. It contains an hexadecimal string that should be compared to the next bytes in the stream. The particle is parsed if the sequences of bytes are identical. The syntax of bsdl:ifNext also allows an array of two hexadecimal strings separated by a dash character to specify a range of allowed values. This allows to test a number of bits other than multiple of eight. For example, the element <xsd:element name="elt1" bsdl:ifNext="08-0F"/> is to be parsed if the next four bits in the bitstream are set to zero and the fifth to one. 3.2
Datatypes Aspects of BSDL-2
BSDL-2 uses the same datatype model than BSDL-1, but with a set of additional constraints, in particular on types relating to sequence of bytes. While for some XML Schema built-in datatypes such as xsd:short or xsd:int, the encoding length is known from the schema (here, respectively 2 and 4 bytes), the length of xsd:hexBinary, xsd:base64 and bsdl:byteRange can only be determined from their lexical representation in the instance. It is therefore necessary to constrain their length in the schema so that BintoBSD knows how many bytes should be read. The xsd:length facet is used to constrain the first two types for this purpose. However, it cannot be used for bsdl:byteRange since this type is considered by XML Schema as an array of integers. The xsd:length facet then constrains the number of items and not the segment length. Furtermore, a new mechanism is required to constrain the length when it is not constant. BSDL-2 introduces a new facet named bsdl:length and declared in the BSDL namespace, which allows to distinguish it from the XML Schema facet. Since XML Schema does not currently allow a user to add his/her own facets, it is added to the xsd:restriction component via the annotation mechanism, i.e. the xsd:annotation/xsd:appinfo combination, and processed by BintoBSD as a regular facet. It contains an XPath expression that should be evaluated as an integer against the instantiated DOM tree, and gives the length in bytes of the data segment. The Document 1 gives a snippet of a Bitstream Syntax Schema for the JPEG2000 coding format showing the use of a variable value defining the length of the jp2:MarkerData element content. Document 1. Example of JPEG2000 Bitstream Syntax Schema <xsd:element name="SIZ"> <xsd:complexType>
1220
S. Devillers
<xsd:sequence> <xsd:element ref="jp2:Marker"/> <xsd:element name="LMarker" type="xsd:unsignedShort"/> <xsd:element name="MarkerData"> <xsd:simpleType> <xsd:restriction base="bsdl:byteRange"> <xsd:annotation><xsd:appinfo>
Lastly, several coding formats use start codes to delimit packets of data, which allows to efficiently locate different packets without having to decode their inner structure. In this case, the packet of data may be read as an xsd:hexBinary, xsd:base64 or bsdl:byteRange following the application needs, but since the length is unknown, a mechanism is required to indicate that the bitstream should be read until a given start code is found. For this purpose, we introduce a new BSDL facet named bsdl:startCode which contains the given sequence of bytes written in hexadecimal format. The BintoBSD parser thus reads the bitstream until the given sequence of bytes is found.
4
Discussion
We have introduced in this paper a language for describing the syntax of a multimedia bitstream. Among the similar languages proposed in the literature, Flavor [9] is probably the most accomplished attempt. This section discusses the strengths and limits of our approach, in particular in comparison to Flavor. 4.1
BSDL and XML Schema
The main characteristic and challenge of our work is that we chose not to develop "yet another language", but rather to build it on top of an existing one. Furthermore, we chose to base it on a declarative language, the function of which is to define constraints on XML documents. We are conscious that, by doing so, we are using XML Schema for a purpose which is not its initial object. However, several of the introduced features define dynamic constraints on XML documents and are thus still relevant to XML Schema. The bsdl:if attribute states a condition on the occurrence of a particle and bsdl:nOccurs specifies a variable number of occurrences. Lastly, the bsdl:length facet provides the same functionality than xsd:length but with a variable value. In our application, XML Schema and BSDL extensions are first used to validate the description. Then, in a second step, we assign specific semantics to the schema
An Extension of BSDL for Multimedia Bitstream Syntax Description
1221
components in the form of a processing model for the description and bitstream generation process, not detailed here. Some other constructs such as the bsdl:ifNext attribute or the bsdl:startCode facet are specific to the bitstream parsing process and irrelevant to XML Schema. However, in order to simplify the syntax, we chose to define the two different types of extensions in a single specification and namespace. 4.2
Use of BSDL for Bitstream Parsing
It should be noted that BSDL does not and cannot have the ambition to parse any bitstream at any level of detail. Most coding formats have been specified without the use of a formal language (except notably for MPEG-4 Systems written in Flavor/SDL) and do not follow any constraints. In particular, the main part of a bitstream - qualified hereafter as the "payload" - is usually the output of an encoding process such as entropy coding, wavelet or Discrete Cosine Transform. Such sequence of bits can only be decoded via an algorithm implemented in a programming language. It is obviously not the ambition of BSDL or Flavor to natively decode such data. In order to handle specific encoding schemes, it is possible to extend the BSDL parser. For this, the user-specific datatype should be defined in a schema with an attribute (not described here) indicating that it is hard-coded. The type definition in the schema is used for validating the lexical representation in the description, but not by the BintoBSD parser. The encoding and decoding methods are then implemented following a provided interface and appended to the system classpath (for a Java implementation). While reading the schema, the BintoBSD parser then dynamically instantiates the corresponding class and runs the encoding or decoding method. On the other hand, we believe that using a formal language to specify a bitstream syntax helps checking its consistency and separating the parsing and decoding processes. While designing a schema for the MPEG-4 Video Object Layer (VOL) format [13] and testing it, we found some inconsistency in the spelling of the mnemonics and some ambiguity on default values of some parameters. Such imprecision is prone to incorrect interpretations and discrepancies in software implementations. The syntax description of MPEG-4 visual part (without the semantics) represents around seventy pages of tables. It is thus a very difficult task, if not impossible, to exclude any error from the text. Writing directly the syntax in a formal language would allow to test and validate its consistency, and track the modifications changes during the specification process. Furthermore, specifying a bitstream syntax usually consists in two different tasks, firstly in formatting the output of an encoding method as seen above, and secondly in organizing the parameters and packets of data as a stream of symbols. Even though the compactness of the global result is a strong requirement, it is vital to be able to efficiently parse the bitstream to access the different segments of data. Typically, for parallelizing some tasks, the parsing process should be achievable at low cost in order to distribute the data segments to be decoded to dedicated processors. For these reasons, using a formal language implicitly sets some constraints on the bitstream syntax and hence benefits the complexity of the parsing process. Another main feature of our approach is that it does not only provide a grammar for parsing the bitstream, but also a persistent representation of the bitstream content in the form of an XML document - the description. In comparison, the principle of
1222
S. Devillers
Flavor is to generate C++ or Java code from the bitstream syntax description, which can be interfaced with the application. In this case, the memory representation of the bitstream content is a hierarchy of C++/Java objects and is thus language dependent. More recently, an extension named XFlavor [14] has been developed to provide an XML representation of the data. On the other hand, BSDL provides both a grammar to parse the bitstream - the schema - and a persistent representation of the data - the description. Having the data in XML format gives access to the extensive family of XML-based languages and tools (XSLT, XPath...). Furthermore, before generating the XML document, the BSDL parser first instantiates a DOM tree, which is a standard, language-independent memory representation of XML data and can thus be easily interfaced with the application. Lastly, a limit of our approach is the inherent verbosity of XML and XML Schema. Several solutions have been proposed to solve this issue. In particular, MPEG-7 BiM [15] is a very efficient method for binarizing an XML document by using its schema. Since BSDL remains compatible with XML Schema, BiM can be used to binarize the bitstream description, which is particularly useful when descriptions grow large, notably for video. As for XML Schema, a number of commercial products now provide graphical interfaces to design schemas in a user-friendly way. Lastly, it should be reminded that BSDL was primarily introduced for the problem of content adaptation, and thus is meant to describe bitstreams mainly at a high syntactical level, which should limit the verbosity of the descriptions.
5
Conclusion
In this paper, we extend the scope of the Bitstream Syntax Description Language (BSDL), which was primarily introduced for the problem of content adaptation. By adding new language mechanisms and defining specific semantics in the context of bitstream parsing, it is possible for a generic processor to use the schema to parse a bitstream. With this respect, BSDL comes close to other description languages such as Flavor. It is interesting to note that both languages tend to the same functionality while BSDL is based on XML Schema and Flavor, though presented as a declarative language, on a C++ like syntax. By using XML Schema beyond its original purpose, we also demonstrate the expressive power of this language. Lastly, we introduce new dynamic constraints on XML documents. Tests have been successfully performed on several multimedia coding formats, including JPEG2000 images, MPEG-4 Video Elementary Streams and MPEG-4 Visual Texture Coding. Other ongoing tests include MPEG-4 Audio AAC and JPEG. Although a generic software cannot pretend to be as efficient as an ad-hoc, specific software, the BSDL method significantly facilitates the process of accessing and reading structured, binary data by providing a declarative approach to bitstream parsing. Acknowledgement. Part of this work was funded by the OZONE research project (IST-2000-30026).
An Extension of BSDL for Multimedia Bitstream Syntax Description
1223
References 1. 2. 3. 4. 5. 6. 7.
8. 9. 10. 11. 12. 13. 14. 15.
th
Amielh M. and Devillers S.: Multimedia Content Adaptation with XML, 8 International Conference on Multimedia Modeling MMM’2001 (Amsterdam, The Netherlands, November 5-7, 2001), Paper Proceedings pp. 127–145. XML Extensible Markup Language 1.0 (Second Edition), W3C Recommendation, October 6, 2000. XSL Transformations Version 1.0, W3C Recommendation, November 16, 1999. XML Schema Part 0: Primer, Part 1: Structures and Part 2: Datatypes, W3C Recommendation, May 2, 2001. Amielh M. and Devillers S.: Bitstream Syntax Description Language: Application of XML th Schema to Multimedia Content Adaptation, 11 International World Wide Web Conference, WWW2002 (Honolulu, May 6-11, 2002), Alternate Paper Tracks. ISO/IEC JTC 1/SC 29/WG 11/N5353 MPEG-21 Digital Item Adaptation CD, December 2002, Awaji Island, Japan. Panis G., Hutter A., Heuer J., Hellwagner H., Kosch H., Timmerer C., Devillers S. and Amielh M.: Bitstream Syntax Description: A Tool for Multimedia Resource Adaptation within MPEG-21, Accepted to Signal Processing: Image Communication, special issue on Multimedia Adaptation. ASN.1 (ISO Standards 8824 and 8825). Eleftheriadis A.: Flavor: A Language for Media Representation, ACM Multimedia '97 Conference, (Seattle, WA, November 1997), Proceedings pp. 1–9. ISO/IEC 14496-1:2001 Information Technology – Coding of Audio-Visual Objects – Part 1: Systems. XML Path Language, W3C Recommendation, 16 November 1999. Document Object Model (DOM) Level 2 Core Specification, Version 1.0, W3C Recommendation 13 November, 2000. ISO/IEC 14496-2:2001 Information Technology – Coding of Audio-Visual Objects – Part 2: Visual. Hong D. and Eleftheriadis A.: XFlavor: Bridging Bits and Objects in Media Representation, IEEE Int'l Conf. on Multimedia and Expo ICME (Lausanne, Switzerland, August 2002). ISO/IEC 15938-1:2002 MPEG-7: Multimedia content description interface – Part 1 : Systems. See also http://www.expway.tv for BiM.
Fast Construction, Easy Configuration, and Flexible Management of a Cluster System Ha Yoon Song, Han-gyoo Kim, and Kee Cheol Lee College of Information and Computer Engineering, Hongik University, Seoul, Korea {song, hkim, lee}@cs.hongik.ac.kr
Abstract. New requirements of cluster systems have arisen. In this paper, we introduce a new clustering scheme for the easy construction and maintenance of a server using the Java Management Extension. Our experiments show that our system and its related applications are easy to construct, expand, and manage with guaranteed performance.
1
Introduction
The internal structure of an application server tends to be a cluster system since it might be impossible for one node server to process huge amounts of services. Regarding the recent trends of non-academic applications, several additions to the traditional requirements exists: The server systems must be fast to construct, easy to expand and configure, and flexible to manage. Additionally, most service handlers of applications are being developed in Java based frameworks. Coping with the current requirements, a JMX (Java Management Extensions) based cluster system is presented. Our experiences show that the cluster systems built in JMX meet all the traditional and new requirements. For example, two man-months were consumed for the construction of a server system, and its performance remained satisfactory as users were added. The rest of the paper is structured as follows. Section 2 will discuss the construction and configuration issues. Section 3 will show the management details, and section 4 will show the performance verifications of our cluster system. We will conclude this paper in the final section.
2
Cluster System Construction and Configuration
We used JMX [1][2] as a core middleware for our cluster system construction. Figure 1 shows the overall structure of the cluster system in the simplest configuration. There are two nodes based on EJB and JMX, and one Manager which manages the whole system. Alternative configurations can be constructed without any modification of cluster components. The manageable object for JMX is an application program based on the JAVA technologies such as EJB (Enterprise Java Beans). Each application program will be mapped on an MBean, and each MBean will be registered on the H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 1224–1228, 2003. c Springer-Verlag Berlin Heidelberg 2003
Fast Construction, Easy Configuration, and Flexible Management
1225
Management Applications
Manager Connector Connector
MBean Server
MBean Server
Clustered Server MBean
MBean
MBean
Java-Based Applications
ServerInfo
MBeanServerInof
AgentID
AgentID
MBeanServerName
IP Address
MBeanServerName
MBeanName
IsAlive
ServiceCnt
ServiceCnt
avgServiceTime
avgServiceTime
Capacity
Capacity
MBean
MBeanInfo
Java-Based Applications MBeanType
Fig. 1. Conceptual structure of a cluster. Fig. 2. Structure of management objects.
MBean server that resides on the same JAVA virtual machine to the applications’. The Monitor of this MBean server (one of the Agent Services) manages the applications and their status information. The Manager system can recognize and resolve the erroneous situation by the reception of events from Monitor whenever Monitor senses exceptional situations. The communication between Manager and Agents can be made by SNMP of JMX [3]. Alternatively, communication can be made by the connector server on Agent side and the connector client on the Manager system, which is used for the actual system implementation in this paper. 2.1
Managed Objects
Manager manages three sorts of objects: ServerInfo, MBeanServerInfo, and MBeanInfo. ServerInfo has the information of a node where Agent resides. MBeanServerInfo has the information of the MBean server created by the Agent. MBeanInfo stands for the management information of each MBean registered on the MBean server. Figure 2 shows the structure of the Managed object and their attributes. Agent Side
Manager Side
Load Balancer
Message Analyzer
Message Type
Value1
Event Type
Variable2
MBean Name
value2
Variable Binding List
...
Fig. 3. Message formats.
MBean
Connector Server
Variable1
M Bean Server
Agent ID
Connector Server
MBean
Manager ID
Message Analyzer
JMX enabled Management Application
Moniter Java Virtual Machine
Java Virtual Machine
MIB
Fig. 4. Implementation details.
1226
2.2
H.Y. Song, H.-g. Kim, and K.C. Lee
Message Formats and Types
Message format is designed to provide management communication between Manager and Agents as shown in figure 3. A message consists of MessageID, AgentID, MessageType with six subfields for six message types, EventType which can be thrown by MBean or MBean server, and VariableBindingList which contains MBean or MBean server information. Message types are MBeanInfoRequest from Manager to Agent, MBeanInfoTransfer from Agent replying Manager, EventTransfer from MBean or MBean server to Manager, MBeanServerSatusRequest from Manager to Agent, MBeanServerStatusTransfer from Agent, and IsAlive from Manager to check if a node is alive. 2.3
Implementation
The implemented testbed is composed of normal computers (nodes) without Manager and nodes with Manager over heterogeneous processors and operating systems. The Manager node has a connector client for the communication with Agent. The detailed component diagram is shown in figure 4. Each application is mapped on an MBean and Agent manages it. The MBean server registers MBeans and Agent services. For this implementation, each node has one MBean server. Whenever each application starts a new service for a client, a new MBean will be created and registered to the MBean server. Among various Agent services, Monitor plays the key role to sense erroneous situations. Monitor checks the number of MBeans on the server and other related information specific to MBeans, and throws a situation-related event so that the Manager recognizes the error occurrence.
M
M
M
M
check the new node
found of a new node
request info of a new node
Load Balancer
M
NEW
M
Load Balancer
M
notify to load balancer of a node removed M
request the node info until no service assigned
MIB
M
MIB manager recognizes the node info
store the info notify to load balancer
Fig. 5. Addition of a new node.
3
modify manager info
node removed
Fig. 6. Removal of an existing node.
Cluster Management
Our cluster system can easily cope with various situations of management. The most representative situations and solutions can be distinguished as follows.
Fast Construction, Easy Configuration, and Flexible Management
1227
– Addition of a new node: Figure 5 shows the detailed node addition procedure. An M inside a node box stands for Manager. Manager checks a new node addition, assigns a new Agent on the node and notifies to Load Balancer. – Removal of an existing Node: Figure 6 shows the removal procedure. Manager notifies the removal of a node to Load Balancer and updates MIB after all the jobs on the node are finished. – Management of an overloaded node (load balancing): Figure 7 shows the load balancing procedure. Monitor recognizes the overload of a node and reports it. Manager notifies to Load Balancer and requests ServiceCnt information to check if other nodes can process more jobs. Each node has its own avgServiceTime, and must check if it can reduce the avgServiceTime. Manager informs Load Balancer of the nodes whose ServiceCnt are less than their capacity. – Management of a node failure: Figure 8 is for a node failure. Monitor’s notification to Manager about a dead node starts the management procedure.
Load Balancer
M
M
M
Load Balancer
M
MIB monitor recognize a problem
Down
manager info update notify to load balance
Monitor
M
MIB
Down
Mbean Server
Load Balancer
M
found a dead node
manager info update notify to load balance of a problematic node
M
M
removed of the node repair
restart of the node
MIB node info notify
load balance recognize restart
Fig. 7. Overloaded node processing
4
Fig. 8. Node failure processing
Load Balancing Experiments
Several exceptional situations are designed and experienced. The graphs of time versus the accumulated number of allotted jobs show the load balancing results. Figure 9 is for the case of two nodes added and one node removed. The processing capacities of each node were purposely set to a small number. Two nodes were added between minutes 3 and 5. For the next 3 minutes, those two nodes started to be allotted but almost no jobs were allotted to the other nodes. At minute 10, one node was ordered to be removed, and no new jobs were allotted to it. After it was completely removed at minute 13, new job allotments to each node were delayed for 2 or 3 minutes because of limited node capacities and a small number of active nodes. Figure 10 shows another scenario: three nodes got down, and recovered and added back to the cluster. Additionally two nodes were added at minutes 3 and
1228
H.Y. Song, H.-g. Kim, and K.C. Lee
17, respectively, and they two started to be allotted 2 or 3 minutes later, limiting the allotments to the other nodes for the time being. Three nodes got down at minutes 9, 11, and 14, and added back to the cluster at minutes 10, 12, and 15, respectively. As a node got down, more jobs were allotted to the other nodes. As a node was newly added or added back, the job allotments to the other nodes were shown to be delayed.
30
25 20 node 1 node 2 node 3 node 4 node 5 node 6 node 7
15 10 5 0
25 node 1 node 2 node 3 node 4 node 5 node 6 node 7
20 15 10 5 0
1
4
7
10 time(min)
13
16
Fig. 9. Two nodes added and one removed.
5
number of services accumulated
number of services accumulated
30
1
4
7
10 13 time(min)
16
19
22
Fig. 10. 3 nodes down, recovered, and 2 added.
Conclusion and Future Works
In this paper we demonstrated the fast construction, the flexible configuration, and the easy management of a cluster system. In our system, the applicationbased management can be transparently separated from the server-based management. It took two man-months to construct our cluster system. Our system can also dynamically cope with the various situations of a cluster management for the high management ability and availability with reasonable performance. Our cluster system has some JMX-related limitations. For example, one Agent cannot have multiple MBean servers. The next version of JMX might be improved so that more dedicated management can be available by multiple MBean servers on one Agent for better clustering managements.
References 1. H. Kreger. Java management extensions for application management. IBM Systems Journal, 40(1), 2001. 2. Sun-Microsystems. Java management extensions instrumentation and agent specifications, v1.1. 2002. 3. Sun-Microsystems. Java management extensions SNMP manager APIs. 1999.
Topic 17 Peer-to-Peer Computing Luc Boug´e, Franck Cappello, Omer Rana, and Bernard Traversat Topic Chairs This Topic is devoted to High-Performance Computing on widely distributed systems utilizing Peer-to-Peer technologies. In contrast with the usual Client-Server organization scheme, these systems are (or attempt at being) highly decentralized, self-organizing with respect to the external context, and with a notion of balanced resource trading. Hence, each node within such a system can provide both Client and Server capability. In the long term, participating nodes may expect to be granted as much resources as they provide. Because of size, autonomy and high volatility of their resources, P2P platforms provide the opportunity for researchers to re-evaluate many fields of Distributed Computing, such as protocols, infrastructures, security, certification, fault tolerance, scheduling, performance, etc. A number of 15 submissions have been received, 5 of which have been selected for regular presentation. They address various aspects of P2P systems: managing hierarchical P2P architectures, revisiting classical distributed computing techniques in a P2P context, instilling P2P techniques in Grid Computing, devising abstract models for the global behavior of P2P systems. One additional research note has also been selected because of its original insights in the context of P2P systems: it is concerned with combatting freeriders, users consuming resources without contributing in a commensurate way. The overall appreciation of the Program Committee is that many submissions describe interesting ideas, but that these ideas are not sufficiently backed up by scientific evidence. In fact, the experimental evaluation is often weak, or sometimes even void, so that it is difficult to decide upon the actual impact of the idea on scientific grounds. This aspect is obviously a major challenge, as evaluating the actual impact of such techniques requires managing and monotoring a large-scale configuration of volatile and faulty agents. Or, probably more realistically, to simulate such a configuration: the recent advent of large simulation equipments (e.g., the Netbed/Emulab Platform in the US <www.emulab.net>, or the Grid Explorer Project in France) opens new opportunities in this direction. The Program Committee would like to strongly encourage the authors to include a detailed evaluation of their ideas in their future papers, together with an extensive discussion of their expected impact. Finally, we would like to thank the Euro-Par 2003 organizers for giving us the opportunity to launch this new Topic, and to all submitters for their contribution.
H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, p. 1229, 2003. c Springer-Verlag Berlin Heidelberg 2003
Hierarchical Peer-to-Peer Systems L. Garc´es-Erice1 , E.W. Biersack1 , P.A. Felber1 , K.W. Ross2 , and G. Urvoy-Keller1 1
Institut EURECOM, 06904 Sophia Antipolis, France {garces|erbi|felber|urvoy}@eurecom.fr 2 Polytechnic University, Brooklyn, NY 11201, USA
[email protected] Abstract. Structured peer-to-peer (P2P) lookup services organize peers into a flat overlay network and offer distributed hash table (DHT) functionality. Data is associated with keys and each peer is responsible for a subset of the keys. In hierarchical DHTs, peers are organized into groups, and each group has its autonomous intra-group overlay network and lookup service. Groups are organized in a top-level overlay network. To find a peer that is responsible for a key, the top-level overlay first determines the group responsible for the key; the responsible group then uses its intra-group overlay to determine the specific peer that is responsible for the key. We provide a general framework and a scalable hierarchical overlay management. We study a two-tier hierarchy using Chord for the top level. Our analysis shows that by using the most reliable peers in the top level, the hierarchical design significantly reduces the expected number of hops.
1
Introduction
Peer-to-peer (P2P) systems are gaining increased popularity, as they make it possible to harness the resources of large populations of networked computers in a cost-effective manner. A central problem of P2P systems is to assign and locate resources among peers. This task is achieved by a P2P lookup service. Several important proposals have been recently put forth for implementing distributed P2P lookup services, including Chord [1], CAN [2], Pastry [3] and Tapestry [4]. In these lookup services, each key for a data item is assigned to the live peer whose node identifier is “closest” to the key (according to some metric). The lookup service determines the peer that is responsible for a given key. The lookup service is implemented by organizing the peers in a structured overlay network, and routing a message through the overlay to the responsible peer. The efficiency of a lookup service is generally measured as a function of the number of peer hops needed to route a message to the responsible peer, as well as the size of the routing table maintained by each peer. For example, Chord requires O(log N ) peer hops and O(log N ) routing table entries when there are N peers in the overlay. Implementations of the distributed lookup service are often referred to as Distributed Hash Tables (DHTs). Chord, CAN, Pastry and Tapestry are all flat DHT designs without hierarchical routing. Each peer is indistinguishable from another in the sense that all peers use the same rules for determining the routes for lookup messages. This approach is strikingly different from routing in the Internet, which uses hierarchical routing. Hierarchical routing in the Internet offers several benefits over non-hierarchical routing, including scalability and administrative autonomy. H. Kosch, L. B¨osz¨orm´enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 1230–1239, 2003. c Springer-Verlag Berlin Heidelberg 2003
Hierarchical Peer-to-Peer Systems
1231
Inspired by hierarchical routing in the Internet, we examine two-tier DHTs in which (i) peers are organized in disjoint groups, and (ii) lookup messages are first routed to the destination group using an inter-group overlay, and then routed to the destination peer using an intra-group overlay. We present a general framework for hierarchical DHTs. Each group maintains its own overlay network and intra-group lookup service. A top-level overlay is defined among the groups. Within each group, a subset of peers are labeled as “superpeers”. Superpeers, which are analogous to gateway routers in hierarchical IP networks, are used by the toplevel overlay to route messages among groups. We consider designs for which peers in the same group are locally close. We describe a cooperative caching scheme that can significantly reduce average data transfer delays. Finally, we also provide a scalable algorithm for assigning peers to groups, identifying superpeers, and maintaining the overlays. After presenting the general framework, we explore in detail a particular instantiation in which Chord is used for the top-level overlay. Using a novel analytical model, we analyze the expected number of peer hops that are required for a lookup in the hierarchical Chord instantiation. Our model explicitly captures inaccuracies in the routing tables due to peer failures. The paper is organized as follows: We first discuss related work in Section 2. We then present the general framework for hierarchical DHT’s in Section 3. We discuss the particular case of a two-tier Chord instantiation in Section 4, and we quantify the improvement of lookup latency due to the hierarchical organization of the peers.
2
Related Work
P2P networks can be classified as being either unstructured or structured. Chord [1], CAN [2], Pastry [3], Tapestry [4], and P-Grid [5], which use highly structured overlays and use hashing for targeted data placement, are examples of structured P2P networks. These P2P networks are all flat designs (P-Grid uses a virtual distributed search tree only for routing purposes). Gnutella [6] and KaZaA [7], whose overlays grow organically and use random data placement, are examples of unstructured P2P networks. Ratnasamy et al. [8] explore using landmark nodes to bin peers into groups. The basic idea is for each peer to measure its round-trip time (RTT) to M landmarks, order the resulting RTTs, and then assign itself to one of M ! groups. Our hierarchical DHT schemes bear little resemblance to the scheme in [8]. Although in [8] the peers are organized in groups according to locality, the lookup algorithm applies only to CAN, does not use superpeers, and is not a multi-level hierarchical algorithm. Our approach has been influenced by KaZaA, an enormously successful unstructured P2P file sharing service. KaZaA designates the more available and powerful peers as supernodes. In KaZaA, when a new peer wants to join, it bins itself with the existing supernodes, and establishes an overlay connection with the supernode that has the shortest RTT. The supernodes are connected through a top-level overlay network. A similar architecture has been proposed in CAP [9], a two-tier unstructured P2P network. Our design is a blend of the supernode/hierarchy/heterogeniety of KaZaA with the lookup services in the structured DHTs.
1232
L. Garc´es-Erice et al.
Brocade [10] proposes to organize the peers in a two-level overlay. All peers form a single overlay OL . Supernodes are typically well connected and situated near network access points, forming another overlay OH . Brocade is not truly hierarchical since all peers are part of OL . Finally, Castro et al. present in [11] a topology-aware version of Pastry [3]. At each hop Pastry presents multiple equivalent choices to route a request. By choosing the closest (smallest network delay) peer at each hop, they try to minimize network delay. However, at each step the possibilities decrease exponentially, so delay is mainly determined by the last hop, usually the longest. We propose large hops to first get to a group, and then shorter local hops inside the group, leading to a more natural caching scheme, as shown later in section 3.4.
3
Hierarchical Framework
We begin by presenting a general framework for a hierarchical DHT. Although we focus on a two-tier hierarchy, the framework can be extended to a general tier hierarchy. Let P denote the set of peers participating in the system. Each peer has a node id. Each peer also has an IP address (dynamic or not). The peers are interconnected through a network of links and switching equipment (routers, bridges, etc.) The peers send lookup query messages to each other using a hierarchical overlay network, as described below. The peers are organized into groups (see group management in Section 3.3). The groups may or may not be such that the peers in the same group are topologically close to each other, depending on the application needs. Each group has a unique group id. Let I be the number of groups, Gi the peers in group i, and gi the id for group i. The groups are organized into a top-level overlay network defined by a directed graph (X, U ), where X = {g1 , . . . , gI } is the set of all the groups and U is a given set of virtual edges between the nodes (that is, groups) in X. The graph (X, U ) is required to be connected, that is, between any two nodes g and g in X there is a directed path from g to g that uses the edges in U . It is important to note that this overlay network defines directed edges among groups and not among specific peers in the groups. Each group is required to have one or more superpeers. Let Si ⊆ Gi be the set of superpeers in group i. Our architecture allows for Si = Gi for all i = 1, . . . , I, in which case all peers are superpeers. We refer to architectures for which all peers are superpeers as the symmetric design. Our architecture also allows |Si | = 1 for all i = 1, . . . , I, in which case each group has exactly one superpeer. Let Ri = Gi − Si be the set of all “regular peers” in group gi . For non-symmetric designs (Si = Gi ), an attempt is made to designate the more powerful peers as superpeers. By “more powerful,” we primarily mean the peers that are up and connected the most (and secondarily, those with high CPU power and/or network connection bandwidth). The superpeers are gateways between the groups: they are used for inter-group query propagation. To this end, we require that if si is a superpeer in Gi , and (gi , gj ) is an edge in the top-level overlay network (X, U ), then si knows the name and the current IP address of at least one superpeer sj ∈ Sj . With this knowledge, si can send query messages to sj . If p is a regular peer, then p can only reach other groups through superpeers. Figure 1 (left) shows a top-level overlay network and possible communication relationships between the corresponding
Hierarchical Peer-to-Peer Systems
1233
List of peers in group
g2
r1
r1
S2
r1
g3
r2
r2
r2 r3
S3
> 103 peers
Top−level overlay network
s1 S1
s4
hash
CAN Group
g4
r3
s3
g1 S4
g4
s2
2 3 Group 10 − 10 peers
r4
g1
Chord Group CARP Group
g2
< 102 peers
r1
r3
r1
g3
> 103 peers r2
r3
r2
Fig. 1. Communication relationships between groups in the overlay network and superpeers in neighboring groups (left). On the right, a ring-like overlay network with a single superpeer per group. Intra-group lookup is implemented using different lookup services (CARP, Chord, CAN).
superpeers. Figure 1 (right) shows an example for which there is one superpeer in each group and the top-level overlay network is a ring. Within each group there is also an overlay network among the peers in the group. 3.1
Hierarchical Lookup Service
Consider a two-level lookup service. Given a key k, we say that group gj is responsible for k if gj is the “closest” group to k among all the groups. Here “closest” is defined by the specific top-level lookup service (e.g., Chord, CAN, Pastry, or Tapestry). Our two-tier DHT operates as follows. Suppose a peer pi ∈ Gi wants to determine the peer that is responsible for a key k. 1. Peer pi sends a query message to one of the superpeers in Si . 2. Once the query reaches a superpeer, the top-level lookup service routes the query through (X, U ) to the group Gj that is responsible for the key k. During this phase, the query only passes through superpeers, hopping from one group to the next. Eventually, the query message arrives at some superpeer sj ∈ Gj . 3. Using the overlay network in group j, the superpeer sj routes the query to the peer pj ∈ Gj that is responsible for the key k. This approach can be generalized to an arbitrary number of levels. A request is first routed through the top-most overlay network to some superpeer at the next level below, which in turn routes the request through its “local” overlay network, and so on until the request finally reaches some peer node at the bottom-most level. The hierarchical architecture has several important advantages when compared to the flat overlay networks. – Exploiting heterogeneous peers: By designating as superpeers the peers that are “up” the most, the top-level overlay network will be more stable than the corresponding flat overlay network. – Transparency: When a key is moved from one peer to another within a group, the search for the peer holding the key is completely transparent to the top-level
1234
L. Garc´es-Erice et al.
algorithm. Similarly, if a group changes its intra-group lookup algorithm, the change is completely transparent to the other groups and to the top-level lookup algorithm. Also, the failure of a regular peer ri ∈ Gi (or the appearance of a new peer) will be local to Gi ; routing tables in peers outside of Gi are not effected. – Faster lookup time: Because the number of groups will be typically orders of magnitude smaller than the total number of peers, queries travel over fewer hops. – Less messages in the wide-area: If the most stable peers form the top-level DHT, most overlay reconstruction messages happen inside groups, which gather peers that are topologically close. Less hops per lookup means also less messages exchanged. Finally, content caching inside groups can further reduce the number of messages that need to get out of the group. 3.2
Intra-group Lookup
The framework we just described is quite flexible, allowing for different independent intra-group overlays: If a group has a small number of peers (say, in the tens), each peer could track all the other peers in its group (their ids and IP addresses); CARP [12] or consistent hashing [13] could be used to assign and locate keys within the group. The number of steps to perform such an intra-group lookup in the destination group is O(1) (g2 in Figure 1, right). If the group is a little larger (say, in the hundreds), then the superpeers could track all the peers in the group. In this case, by forwarding a query to a local superpeer, a peer can do a local lookup in O(1) steps (g1 in Figure 1, right). Finally, for larger groups, a DHT such as Chord, CAN, Pastry, or Tapestry can be used within the group (g3 and g4 in Figure 1, right side). A local lookup takes O(log M ) hops, for M peers in the group. 3.3
Hierarchy and Group Management
We now briefly describe the protocols used to manage groups: consider peer p joining the hierarchical DHT. We assume that p is able to get the id g of the group it belongs to (e.g., g may correspond to the name of p’s ISP or university campus). First, p contacts and asks another peer p already part of the P2P network to look up key g. Following the first step of the hierarchical lookup, p locates and returns the IP address of the superpeer(s) of the responsible group. If the group id of the returned superpeer(s) is precisely g, then p joins the group using the regular join mechanisms of the underlying intra-group DHT; additionally, p notifies the superpeer(s) of its CPU and bandwidth resources. If the group id is not g, then a new group is created with id g and p as only (super)peer. In a network with m superpeers per group, the first m peers to join a group g become the superpeers of that group. Because superpeers are expected to be the most stable nodes, we let superpeers monitor the peers that join a group and present “good” characteristics. Superpeers keep an ordered list of the superpeer candidates: the longer a peer remains connected and the higher its resources, the better a superpeer candidate it becomes. This list is sent periodically to the regular peers of the group. When a superpeer fails or disconnects, the first regular peer in the list becomes superpeer and joins the top-level
Hierarchical Peer-to-Peer Systems
1235
overlay. It informs all peers in its group, as well as the superpeers of the neighboring groups. We are thus able to provide stability to the top-level overlay using multiple superpeers, promoting the most stable peers as superpeers, and rapidly repairing the infrequent failures or departures of superpeers. 3.4
Content Caching
In many P2P applications, once a peer p determines the peer p that is responsible for a key, p then asks p for the file associated with the key. If the path from p to p traverses a congested or low-speed link, the file transfer delay will be long. In many hierarchical DHT setups, we expect the peers in a same group to be topologically close and to be interconnected by high-speed links (corporate or university campus). By frequently confining file transfers to intra-group transfers, we reduce traffic loads on the access links between the groups and higher-tier ISPs. Such hierarchical setups can be naturally extended to implement cooperative caching: when a peer p ∈ Gi wants to obtain the file associated with some key k, it first uses group Gi ’s intra-lookup algorithm to find the peer p ∈ Gi that would be responsible for k if Gi were the entire set of peers. If p has a local copy of the file associated with k, it returns the file to p; otherwise, p obtains the file (using the hierarchical DHT), caches a copy, and forwards the file to p. Files are cached in the groups where they have been previously requested. Standard analytical techniques to quantify the reduction in average file transfer time and load on access links can be found in [14].
4
Chord Instantiation
For the remainder of this paper we focus on a specific top-level DHT, namely, Chord. In Chord, each peer and each key has a m-bit id. Ids are ordered on a circle modulo 2m (see Figure 2, left). Key k is assigned to the first peer whose identifier is equal to or follows k in the identifier space. This peer is called the successor of key k. Each peer tracks its successor and predecessor peer in the ring. In addition, each peer tracks m other peers, called fingers; specifically, a peer with id p tracks all the successors of the ids p + 2j−1 for each j = 1, . . . , m (note that p’s first finger is in fact its successor). The successor, predecessor, and fingers make up the Chord routing table. During a lookup, a peer forwards a query to the finger with the largest id that precedes the key value. The process is repeated from peer to peer until the peer preceding the key is reached, which is the “closest” peer to the key. When there are P peers, the average number of hops needed to reach the destination is O(log P ) [1]. 4.1
Inter-group Chord Ring
In the top-level overlay network, each “node” is actually a group of peers. This implies that the top-level lookup system must manage an overlay of groups, each of which is represented by a set of superpeers. Chord requires some adaptations to manage groups instead of nodes. We will refer to the modified version of Chord as “top-level Chord”.
1236
L. Garc´es-Erice et al. r
successor( n+2
i−2
r
successor(n+2
i−1
S
)
fingers r
i
n
r
S
S
r
successor(n+2 )
r
S
succ. r
finger i
S
Group Routing Table
g finger i−1
r
)
S r
pred.
finger i+1 S
S
r
r r
Fig. 2. Normal Chord routing (left) and hierarchical Chord routing (right).
Each node in top-level Chord has a predecessor and successor vector, holding the IP addresses of the superpeers of the predecessor and successor group in the ring, respectively. Each finger is also a vector. The routing table of a top-level Chord with two superpeers per group is shown in Figure 2 (right). The population of groups in the top-level overlay network is expected to be rather stable. However, individual superpeers may fail and disconnect the top-level Chord ring. When the identity of the superpeers Si of a group gi changes, the new superpeers eagerly update the vectors of the predecessor and successor groups. This guarantees that each group has an up-to-date view of its neighboring groups and that the ring is never disconnected. Fingers improve the lookup performance, but are not necessary for successfully routing requests. We lazily update the finger tables when we detect that they contain invalid references (similarly to the lazy update of the fingers in regular Chord rings [1]). It is worth noticing that the regular Chord must perform a lookup operation to find a lost finger. Due to the redundancy that our multiple superpeer approach provides, we can choose without delay another superpeer in the finger vector for the same group. To route a request to a group pointed to by a vector (successor or finger), we choose a random IP address from the vector and forward the request to that superpeer, thus balancing load among superpeers. 4.2
Lookup Latency with Hierarchical Chord
In this section, we quantify the improvement of lookup latency due to the hierarchical organization of the peers. To this end, we compare the lookup performance of the flat Chord and a two-tier hierarchical DHT in which Chord is used for the top level overlay, and arbitrary DHTs are used for the bottom level overlays. For each bottom level group, we only suppose that the peers in the group are topologically close so that intra-group lookup delays are negligible. In order to make a fair comparison, we suppose that both the flat and hierarchical DHTs have the same number of peers, denoted by P . Let I be the number of groups in the hierarchical design. Because peers are joining and leaving the ring, the finger entries in the peers will not all be accurate. This is more than probable, since fingers are updated lazily. To capture the heterogeneity of the peers, we suppose that there are two categories of peers: Stable peers, for which each peer is down with probability ps . Instable peers, for which each peer is down with probability pr , with
Hierarchical Peer-to-Peer Systems
1237
pr ps . We suppose that the vast majority of the peers are instable peers. In real P2P networks, like Gnutella, most peers just remain connected the time of getting data from other peers [15]. For the hierarchical organization, we select superpeers from the set of stable peers, and we suppose there is at least one stable peer in each group. Because there are many more instable peers than stable peers, the probability that a randomly chosen Chord node is down in the flat DHT is approximately pr . In the hierarchical system, as all the superpeers are stable peers, the probability that a Chord node is down is ps . To compare the lookup delay for flat and hierarchical DHTs, we thus only need to consider a Chord ring with N peers, with each peer having the same probability p of being down. The flat DHT corresponds to (N, p) = (P, pr ) and the hierarchical DHT corresponds to (N, p) = (I, ps ). We now proceed to analyze the lookup of the Chord ring (N, p). To simplify the analysis, we assume the N peers are equally spaced on the m ring, i.e., the distance between two adjacent peers is 2N . Our model implies that when a peer attempts to contact another in its finger table, the peer in the finger table will be down with probability p, except if this is the successor peer, for which we suppose that the finger entry is always correct (i.e., the successor is up or the peer is able to find the new successor. This assures the correct routing of lookup queries). Given an initial peer and a randomly generated key, let the random variable H denote the number of Chord hops needed to reach the target peer, that is, to reach the peer responsible for the key. Let T be the random variable that is the clockwise distance in number of peers from the initial peer to the target peer. We want to compute the expectation E[H]. Clearly E[H] =
N −1
P (T = n)E[H|T = n] =
n=0
N −1 1 E[H|T = n] N n=0
(1)
From (1), it suffices to calculate E[H|T = n] to compute E[H]. Let h(n) = E[H|T = m n]. Note that h(0) = 0 and h(1) = 1. Let jn = max{j : 2j ≤ 2Nn }. The value jn represents the number of finger entries that precede the target peer, excluding finger 0, the successor. For each of the finger entries, the probability that the corresponding peer is up is p. Starting at the initial peer, when hopping to the next peer, the query will advance 2 jn
2m /N peers if the jn th finger peer is up; if this peer is down but the (jn − 1)th finger jn −1
peer is up, the query will advance 22m /N ; and so on. Let qn (i) denote the probability that the ith finger is used. We therefore have h(n) = 1 +
jn
qn (i)h n −
i=0
2i 2m /N
(2)
The probability that the ith finger is used is given by qn (i) = pjn −i (1 − p) for i = 1, . . . , jn , and by qn (0) = pjn . Combining Equation 2 with the above expression for qn (i) we obtain h(n) = 1 + pjn h(n − 1) + (1 − p)
jn i=1
pjn −i h n −
2i 2m /N
(3)
Using this recursion, we can calculate all the h(n)’s beginning at h(0) = 0. We then use Equation 1 to obtain the expected number of hops, E[H].
Mean number of hops per look-up
1238
L. Garc´es-Erice et al.
120 110 100 90 80 70 60 50 40 30 20 10 0
224 nodes 20 216 nodes 210 nodes 2 nodes
0
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
1
Probability of node failure
Fig. 3. Nb. of hops per lookup in Chord.
P P P P
= = = =
16
2 ,I 220 , I 224 , I 224 , I
= = = =
10
2 216 220 216
Flat pr = 0.5 pr 17 22 28 28
= 0.8 43 59 83 83
Hierarchical ps = 0 5 8 10 8
Fig. 4. Flat vs. hierarchical networks.
In Figure 3, we plot the expected number of hops in a lookup as a function of the availability of the peers in a Chord system, for different values of N . With smaller values of p, although peers in the ring are not totally reliable, we are still able to advance quite quickly on the ring. Indeed, while the best finger for a target peer is unavailable with probability p, the probability of the second best choice to be also down is p2 , which is far smaller than p. Despite the good scalability of the Chord lookup algorithm in a flat configuration, the hierarchical architecture can yet significantly decrease the lookup delay. Figure 4 gives the expected number of hops for the flat and hierarchical schemes, for different values of P , I, and pr (ps = 0). We suppose in all cases groups of PI peers. Since the number of steps is directly related to the lookup delay, we can conclude that the average lookup delay is severely improved in the hierarchical DHT.
5
Conclusion
Hierarchical organizations in general improve overall system scalability. In this paper, we have proposed a generic framework for the hierarchical organization of peer-to-peer overlay network, and we have demonstrated the various advantages it offers over a flat organization. A hierarchical design offers higher stability by using more “reliable” peers (superpeers) at the top levels. It can use various inter- and intra-group lookup algorithms simultaneously, and treats join/leave events and key migration as local events that affect a single group. By gathering peers into groups based on topological proximity, a hierarchical organization also generates less messages in the wide area and can significantly improve the lookup performance. Finally, our architecture is ideally suited for caching popular content in local groups. We have presented an instantiation of our hierarchical peer organization using Chord at the top level. The Chord lookup algorithm required only minor adaptations to deal with groups instead of individual peers. When all peers are available, a hierarchical organization reduces the length of the lookup path by a factor P of log log I , for I groups and P peers. A hierarchical organization reduces the length of the lookup path dramatically when superpeers are far more stable than regular peers.
Hierarchical Peer-to-Peer Systems
1239
References 1. I. Stoica, R. Morris, D. Karger, M. Kaashoek, and H. Balakrishnan, “Chord: A scalable peerto-peer lookup service for internet applications,” in Proc. ACM SIGCOMM, 2001. 2. S. Ratnasamy, M. Handley, R. Karp, and S. Shenker, “A scalable content-addressable network,” in Proceedings of SIGCOMM 2001, Aug. 2001. 3. A. Rowstron and P. Druschel, “Pastry: Scalable, distributed object location and routing for large-scale peer-to-peer systems,” in IFIP/ACM International Conference on Distributed Systems Platforms (Middleware), (Heidelberg, Germany), pp. 329–350, November 2001. 4. B. Y. Zhao, J. Kubiatowicz, and A. D. Joseph, “Tapestry: An infrastructure for fault-tolerant wide-area location and routing,” Tech. Rep. UCB/CSD-01-1141, Computer Science Division, University of California, Berkeley, Apr 2001. 5. K. Aberer, “P-grid: A self-organizing access structure for P2P information systems,” in Proceedings of the Sixth International Conference on Cooperative Information Systems (CoopIS 2001), (Trento, Italy), 2001. 6. “Gnutella.” http://gnutella.wego.com. 7. “Kazaa.” http://www.kazaa.com. 8. S. Ratnasamy, M. Handley, R. Karp, and S. Shenker, “Topologically-aware overlay construction and server selection,” in Proceedings of Infocom’02, (New York City, NY), 2002. 9. B. Krishnamurthy, J. Wang, and Y. Xie, “Early measurements of a cluster-based architecture for P2P systems,” in ACM SIGCOMM Internet Measurement Workshop, (San Francisco, CA), November 2001. 10. B. Y. Zhao, Y. Duan, L. Huang, A. D. Joseph, and J. D. Kubiatowicz, “Brocade: Landmark routing on overlay networks,” in In Proceedings of IPTPS’02, (Cambridge, MA), March 2002. 11. M. Castro, P. Druschel, Y. C. Hu, and A. Rowstron, “Exploiting network proximity in peer-topeer overlay networks,” Tech. Rep. MSR-TR-2002-82, Microsoft Research, One Microsoft Way, Redmond, WA 98052, 2002. 12. K. W. Ross, “Hash-routing for collections of shared web caches,” IEEE Network Magazine, vol. 11, 7, pp. 37–44, Nov-Dec 1997. 13. D. Karger, A. Sherman, A. Berkhemier, B. Bogstad, R. Dhanidina, K. Iwamoto, B. Kim, L. Matkins, andY.Yerushalmi, “Web caching with consistent hashing,” in Eighth International World Wide Web Conference, May 1999. 14. J. F. Kurose and K. W. Ross, Computer Networks: A Top-Down Approach Featuring the Internet, 2nd edition. Addison Wesley, 2002. 15. E. Adar and B. A. Huberman, “Free riding on gnutella,” First Monday, vol. 5, Oct. 2000.
Enabling Peer-to-Peer Interactions for Scientific Applications on the Grid Vincent Matossian and Manish Parashar The Applied Software Systems Laboratory Department of Electrical and Computer Engineering Rutgers University, Piscataway NJ 08855, USA {vincentm,parashar}@caip.rutgers.edu
Abstract. P2P and Grid communities are actively working on deploying and standardizing infrastructure, protocols, and mechanisms, to support decentralized interactions across distributed resources. Such an infrastructure will enable new classes of applications based on continuous, seamless and secure interactions, where the application components, Grid services, resources and data interact as peers. This paper presents the design, implementation and evaluation of a peerto-peer messaging framework that builds on the JXTA protocols to support the interaction and associated messaging semantics required by these applications.
1
Introduction
The emergence of Grid applications coincides with that of Peer-to-Peer (P2P) applications such as Napster or SETI@HOME[1]. This parallel is manifested in the similarities in infrastructural requirements of Grid and P2P systems, such as the underlying decentralized (overlay) network architecture, the dynamic discovery of resources, the aggregation of distributed resources, or the need for system integrity and security guarantees. A key requirement for enabling the application-level interactions on the Grid is a P2P messaging layer that supports the interactions and their associated messaging semantics. The overall objective of this work is to prototype such a messaging framework to support Grid applications in a P2P environment. A number of messaging solutions for P2P/Grid systems have been proposed in recent years. These include Message Oriented Middleware (MOM) systems such as JMS [2], LeSubscribe [3], and IBM MQSeries[4], as well as Grid oriented systems such as GridRPC [5], NaradaBrokering [6], and ICENI[7]. This paper presents Pawn, a publisher/subscriber messaging substrate that offers interaction services for distributed object management, monitoring and steering, group formation, and collaboration through guaranteed, flexible, and stateful messaging. Pawn combines properties from messaging and P2P messaging on the Grid to provide advanced messaging semantics such as guaranteed message delivery, push, pull, transactions, and request/response interactions. It also supports synchronous and asynchronous communications, coordination through message ordering, and remote procedure calls. Unlike
Support for this work was provided by the NSF via grants numbers ACI 9984357 (CAREERS), EIA 0103674 (NGS) and EIA-0120934 (ITR), DOE ASCI/ASAP (Caltech) via grant numbers PC295251 and 1052856.
H. Kosch, L. B¨osz¨orm´enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 1240–1247, 2003. c Springer-Verlag Berlin Heidelberg 2003
Enabling Peer-to-Peer Interactions for Scientific Applications on the Grid
1241
other publisher/subscriber systems, Pawn focuses on interaction services to support application monitoring and steering, collaboration, and application execution on the Grid. This paper makes three contributions: (1) the definition of messaging requirements for scientific investigation in a P2P Grid environment, (2) the identification and implementation of corresponding services and mechanisms in Pawn, (3) the deployment of the Pawn messaging substrate and its evaluation using a “real-world” Grid application. The rest of this paper is organized as follows. Section 2 presents the motivating application process and describes the underlying interactions. Section 3 presents the design and implementation of Pawn. Section 4 describes the use of Pawn to enable the interactions required by the target application. Section 5 presents an experimental evaluation of Pawn. Section 6 presents some conclusions.
2
Motivating Application: Enabling Autonomic Oil Reservoir Optimization
The research presented in this paper and the Pawn messaging substrate is motivated by the autonomic oil reservoir optimization process on the Grid. The goal of the process is to dynamically optimize the placement and configuration of oil wells to maximize revenue. The overall application scenario is illustrated in Figure 1. The peer components involved include: Integrated Parallel Accurate Reservoir Simulator (IPARS) [8], providing sophisFig. 1. Autonomous optimization in IPARS using ticated simulation components that encapsulate complex mathematical VFSA models of the physical interaction in the subsurface. IPARS Factory, responsible for configuring IPARS simulations, executing them on resources on the Grid and managing their execution. Very Fast Simulated Annealing (VFSA), an optimization service based on statistical physics. Economic Modeling Service that uses IPARS simulation outputs and current market parameters (oil prices, costs, etc.) to compute estimated revenues for a particular reservoir configuration. DISCOVER Middleware that integrates Globus [9] Grid services (GSI, MDS, GRAM, and GASS), via the CorbaCoG [10], and DISCOVER remote monitoring, interactive steering, and collaboration services. DISCOVER Collaborative Portals providing experts (scientists, engineers) with collaborative access to the other peers components. These entities need to dynamically discover and interact with one another as peers to achieve the overall application objectives. Figure 1 shows the different interactions involved numbered 1 through 8. (1) The experts use the portals to interact with the DISCOVER middleware and the Globus Grid services to discover and allocate appropriate resource, and to deploy the IPARS Factory, VFSA and Economic model peers.
1242
V. Matossian and M. Parashar
(2) The IPARS Factory discovers and interacts with the VFSA service peer to configure and initialize it. (3) The experts interact with the IPARS Factory and VFSA to define application configuration parameters. (4) The IPARS Factory then interacts with the DISCOVER middleware to discover and allocate resources and to configure and execute IPARS simulations. (5) The IPARS simulation now interacts with the Economic model to determine current revenues, and discovers and interacts with the VFSA service when it needs optimization. (6) VFSA provides IPARS Factory with optimized well information, which then (7) launches new IPARS simulations. (8) Experts can at anytime discover, collaboratively monitor and interactively steer IPARS simulations, configure the other services and drive the scientific discovery process. Once the optimal well parameters are determined, the IPARS Factory configures and deploys a production IPARS run.
3
Design and Implementation of Pawn
A conceptual overview of the Pawn framework is presented in Figure 2 and is composed of peers (computing, storage, or user peers), network and interaction services, and mechanisms. These components are layered to represent the requirements stack enabling interactions in a Grid environment. The figure can be read from bottom to top as “Peers compose messages handled by services through specific interaction modalities”. Pawn builds on Project JXTA[11], a general-purpose peer-to-peer framework. JXTA concepts include peer, peergroup, advertisement, module, pipe, rendezvous, and security. JXTA defines protocols for : discovering peers (Peer Discovery Protocol, PDP), binding virtual end-to-end communication channels between peers (Pipe Binding Protocol, PBP), resolving queries (Peer Resolver Protocol, PRP), obtaining information on a particular peer, such as its available memory or Fig. 2. Pawn requirements stack. CPU load (Peer Information Protocol, PIP), propagating messages in a peergroup (Rendezvous protocol, RVP), determining and routing from a source to a destination using available transmission protocols (Endpoint Routing Protocol, ERP). Protocols in JXTA define the format of the messages exchanged as well as the behavior adopted on receiving a message. In Pawn, peers can implement one or more services (behaviors); the combination of services implemented by a peer defines its role. Typical roles for a peer are client, application or rendezvous. A client peer deploys applications on resources and accesses them for interactive monitoring and/or steering. It also collaborates with other peers in the peergroup. An application peer exports its application interfaces and controls to a peergroup. These interfaces are used by other peers to interact with the application. The rendezvous peer distributes and relays messages, and filters them en route to their destination.
Enabling Peer-to-Peer Interactions for Scientific Applications on the Grid
3.1
1243
Pawn Services
A network service is a functionality that can be implemented by a peer and made available to a peergroup. File-sharing or printing are typical examples of network services. In Pawn, network services are application-centric and provide the mechanisms to query, respond, subscribe, or publish information to a peergroup. Pawn offers four key services to enable dynamic collaborations and autonomic interactions in scientific computing environments. These services are Application Runtime Control, Application Monitoring and Steering, Application Execution, and Group Communication. Application Execution service [AEX]: The Application Execution service enables a peer to remotely start, stop, get the status of, or restart an application peer. This service requires a messaging mechanism supporting synchronous and guaranteed remote calls for resource allocation and application deployment (transaction oriented interaction). Application Monitoring and Steering service [AMS]: The Application Monitoring and Steering service handles Request/Response interactions, application querying (i.e. PULL), and dynamic steering (i.e. PUSH) of application parameters. It requires support for synchronous and asynchronous communications, guaranteed message delivery, and dynamic data injection (e.g. push information to an application at runtime). Application Runtime Control service [ARC]: The application runtime control service announces the existence of an application to the peergroup, sends application responses, publishes application update messages, and notifies the peergroup of an application termination. This service requires a mechanism to push information to the peergroup and respond to queries (i.e. PUSH and Request/Response interactions). Collaboration Service [Group communication, Presence]: The collaboration service supports collaborative tools and group communications. Collaborating peers need to establish direct end-to-end communications through synchronous/asynchronous channels (e.g. for file transfer or text communication), and be able to publish information to the peergroup (Transcation and PULL interactions). 3.2
Implementation of Pawn Services
Pawn builds on the current Java implementation of the JXTA protocols. JXTA defines unicast pipes that provide a communication channel between two endpoints, and propagate pipes that can propagate a message to a peergroup. It also defines the Resolver Service that sends and receives messages in an asynchronous manner. Recipients of the message can be a specific peer or an entire peergroup. The pipe and resolver service use the available underlying transport protocol (TCP, HTTP, TLS). To realize the four services identified above, Pawn extends the pipe and resolver services to provide stateful and guaranteed messaging. This messaging is then used to enable the key application-level interactions such as synchronous/asynchronous communication, dynamic data injection, and remote procedure calls. Stateful Messages: In Pawn, every message contains a source and destination identifier, a message type, a message identifier, a payload, and a handler tag. The handler tag uniquely identifies the service that will process the message. State is maintained through the payload that contains system/application parameters. These messages are defined in XML to provide platform-independence.
1244
V. Matossian and M. Parashar
Message guarantees: Pawn implements application-level communication guarantees by combining stateful messages and a per-message acknowledgment table maintained at every peer. Guarantees are provided by using FIFO message queues to handle incoming and outgoing messages. Every outgoing message that expects a response is flagged in the table as awaiting acknowledgment. This flag is removed once the message is acknowledged. Messages contain a default timeout value representing an upper limit on the estimated response time. If an acknowledgement is not received and/or the timeout value expires, the message is resent, blocking or not blocking the current process depending on the communication type (synchronous or asynchronous). The message identifier is composed from the destination and sender’s unique peer identifiers. It is incremented for every transaction during a session (interval between a peer joining and leaving a peergroup) to enable the application-level message ordering guarantees. Synchronous/Asynchronous communication: Communication in JXTA can be synchronous when using blocking pipes, or asynchronous when using non-blocking pipes or the resolver service. In order to provide reliable messaging, Pawn combines these communication modalities with stateful messaging and guarantee mechanism.
Fig. 3. AMS to ARC interaction
Dynamic Data Injection: In Pawn, every peer advertisement contains a pipe advertisement, which uniquely identifies a communication channel to it. This pipe is used by other peers to create an end-to-end channel to send and receive messages dynamically. Every interacting peer implements a message handler that listens for incoming messages on the peer’s input pipe channel. The application-sensitive data contained in the message payload can be dynamically passed to the application/service identified by the handler tag field. Remote Method Calls (PawnRPC): The PawnRPC mechanism provides the low-level constructs for building applications interactions across distributed peers. Using PawnRPC, a peer can invoke a method on a remote peer dynamically by passing its request as an XML message through a pipe. The interfaces of the methods that can be remotely invoked are published as part of the peer advertisement during peer discovery. The XML message is a composition of the destination address, the remote method name, the arguments of the method, and the arguments associated types. Upon receiving an RPC message, a peer locally checks the credentials of the sender, and if the sender is authorized, the peer invokes the appropriate method and returns a response to the requesting
Enabling Peer-to-Peer Interactions for Scientific Applications on the Grid
1245
peer. The process may be done in a synchronous or asynchronous manner. PawnRPC uses the messaging guarantees to assure delivery ordering, and stateful messages to tolerate failure.
4
Enabling Autonomic Reservoir Optimization Using Pawn
In this section we describe how the interaction services provided by Pawn are used to support the autonomic oil reservoir process outlined in Figure 1 (Section 2). Every interacting component described in Section 2 is a peer that implements Pawn services. The IPARS Factory, VFSA, and the DISCOVER middleware are Application peers and implement ARC and AEX services. The DISCOVER portals are Client peers and implement AMS and Group communication services. Key operations using Pawn are described below. Peer Deployment: The Application Execution service in Pawn uses the CorbaCoG kit (and the Globus Grid services it provides) in conjunction with the DISCOVER middleware to provide client peers access to Grid resources and services, and to deploy application peers. Autonomic Optimization: Autonomic optimization involves interactions between the IPARS simulation, VFSA service and the IPARS Factory. VFSA provides the IPARS Factory with an initial guess based on the initialization parameters entered by the client. The IPARS factory then configures and spawns an IPARS instance. The simulation output is used by the Economic Model to generate the estimated revenue. This revenue is normalized and then fed back to VFSA, which generates a subsequent guess. The process continues until the terminating condition is reached (e.g. revenue stabilizes). These interactions build on the Application Runtime Control service (ARC) and the PawnRPC mechanism. Collaboration and Interactive Monitoring and Steering: Users can collaboratively connect to, monitor, and interactively steer the IPARS Factory, VFSA, Economic model and an IPARS simulation instance. The Collaboration Services enable communication with other users, as well as the interactions between the Application Monitoring and Steering service (AMS) and the Application Runtime Control service (ARC), such an interaction is presented on Figure 3.
5
Experimental Evaluation of Pawn
Pawn was deployed on 20 hosts (1.5GHz Pentium IV processors with 512 MB of RAM, running Linux RedHat 7.2) located at Rutgers University. The Wide Area measurements were performed using two PCs (750 MHz Pentium III with 256 MB RAM and a 350 MHz Pentium II with 256 MB RAM) deployed on a private ISP. The evaluation consisted of the following experiments: Round Trip Time communication (RTT) over LAN: This experiment evaluates the round trip time of messages sent from a peer running the Application Monitoring and Steering service to peers running the Application Runtime Control service over a LAN. The message size varies from 10 bytes to 1 Megabyte. The Pawn services (AMS and ARC) build the query and response messages that are then sent using the JXTA Endpoint
1246
V. Matossian and M. Parashar
Fig. 4. RTT measurement in a LAN
Fig. 5. Effectiveness of message queueing
Fig. 6. PawnRPC over JXTA pipes
Fig. 7. Memory cost of JXTA and Pawn
Routing Protocol. The overall message transfer time is tightly bound to the performance of the JXTA platform, and is likely to improve with the next generation JXTA platform. The results are plotted in Figure 4. Note that the difference in RTT between 2 peers and 20 peers decreases as the message size increases. Effectiveness of message queuing: This experiment compares the behavior of a Pawn rendezvous peer implementing the application-level message queuing to the behavior of a core JXTA rendezvous peer. The number of messages published on the rendezvous peer range from 10 to 500. The ratio of messages received is plotted in Figure 5. It can be seen that the message queuing functionality guarantees that no application-level messages are dropped even under heavy load. Memory costs of JXTA and Pawn: In order to bootstrap a peer in Pawn, every peer has to load a core application implementing the JXTA and Pawn protocols. Figure 7 plots the memory used for loading the Java Virtual Machine, and the core JXTA protocols for every type of peer in Pawn. Pawn services add, on an average, an overhead of 20% to the JXTA core. Note that the client peer loads a Graphical User Interface in addition to the AMS and Collaboration services. Overhead of PawnRPC on JXTA pipes: Figure 6 shows a comparison between PawnRPC and JXTA pipes. Using PawnRPC, a message describing the remote call is marshaled and sent to the remote peer. The remote peer unmarshals the request, processes it before marshaling and sending back a response to the requesting peer. The marshal-
Enabling Peer-to-Peer Interactions for Scientific Applications on the Grid
1247
ing, unmarshaling, and invocation add an overhead on the plain pipe transaction. This overhead remains however less than 50% on average.
6
Summary and Conclusions
This paper presented the design, implementation, and evaluation of Pawn, a peer-to-peer messaging substrate that builds on project JXTA to support peer-to-peer interactions for scientific applications on the Grid. Pawn provides stateful and guaranteed messaging to enable key application-level interactions such as synchronous/asynchronous communication, dynamic data injection, and remote procedure calls. It exports these interaction modalities through services at every step of the scientific investigation process, from application deployment to interactive monitoring and steering and group collaboration. The use of Pawn to enable peer-to-peer interactions for an oil reservoir optimization application on the Grid was presented. Pawn is motivated by our conviction that the next generation of scientific and engineering Grid applications will be based on continuous, seamless and secure interactions, where the application components, Grid services, resources (systems, CPUs, instruments, storage) and data (archives, sensors) interact as peers.
References 1. SETI@Home. Internet: http://setiathome.ssl.berkeley.edu (1998) 2. Monson-Haefel, R., Chappell, D.: Java Message Service. O’Reilly & Associates, Sebastopol, CA, USA (2000) 3. Fabret, F., Jacobsen, A., Llirbat, F., Pereira, J., Ross, K.A., Shasha, D.: Filtering Algorithms and Implementation for Very Fast Publish/Subscribe Systems. ACM SIGMOD Record 30 (2001) 115–126 4. IBM MQSeries. Internet: http://www-3.ibm.com/ software/ts/mqseries/ (2002) 5. Seymour, K., Nakada, H., Matsuoka, S., Dongarra, J., Lee, C., Casanova, H.: Overview of GridRPC: A Remote Procedure Call API for Grid Computing. In Parashar, M., ed.: Proceedings of the Third International Workshop on Grid Computing (GRID 2002), Baltimore,MD, USA, Springer (2002) 274–278 6. Fox, G., Pallickara, S., Rao, X.: A Scaleable Event Infrastructure for Peer to Peer Grids. In: Proceedings of the 2002 joint ACM-ISCOPE conference on Java Grande, Seattle, Washington, USA, ACM Press (2002) 66–75 7. Furmento, N., Lee, W., Mayer, A., Newhouse, S., Darlington, J.: ICENI: An Open Grid Service Architecture Implemented with Jini. In: SuperComputing 2002 (SC2002), Baltimore, MD, USA (2002) 10 pages in CDROM. 8. IPARS: Integrated Parallel Reservoir Simulator. Internet: http://www.ticam.utexas.edu/CSM (2000) Center for Subsurface Modeling, University of Texas at Austin. 9. Foster, I., Kesselman, C.: The Globus Project: A Status Report. In: IPPS/SPDP’98 Heterogeneous Computing Workshop, Orlando, Florida, USA (1998) 4–18 10. Parashar, M., Laszewski, G.V., Verma, S., Gawor, J., Keahey, K., Rehn, H.N.: A CORBA Commodity Grid Kit. Special Issue on Grid Computing Environments, Concurrency and Computation: Practice and Experience 14 (2002) 1057–1074 11. Project JXTA. Internet: http://www.jxta.org (2001)
A Spontaneous Overlay Search Tree Hung-Chang Hsiao, Chuan-Mao Lin, and Chung-Ta King* Department of Computer Science National Tsing-Hua University Hsinchu, Taiwan 300
[email protected] Abstract. It is often necessary to maintain a data structure, for example a search tree, for a set of computing nodes, which are interconnected dynamically and spontaneously. Instead of relying on a centralized “official” server to maintain the data structure, a grass-root and spontaneous approach is to distribute data structure to the participating nodes by taking advantages of their distributed resources. This study shows the feasibility of such an approach by designing a distributed search tree structure, called Pyramid, to operate atop end systems without relying on a centralized server. Pyramid allows the participating nodes to manipulate the data structure transparently through high-level operations such as search, insert, delete, update and query. Pyramid is designed based on the peer-to-peer model. Its self-configuration and -healing features enable the manipulation of information in an unreliable and unstable environment. A wireless application based on a prototype of Pyramid is presented.
1 Introduction Imagining in a country fair in the not distant future, a talent contest is being held. While the official judges are busy with the scoring, the audiences also like to take an active role and make their choices. They use their PDA, cellular phone, smart watch, etc., to enter their ranking. They can also query for the current standings of the performers. A straightforward way of implementing this system is to call for the fair organizer to provide a centralized, official web site to collect, maintain, and disseminate the audiences’ scores. However, for one reason or another, we may want to resort to a more grass-root approach, in which no “official” web site is needed and the voting community is formed spontaneously using only the devices carries by the audiences. How this can be done? Apparently, the devices need to be connected through one or more networks, perhaps through an ad hoc network. Beyond the basic connectivity, we need middleware services to maintain the scores and answer queries. One possibility is to overlay a data structure, say a search tree, over the spontaneous network formed by the devices. We then have to address the following issues. * This work was supported in part by National Science Council, R.O.C., under Grant NSC 90-2213-E-007076 and by Ministry of Education, R.O.C., under Grant MOE 89-E-FA04-1-4.
H. Kosch, L. Böszörményi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 1248–1256, 2003. © Springer-Verlag Berlin Heidelberg 2003
A Spontaneous Overlay Search Tree
1249
•
We cannot rely on any particular device to host the search tree, because the device may not be reliable. The device may introduce a single point of failure and a performance bottleneck.
•
A distributed search tree co-managed by the participating devices is thus desirable. However, the devices may join and leave at any time, and the network connectivitymay be unreliable and unstable. The system thus needs to be adaptive, self-organized, and robust. The number of participating devices may be large and change constantly. The system therefore has to be scalable and incur little overhead to manage.
•
This study designs and implements a distributed search tree structure, called Pyramid, which is robust and scalable for storing, deleting, updating and querying data items scattered in a dynamic network. Pyramid is designed based on the peer-topeer (P2P) model and aimed at aggregating the distributed resources in the network to aid the manipulation of the deployed search tree. After the search tree is created, any authorized end system should be able to transparently manipulate the tree via highlevel operations. To accommodate the dynamics of the networked environment, Pyramid has to manage dynamic node configuration, utilize untrusting nodes, and tolerate faults, all without centralized administration. If nodes and communication links, overlaid by the search tree, fail, then the search tree performs self-healing. Pyramid is implemented on top of Tornado [4], a capability-aware P2P data naming, routing and storing middleware. The proposed Pyramid search tree has the following features: (1) it can utilize distributed resources and does not depend on a centralized sever. (2) It is selforganized and can be operated in an unreliable environment. (3) It is shown highly available by probability analysis and simulation. (4) It can be maintained with nearly zero-memory overhead. (5) It is very scalable by following an almost stateless design. A Pyramid tree based on such a wireless network is deployed. An application, the virtual music scoring board, is implemented to demonstrate the use of Pyramid.
2 Related Works Distributed data structures have been investigated by [3][5][6]. These studies assumed that each node in a system had global knowledge of all participating nodes, including their network addresses. This knowledge was used to deliver messages to others. In [6], a distributed linear hash structure was proposed, and the issues of distributing hash data structures in a cluster computer environment were investigated in [3]. Distributed tree structures were studied in [5], which similarly assumed that each node had global knowledge of the network address of every node within a single administrative domain. The targeted system was reliable but was centrally controlled. Pyramid considers a computing environment which is quite different from the above. The environment consists of dynamic nodes assembled in an ad hoc fashion centralized and artificial control is infeasible. Information retrieval can be roughly categorized as (1) a search for an exact keyword, such as in [4][7][8][9][10]; (2) a search for partial keywords matching, such as in [1][2]. These studies either are based on flooding or rely on a virtual data
1250
H.-C. Hsiao, C.-M. Lin, and C.-T. King
structure to organize related data items and/or machines. Pyramid is different from the above by not concentrating on a single type of search, but rather directly implementing an overlay search tree in an untethered environment. Search trees provided by Pyramid are capable of supporting structured queries such as SQL that cannot be implemented by methods mentioned above.
3 Pyramid The ubiquitous wired and wireless Internet allows scattered resources in devices such as sensors, PDAs, desktop PCs, workstations and servers to be aggregated to provide distributed data and computation services. Pyramid is a generic middleware on top of Tornado that provides distributed tree data structures in spontaneous network environments. Each node participating in Pyramid contributes some of its storage and computational power to help assemble and access the tree structures in the system. A participating node can act as a client to request tree operations from other nodes in Pyramid, while simultaneously act as a server to supply data items stored in the trees to other nodes. 3.1 Overview Pyramid maintains search tree data structures over spontaneous networks. Each participating node can construct a tree structure that will be maintained by multiple nodes in the system. Nodes can access a tree using high-level operations, described below. The integrity of constructed data structures can be ensured by the RSA algorithm, for example. Nodes that intend to access a particular tree must first obtain the corresponding public keys, and only those nodes with private keys can modify the contents of such a data structure. Pyramid is implemented on top of Tornado and is therefore built according to the hash-based addressing scheme. This allows Pyramid to support nomadic data structures by decoupling the location of a data structure from its address. The hashbased scheme also supports anonymity. In other words, nodes that intend to access tree structures do not necessarily know where they should locate or how to organize the data structure. The scheme thus leverages the reliability of the underlying system and the durability of the data structures, while reducing malicious attacks on the nodes. Pyramid provides the following interfaces to access the data of a search tree. • create (s, t ) : Construct a search tree s with a lease time t in Pyramid.
•
insert (s, v ) : Insert a key value v into the named tree s.
•
delete (s, v ) : Remove v from s.
•
update (s, v1 , v2 ) : Update a key value v1 in s to v 2 .
•
search (s, v ) : Search for exact v in s.
•
query (s, v1 , v2 ) : Collect a set of values from v1 to v 2 in s.
A Spontaneous Overlay Search Tree create, insert, delete, update, search, query
create, insert, delete, update, search, query
1251
create, insert, delete, update, search, query
Arrange a meeting from 14:00~15:30 tomorrow
Search all shops that have at least 8 seats available from 14:00~15:30 tomorrow
12:30 | 20:30
Bob
10:00 | 16:30
8:30 | 17:30
create, insert, delete, update, search, query
create, insert, delete, update, search, query
(a)
14:00 | 16:30
11:30 | 18:30 9:00 | 17:30
7:00 | 18:00
(b)
Fig. 1. A usage example of Pyramid. (a) There are two ad hoc Pyramid spaces, one organizes the schedules of colleagues and the other maintains the reservation information of each participating coffee shop. (b) A tree structure represents the Pyramid space for the schedules of colleagues by referring to the time values in Italics.
Figure 1(a) illustrates an example usage of Pyramid. Note that with the aid of Tornado, Pyramid can function over a global area. The example here, which shows ad hoc Pyramid spaces in a localized setting, demonstrates the diversity of Pyramid. In this example, two Pyramid spaces are presented: the schedules of colleagues, and the reservation information of coffee shops. Suppose Bob would like to call for a meeting with his colleagues currently in the office. He issues a request query to the Pyramid space “the schedules of colleagues”. The issued request specifies the date and time slots for meeting. Meanwhile, Bob searches all nearby coffee shops to reserve a place for the meeting. Figure 1(b) shows a Pyramid tree that represents the schedules of colleagues. Each colleague may insert, delete and update his own schedule in the Pyramid tree. Note that a colleague may enter or leave the office at any time, and thus his schedule may join or depart from the Pyramid tree. Similarly, the Pyramid space, “reservation information of coffee shops”, tracks the seat availability of each participating coffee shop. 3.2
Design
Pyramid maintains an m-way search tree over a spontaneous network. A tree node r hosts m key values and pointers, and has the following properties. (1) Key1r < Key 2 r < Key 3 r < L < Key mr and " (2) Keyvr < Keyvr < Key(v +1)r , " where Key kr denotes the keys hosted by node r, 1 ≤ k ≤ m and Key1r = −∞. Keyvr
represents a key in the v-th sub-tree Sub - treev associated with node r, where
1252
H.-C. Hsiao, C.-M. Lin, and C.-T. King
1≤ v ≤ m −1 . No more than m pointers point to m m-way search sub-trees in each tree node. For example, in Figure 2, Key1" , Key 2" , Key 3" and Key m" in the sub-trees
Sub - tree1 , Sub - tree2 , Sub-tree3 and Sub - treem , respectively, satisfy Key1 < Key1" < Key2 , Key2 < Key2" < Key3 , Key3 < Key3" < Key4 and Keym < Keym" . 3.2.1 Naming Each tree node in Pyramid is addressed using the hash-based naming scheme supplied Tornado. Thus, given its hashing ID, a node can be dynamically mapped to a virtual home in Tornado. To initialize a search tree in Pyramid, a request is first sent to the root of the tree. The root controls access to data items in the tree. The hashing value of the root of a tree named C (for example, schedules of colleagues), is calculated as, (3) (C ) ,
where H is the hashing function applied by Tornado. The hashing value Civ of each
Sub - treev ’s root of a node with the hashing value i in C is defined as if 1 ≤ v ≤ m − 1, then, Civ = H (C ) ⊕ Key vi ⊕ Key (v + 1)i . (4) Otherwise, if v = m , then, (5) Civ = H (C ) ⊕ Keyvi , where ⊕ stands for concatenation. Notably, (1) each tree node also maintains a pointer that points to the hashing address of the node’s parent. (2) Given the naming scheme for each sub-tree, Pyramid do not need to maintain pointers that point to a set of sub-trees, providing state-less tree construction and maintenance.
(
(
)
(
))
3.2.2 Creating a Tree First, the hashing value of the namespace that represents a given tree structure is calculated. Then, the tree root node is stored in the corresponding virtual home with the hashing key closest to the hashing value of the tree root node. Notably, Tornado provides the Publish operation, where Publish(x, y ) stores a data item to a virtual home with the hashing key closest to x in the Tornado space called y. Since the virtual home that hosts a tree root node in Pyramid may fail and the communication links to the home may be broken, Pyramid relies on the fault-resilient route supported by Tornado. The hash-based naming delay the binding of the actual network address (for example, the IP address) of a tree root node to the name of the overlaying virtual home. The k-replication mechanism of Tornado can further increase the reliability of Pyramid. The mechanism sends replicas of k identical copies to k Tornado virtual homes whose IDs are closest to that of the tree root node. Each Pyramid tree root node is thereby implicitly replicated to k spaces deployed over distinct Tornado virtual homes. When the home that hosts a Pyramid root node fails, the contents stored in the failing node in all instance can be recovered from the replica homes. The
A Spontaneous Overlay Search Tree
1253
tree data structure need not be reconstructed since the replica home already has the most up-to-date key values maintained by the failing home. 3.2.3 Exact Search To find a particular key in a Pyramid tree, a search request is firstly sent to the tree root node. This is done by issuing a routing request to a virtual home in Tornado whose ID is closest to the hashing value of the designated namespace. After the virtual home has been consulted, it fetches the keys associated with the designated Pyramid tree. If the consulted home cannot yield the requested key, the request is routed to a virtual home whose ID is closest to that of the sub-tree’s pointer, where such a sub-tree may store the requested key (Line 6.c). If the search is successful, the hashing address of the tree node that hosts the requested key is returned. Otherwise, the hash ID of a tree node that can host the requested key is returned. 3.2.4 Key Insertion To insert a key into a Pyramid tree, the search algorithm is first performed. This algorithm routes the insertion request to a tree node r that is responsible for accommodating the inserted key. If the inserted key cannot be found in tree nodes along the route toward node r, then node r stores the inserted key while examining whether the number of valid keys it has maintained exceeds. If the number of keys exceeds m, then node r informs its parent to segment the keys hosted by node r. The parent accomplishes the segmentation by issuing Publish operations to introduce two tree nodes as the node’s sub-trees. One tree node replicates the first to the (m 2 − 1) th keys from node r and the other copies the (m − m 2) -th to the m-th keys. Then,
node r’s parent performs similar operations after the m 2 -th key hosted by node r is
stored in node r’s parent. Note that after node r’s parent inserts node r’s m 2 -th key into its local storage, the pointer that points to node r is no longer maintained and the keys stored in node r are gradually removed when their associated time contracts have expired. To maintain the reliability of a tree, first, tree nodes introduced by segmentation are spontaneously replicated to k virtual homes in Tornado. The root node mentioned in Section 3.2.2 is also replicated. After a virtual home hosting a particular tree node fails or its communication link is broken, requests to such a failing home are routed to another virtual home that includes identical copies which can be provided by the failure home. Second, producers of data should proactively and periodically refresh published keys using operations similar to those performed by the insertion algorithm. If a key to be refreshed does not appear in the designated tree, then it is added thereto. Third, the tree constructor periodically issues the re-publishing messages to the tree roots that it constructed. When a tree root receives such a message, it repeatedly broadcasts the re-publishing messages to each of its sub-trees according to the valid keys that it maintains. Conceptually, the re-publishing messages are epidemically delivered to all tree nodes. If the virtual home whose hash address is closest to the tree node’s ID i of the named tree C that receives the re-publish message, is a newly joining home and could be responsible for hosting the key values in the sub-tree Civ ,
1254
H.-C. Hsiao, C.-M. Lin, and C.-T. King
then this virtual home allocates storage space to the tree node that represents key values from Key vi to Key v (i +1) if 1 ≤ v ≤ m − 1 or the values above Key vi otherwise. This allocation ensures that a Pyramid tree is deployed over Tornado virtual homes whose IDs are closest to those of tree nodes. 3.2.5 Deleting and Updating Keys In Pyramid, only a data producer can erase the keys generated by itself. To delete a key x, a data producer can explicitly nullify the time contract of key x or simply not re-publish such a key. After key x’s time contract has expired or is nullified, it is removed from its associated tree. Then, the tree node with the hashing address i, hosting key x, re-publishes a new sub-tree to host key values from Key (v −1)i to
Key (v +1)i if Key (v −1)i < key x Key (m −1)i if key x is Key mi .
< Key (v +1)i if 1 < v ≤ m − 1 , or key values above
Note that key u originally appears in the sub-tree Sub-treev-1 or Sub-treev . It will not be available if the sub-tree responsible for the keys from Key (v −1)i to Key (v +1)i is renewed following the deletion of a key. However, a data producer will re-publish the keys it generated and then key u will be made available by its re-insertion into the tree (Section 3.2.4). To update a key from x1 to x 2 in Pyramid, x1 can firstly be simply deleted from the designated tree and then x 2 inserted. After a key is deleted or updated, the probability of failing to access a key due to changes in the mapping from tree nodes to Tornado nodes previously stored in a complete m-way search tree with height h can be similarly estimated as ∇(m, h ) , as shown in Equation 9, where ∇(m, h ) → 0 when h → ∞ .
3.2.6 Sub-Range Query To collect keys with values from v1 to v 2 in a Pyramid tree, a corresponding request is firstly issued to the tree root node of the designated tree. The tree root then invokes several sub-range queries if multiple keys have values between v1 and v 2 . Tree nodes that receive the sub-range query requests perform similar operations.
4 Implementation and an Application Pyramid has been implemented on top of a light Tornado, called Trailblazer, over a wireless network. Each wireless device in the Trailblazer network is a WINCE-based HP568 PDA. The PDA is equipped with a 206 MHz Intel SA-1100 microprocessor, 64 Mbytes RAM and an 802.11b Wireless CF interface. The wireless devices are randomly named using a uniformly random hash function. Trailblazer guarantees that a message with a designated hash address can be efficiently routed to a wireless device whose hash key is closest to the specified one. Each wireless device provides
A Spontaneous Overlay Search Tree
(a)
(b)
1255
(c)
Fig. 2. The user interfaces of the virtual music scoring board, where (a) shows the scoring interface, (b) shows the query interface and (c) shows the results for the top-10 favorite records.
some storage space and every device in Trailblazer can access the aggregated storage space. As a proof-of-concept, an application called the virtual music scoring board is implemented. Users can dynamically participate in the virtual music scoring board and access related information, e.g. the top-ten most popular music records. Users can also enter their scores into the virtual music scoring board. The trailblazer network is dynamically organized on top of the participating wireless devices. One Pyramid tree, based on the Trailblazer network, is deployed and used to handle all the music records that the participating users are interested. The Pyramid tree is based on the lexical order of the named musical records. Each user has a local profile that represents a set of favorite musical records with associated grades. Figure 2(a) shows that each user can mark a particular record as “good”, “fair” or “poor”. A user can submit his or her scores to the shared Pyramid tree. Each user can then search for the most popular music record or submit a range query for finding the records that are from x % to y % most popular, where x is smaller than y (Figure 2(b)). A user can also query the score of a particular album submitted by all participating users. Figure 2(c) presents the top-ten popular music records.
5 Conclusions This study presents a novel design for search tree data structures, which is based on the peer-to-peer model. Pyramid can utilize resources of widely distributed end systems without centralized administration. A set of rich operations, including search, insert, delete, update and query, is provided to manipulate data items in the tree. The search tree can be adapted to the changing environment while providing robust operations and ensuring data availability. As a proof-of-concept, a wireless
1256
H.-C. Hsiao, C.-M. Lin, and C.-T. King
application based on a prototype of Pyramid is presented. We believe that Pyramid is generic and it can be applied to other peer-to-peer infrastructures.
References [1]
E. Cohen, A. Fiat, and H. Kaplan. “A Case for Associative Peer to Peer Overlays,” In ACM Workshop on Hot Topics in Networks (HotNets-I), October 2002. [2] Crespo and H. Garcia-Molina. “Routing Indices for Peer-to-Peer Systems,” In International Conference on Distributed Computing Systems, pages 19–28, July 2002. [3] S. D. Gribble, E. A. Brewer, J. M. Hellerstein, and D. Culler. “Scalable, Distributed Data Structures for Internet Service Construction,” In USENIX Symposium on Operating Systems Design and Implementation, October 2000. [4] H.-C. Hsiao and C.-T. King. “Tornado: Capability-Aware Peer-to-Peer Storage Networks,” In IEEE International Conference on Parallel and Distributed Processing Symposium (IPDPS 2003), April 2003. [5] Kroll and P. Widmayer. “Distributing a Search Tree among a Growing Number of Processors,” In Proceedings of ACM SIGMOD Conference, pages 265–276, June 1994. [6] W. Litwin, M.-A. Neimat, and D. A. Schneider. “LH*—A Scalable, Distributed Data Structure,” ACM Transactions on Database Systems, 21(4), pages 480–525, December 1996. [7] S. Ratnasamy, P. Francis, M. Handley, R. Karp, and S. Shenker. “A Scalable ContentAddressable Network,” In ACM SIGCOMM, pages 161–172, August 2001. [8] Rowstron and P. Druschel. “Pastry: Scalable, Distributed Object Location and Routing for Large-Scale Peer-to-Peer Systems,” In Proceedings of the IFIP/ACM International Conference on Distributed Systems Platforms (Middleware 2001), November 2001. [9] Stoica, R. Morris, D. Karger, M. F. Kaashoek, and H. Balakrishnan. “Chord: A Scalable Peer-to-Peer Lookup Service for Internet Applications,” In ACM SIGCOMM, pages 149– 160, August 2001. [10] B. Y. Zhao, J. D. Kubiatowicz, and A. D. Joseph. “Tapestry: An Infrastructure for FaultTolerant Wide-Area Location and Routing,” Technical Report UCB/CSD-01-1141, April 2000.
Fault Tolerant Peer-to-Peer Dissemination Network Konstantinos G. Zerfiridis and Helen D. Karatza Department of Informatics, Aristotle University of Thessaloniki, 54124 Thessaloniki, Greece {zerf,karatza}@csd.auth.gr
Abstract. The widespread use of broadband networks gave new ground for Peer-to-Peer systems. The evolution of these systems made P2P file sharing networks one of the most popular ways of sharing content. Due to their distributed nature, such networks tend to be a reliable source for highly anticipated files. However, along with the benefits of P2P networks, certain patterns became apparent. Uneven flow of data and intersperse congestion points could result on increased mean response time or even network failure. In this paper the structure of Peercast, an agent based dissemination network, is presented. Emphasis is given to the self organizing nature of the dissemination tree, and simulation results depict its behavior under strenuous conditions.. . .
1
Introduction
While today the servers are able to acquire more bandwidth, they can not keep up with the rapidly increasing requests of the users. The demand for faster service increased as broadband connections became available. But if a file of considerable size has to be disseminated to a considerable amount of receivers, the network could be saturated quickly, clogging the host computer. Such is the case for example when any highly anticipated software is released and several people are trying to download it at the same time. This became known as the midnight madness problem [1]. As today’s needs for data transfer steadily increase, traditional ways of making data available to the masses become obsolete. Conventional FTP servers can no longer serve as a way of distributing large amounts of data. Mirroring the required content on several dispersed servers, cannot always compensate for the rapid traffic increase. The main architecture used for casting data through the Internet is IP multicast, which mainly targets real time non-reliable applications. It extends the IP architecture so that packets travel only once on the same parts of a network to reach multiple receivers. A transmitted packet is replicated only if it needs to, on network routers along the way to the receivers. Although it has been considered as the foundation for Internet distribution and it is available in most routers and on most operating systems, IP multicast has not so far lived up to early expectations. Its fundamental problem is that it requires that all recipients H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 1257–1264, 2003. c Springer-Verlag Berlin Heidelberg 2003
1258
K.G. Zerfiridis and H.D. Karatza
receive the content at the same time. The most popular solution to this problem was to multicast the content multiple times until all of the recipients obtain it. IP multicast might be considered ideal for applications that require relatively high and constant throughput but not much delay. However it is not suitable for applications that may tolerate significant delays but no losses. This is the case with file distribution. These days, a new way of disseminating files emerged. File sharing networks [2] are perhaps the most commonly used Peer-to-Peer applications. Such systems have been used for diverse applications: combining the computational power of thousands of computers, forming collaborative communities, instant messaging, etc. P2P file sharing networks’ main purpose is to create a common pool of files where everybody can search and retrieve any shared files. Depending on the algorithm used, these sharing networks can be divided in two groups. Networks that maintain a single database of peers and their content references are known as centralized. Such file sharing networks [3] have several advantages, such as easy control and maintenance, and some disadvantages as, for example, server overload. On the other hand, dynamically reorganizing networks such as Gnutella [4], have a rather more elaborate service discovery mechanism, avoiding this way the use of a centralized server. Those kinds of networks are known as decentralized, and their main advantage is the absence of a single point of failure. File sharing networks had never been designed for file dissemination. Nevertheless people turn to them to find highly anticipated software or even video files, when the official server stops responding due to high demand. Extensive research has been done about how existing P2P networks operate over time and how they can be optimized [4,5]. However, the dissemination process of highly anticipated files on P2P networks over unreliable network connections remains unexplored. Peercast, a P2P network first presented in [6], is designed to assist the dissemination of a file in a heterogeneous network of clients. The purpose of this paper is to show Peercast’s performance in the long term under different network conditions. Simulation results depict how network failures can affect this process. The structure of this paper is as follows. In section 2 the Peercast’s structure is shown, along with its latest extensions. Section 3 elaborates on the simulation model of the system and the simulated network failures. The results and drawn conclusions are summarized in section 4 and finally, section 5 presents suggestions for further research.
2
The Network
When a file needs to be downloaded by more clients than the server can handle, alternative algorithms have to be utilized. The naive way of avoiding retransmissions is to pipeline the file through all the clients. But this is not a viable solution because clients might have to indefinitely wait to be served. The proposed algorithm uses a self-organizing tree of clients. The server can upload the file to a certain number of clients simultaneously. When the server
Fault Tolerant Peer-to-Peer Dissemination Network
1259
successfully uploads a file to a client, it keeps a reference of this client to a short list (up to 100 entries). The server has a queue, but most of the clients are expected to find this queue full. This is the case especially at the beginning of the dissemination process, as clients arrive faster than the server can handle. In this case, the server sends to the client the list of clients that already downloaded the file. This way, the new client can download the file from a peer that was already served, removing the congestion from the server. When a client finishes the download, it acts as a server for other clients. Similarly to the server, the clients have a short queue. If a client A requests the file from a client B that has it, and that client B can not serve client A immediately, A is queued. If the queue is full, client B sends its own list of clients that it served to client A, so that it can continue searching. If a client is not able to be served or queued, it retries after a certain period of time to contact the server. The peers are not expected to stay on-line for ever. But when a peer leaves the network, the dissemination tree is left in an inconsistent state. That’s because the clients who were served from that peer are no longer accessible from the tree’s root. In order to take advantage of all the peers that are willing to help in the dissemination process, the clients that are not occupied by serving other clients, periodically check with their parent peer. If the parent is not on-line or it is not accessible due to network failure, the client contacts the server and assigns itself as the server’s child. If this fails, it requests from the server its list of served clients and tries to assigns itself to one of those clients. If this fails also, the client waits for a certain amount of time and retries. As it was mentioned earlier, in order to avoid server explosion, the list of children that the server has, is limited. If this list is full and a new child has to be added, the server removes from the list its oldest child, accepts the new one, and forces the new peer to adopt the removed client to its own list. This way, the server always has a list of serving peers that recently became available and therefore are more likely to have empty queues and stay on-line longer. In order to utilize all the available upload bandwidth, a single peer can serve several clients concurrently. Additionally, each client can initiate multiple concurrent download connections in order to utilize all the available download bandwidth. At the end of the transfer, the downloading client chooses randomly one of the assisting peers and requests to be listed on that one. The way to derive the optimal number of simultaneous upload connections and queue size is discussed in [6]. In brief, Peercast dynamically increases the number of connection slots used when the rate of arrivals balances with the rate of clients being served as shown in figure 1a. As a mean of keeping track of the number of served clients, the server instructs a small percentage (5%) of the arriving clients, to send back a message when they finish downloading the file. This way the server can estimate when the balance will occur. At that point it propagates a message to all the peers in its list and to every new client in order to increase the number of concurrent connections and decrease the queue size.
1260
K.G. Zerfiridis and H.D. Karatza 500000
90000
450000
80000
400000
70000
350000
Mean response time
100000
Number of clients
60000 50000 40000 30000
300000 250000 200000 150000
20000 100000 10000 50000 0 0
25
50
75
100
125
150 175 200 Time (hours)
225
250
total served clients
on-line clients
served, on-line clients
arrivals
275
300
325
0 12
36
60
84
108
132 156 180 Hours in simulation
0% failure
40% failure
204
228
252
276
80% failure
Fig. 1. a) Network’s state over time, b) Mean response time in 12-hour intervals according to each client’s arrival
Several issues arise about the performance of this algorithm under different network conditions in a heterogeneous network of clients. For example, how is the mean response time affected by several local congestion points or network failures? Can such problems affect dramatically the dissemination process? How does the system respond if a considerable number of peers refuse to assist?
3
Simulation Model
In this section details are presented about the simulation model for the proposed network, and show how different strategies might affect the dissemination process. The system was populated with clients arriving according to the exponential distribution. The simulation period was set to be 2 weeks (1209600 seconds). During the first week the mean interarrival time was incremented linearly from 5 to 20 sec in order to simulate demand on a highly anticipated file. For the second week the exponential distribution was used with 20 sec mean interarrival time. The file size was set to be 650MB (the size of a full CD). All the clients that populated the system were set to have broadband connections to the Internet, resembling cable modems and DSL. This is done in order to use a realistic model. As in many cases, such connections have different download and upload speeds. Four different categories of users were used. The first category (10% of the clients) had download and upload speed of 256 Kbps, the second (40%) had 384 Kbps and 128 Kbps, the third (20%) had 384 Kbps (download and upload), and the fourth (30%) had 1.5 Mbps and 384 Kbps respectively. This configuration is a theoretical model, and is used to compare how the same network performs under different conditions. These kinds of clients are always on-line. However, they are not expected to share that file for ever. Therefore they were set to leave the dissemination network with exponential distribution and mean time of four days. The server (a client from category 4) was set to never go off line. An additional difference
Fault Tolerant Peer-to-Peer Dissemination Network
1261
between the server and the clients is that the server keeps a limited list of up to 100 clients that it served, whereas the clients have an unlimited list. That’s because clients are not expected to stay on the network for ever, and therefore they do not run the risk of collapsing because of an overwhelming list. The actual connection speed between two clients is calculated at the beginning of each session, taking into consideration the theoretical maximum speed they could achieve and an exponentially distributed surcharge, in order to simulate additional network traffic and sparse bottlenecks. If a new client cannot be served or queued immediately, it waits for 600 seconds and retries. At the beginning of the dissemination, two download and two upload connections were used in order to speed up the creation of a critical mass of served clients. The critical mass is the point where the rate of served clients in the system starts to decline (figure 1a). That happens when the rate of arriving and the rate of departing clients balance out. When the server estimates that the critical mass has been reached, it propagates a message through the peers in its list, notifying the clients to change the upload/download slots from 2 to 4. Additionally, the waiting queue on each client drops from 8 to 4 entries. The number of concurrent uploading and downloading streams at any time and the queue size are derived from simulation results showed in [6]. This switch is done in order to reduce mean response time, as newly arriving clients should not be queued on long queues. When the critical mass has been reached, they are more likely to find service from clients found deeper in the dissemination tree, optimizing this way all the available network’s resources. As it was mentioned earlier, the behavior of this network can change significantly under certain conditions. The system’s performance is investigated at the beginning (2 weeks) of the dissemination, under different conditions. Our focus is on how the system behaves under network failure. More specifically, the simulations tested the system’s performance when a certain percentage of connections failed. At any given time, the connection between any two clients (including the server) was set to have a predefined chance of failure. The system was tested using 0, 40 and 80 percent chance that any given client cannot contact another peer in order to be served or queued on it. This is done in order to simulate local network bottlenecks and network or system failures. If a client cannot contact a peer, it tries to find another. If no clients are found, it contacts the server to request an updated list of clients. In the case that even the server is not accessible, it retries after 600 seconds. Additional simulations show the system’s behavior when a percentage of the clients refuse assistance to other peers. In this case it is assumed that those clients go off-line immediately after they finish the download, and do not rejoin the network later on. This is expected to decrease dramatically the performance of the dissemination process. Nevertheless it is a behavior that can be expected. We test the system’s performance when 0, 10 and 40 percent of the clients depart from the network immediately after they have been served. All of these simulations are done using a reliable network (0% connection failures). It should be noted that all network failure simulations were done using 10% of such clients.
1262
K.G. Zerfiridis and H.D. Karatza 35000
90000 80000
30000
70000 On-line & served clients
Total clients served
25000 60000 50000 40000 30000
20000
15000
10000 20000 5000
10000 0
0% failure
40% failure
80% failure
0% failure
Time (hours) 40% failure
333
318
304
289
275
260
246
231
217
203
188
174
159
145
130
87
116
73
101
58
44
0
29
15
333
318
304
289
275
260
246
231
217
203
188
174
159
145
130
116
87
73
101
58
44
0
29
15
0 Time (hours)
80% failure
Fig. 2. Network failure tests over time: a) total clients served, b) served clients that are still on-line
The theoretical case that all the clients are willing to cooperate in the process for a certain period of time is examined in order to compare the degradation of performance that occurs to the rest of the cases.
4
Simulation Results and Conclusion
Figure 2a reveals that local network or system failures have an effect on the dissemination process, but not significant. More specifically, at the beginning of the dissemination there is a drop at the number of served clients. This is more obvious in the 80% network failure simulation. Nevertheless, as shown in table 1, the mean response time increased only by an average of 3.5% for the 40% case and 8.5% for the 80% case. This disproportional drop of performance is justifiable, as clients that are temporarily inaccessible by one peer, can be accessible by another. Figure 2a depicts that the network’s resources are volatile at the beginning of the process, but as more peers join the network, their resource utilization increases steadily. In figure 2b, the number of currently on-line peers is shown to be increased in the case of 80% network failure. That’s because the server is not able to determine accurately when the critical mass has been built. This is also responsible for the higher mean response time shown in table 1. The increased mean response time in all cases can be explained as the clients that arrive early on the dissemination process have to wait for a long period of time to be served. When the rate of arrivals balances with the rate of clients being served, the mean response time stabilizes to lower levels. Therefore, clients arriving later in the system benefit from a faster service. This is depicted in figure 1b where mean response time is shown in 12 hour intervals according to each client’s arrival in the system. An additional test that was executed at the end of the simulations showed the integrity of the dissemination tree under these conditions. More specifically, in the case of the 40% network failure, by iterating through the tree, 0.5% of the
Fault Tolerant Peer-to-Peer Dissemination Network
1263
Fig. 3. Percentage of clients departs from the network immediately after completing the download a) total clients served over time, b) served clients that are still on-line
currently on-line serving-peers were found to be unreachable. This percentage increased to 10% for the 80% case. Those clients were in a timeout loop in order to try to reassign themselves in the tree. This shows that the nature of the self-organizing tree is relatively reliable to harsh conditions. In figure 3a, the degradation of performance is shown to be proportional to the number of clients that depart from the system without assisting in the dissemination process. This is also shown in table 2 where the mean response time increased by an average of 4% when 10% of the clients leave the system as soon as they are served, and by 45% in the 40% case. Additionally, figure 3b shows that the critical mass is reached much earlier in the first case. That’s because it is essential for the performance of the dissemination process that each client within this system acts as a server for other peers when it is served. Further simulation results, not presented here due to space limitations, show that if a file of smaller size is used the impact of the clients that refuse to serve other peers is getting smaller. Overall, the system’s behavior in strenuous conditions can be considered satisfactory. Its performance decrease is minimal in network failures, and the decentralized self-organizing tree of served clients proved to be reliable and scalable. The utilization of peer-to-peer technology for this task revealed a flexible way of reliably disseminating static data in a high number of clients, as long as the interested parties act collectively.
5
Future Work
Additional simulation experiments are under way, using distributions varying with time for more realistic long run simulations, as depicted in [7]. Peercast is an evolving platform. For the current P2P network implementation we used a monolithic approach: all the data has to be sent to a client, before this client starts sending it to another peer. A new version that replicates groups of 256KB packets, to adjacent peers as they arrive, is under way. This is expected to
1264
K.G. Zerfiridis and H.D. Karatza
Table 1. Mean response time over different amount of network failures (10% dropouts) Net failures 256/256 384/128 384/384 1.5/384 0% 40% 80%
210377 216783 227610
199498 207291 216455
199672 207037 217156
182107 188381 197314
Table 2. Mean response time over different percentage of clients leaving immediately Dropouts 256/256 384/128 384/384 1.5/384 0% 10% 40%
201812 210377 289458
192129 199498 278221
192077 199672 278129
174603 182107 257649
alleviate the problems that are caused from peers that go off-line immediately or soon after they finish downloading the requested file. The synchronization between the peers is done in predetermine time intervals, called epochs [8]. The peers are segmented in virtual groups according to their bandwidth and the epoch size depends on an estimation of the minimum bandwidth between the peers that form each dissemination group. Simulation results from this network are expected to show alleviation of several issues raised in this paper.
References 1. Schooler, E., Gemmell, J.: Using Multicast FEC to solve the Midnight Madness Problem. Technical Report, Microsoft research (1997) 2. Parameswaran, M., Susarla, A., Whinston, A.B.: P2P Networking: An Information Sharing Alternative. IEEE Computer Journal, Vol. 34. (2001) 31-38. 3. Shirky C.: Peer-to-Peer: Harnessing the Benefits of a Disruptive Technology / Listening to Napster. I.A. Oram (ed.), O’Reilly & Associates. 4. Ripeanu, M., Foster, I., Iamnitchi, A.: Mapping the Gnutella Network: Properties of large scale peer to peer systems and implications for system design. Internet Computing Journal, IEEE Computer Society (2002) 50–57 5. Markatos, E.P.: Tracing a large-scale Peer to Peer System: an hour in the life of Gnutella. Proceedings of the CCGrid 2002, Second IEEE/ACM International Symposium on Cluster Computing and the Grid (2002) 65–74 6. Zerfiridis K.G., Karatza, H.D.: Large Scale Dissemination using a Peer-to-Peer Network. Proceedings of the 3rd IEEE/ACM International Symposium on Cluster Computing and the Grid 2003, Tokyo (2003) 421–427 7. Karatza, H.D.: Task Scheduling Performance in Distributed Systems with Time Varying Workload. Neural, Parallel & Scientific Computations, Vol. 10. Dynamic Publishers, Atlanta, (2002) 325–338 8. Karatza, H.D., Hilzer R.C.: Epoch Load Sharing in a Network of Workstations. Proceedings of the 34th Annual Simulation Symposium, IEEE Computer Society Press, SCS, Seattle, Washington (2001) 36–42
Exploring the Catallactic Coordination Approach for Peer-to-Peer Systems* 2
Oscar Ardaiz1, Pau Artigas1, Torsten Eymann , Felix Freitag1, Roc Messeguer1, 2 Leandro Navarro1, and Michael Reinicke 1
Computer Architecture Department, Polytechnic University of Catalonia, Spain {oardaiz,partigas,felix,meseguer,leandro}@ac.upc.es 2 Institute for Computer Science and Social Studies Albert-Ludwigs-University Freiburg, Germany {reinicke,eymann}@iig.uni-freiburg.de
Abstract. Efficient discovery and resource allocation is one of the challenges of current Peer-to-Peer systems. In centralized approaches, the user requests can be matched to the fastest, cheapest or most available resource. This approach, however, shows scalability limits. In this paper, we explore the catallactic coordination as a decentralized economic approach for resource allocation in peer-topeer networks. The economic model of the catallaxy is based on the selfinterested maximization of utility and the negotiation of prices between agents. We evaluate the feasibility of our approach by means of simulations and compare the proposed system with a centralized baseline approach. Our results indicate that while in the catallacic approach the number of control messages exchanged between the peers grows due to the negotiation process, its service provision rate is fairly constant in different dynamic environments.
1 Introduction Peer-to-peer (P2P) systems are a class of distributed systems or applications to achieve a certain functionality in a decentralized manner. In this model, peers give resources and receive resources in turn. These resources can consist of computing power, storage of data or content, network bandwidth or presence. Application domains are distributed computing, data and content sharing, and collaborative applications. A number of successful applications like Napster [9], Gnutella [5], and Freenet [4] for file and content sharing, and SETI@home [11] for distributed computing have demonstrated the feasibility of this approach. Current P2P systems, however, have mainly focused on the exchange of objects like files and music clips, which are “small”. In future systems, however, the content *
This work was supported in part by the Ministry of Science and Technology of Spain under Contract TIC2002-04258-C03-01 and TIC2001-5193-E, and the European Union under Contract IST-2001-34030 CATNET.
H. Kosch, L. Böszörményi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 1265–1272, 2003. © Springer-Verlag Berlin Heidelberg 2003
1266
O. Ardaiz et al.
will be of any form, including audio, video, and large data sets. Peer-to-peer systems may be used to set up multi-cast services for large-scale global audiences, provide services for storing ultra large data sets, and to allow the execution of parallel applications requiring teraflops of processing power. To achieve an efficient performance of such systems, more intelligent decisions than we have in today´s systems are required, concerning particularly from where the content should be retrieved and on which path it should travel. In this paper, we propose the usage of the economic paradigm of the catallaxis [3] for the decentralized resource allocation in P2P networks. In the catallactic coordination model the decisions of the peers are based on economic principles, being aware that resources like bandwidth, processing power and storage are limited. Peers are in negotiation with each other to optimize their own benefits, and the decision of service provision takes into account their cost and benefits involved. Recent research in Grid computing has also recognized the value of price generation and negotiation, and in general the use of economic models for trading resources and services and for the regulation of supply and demand of resources in increasingly large-scale and complex Grid environment. Examples are the Nimrod/G Resource Broker and the GridBus project [1; 6]. In order to study the catallactic coordination in P2P networks, we use the CATNET application layer simulator [2], which builds on top of a TCP/IP network simulator the agents to form a peer-to-peer network. We evaluate the proposed system with several experiments and compare the achieved service provision with a baseline system. In the following section 2 we describe the motivation of our approach. In section 3 we explain the used simulator and the experimental framework. Section 4 contains the evaluation of the proposed system and the discussion of the results. In section 5 we conclude the paper.
2 Decentralized Economic Coordination with the Catallaxy Paradigm Application layer networks such as peer-to-peer networks are software architectures which allow the provision of services by connecting a large number of individual computers for information search, content download, parallel processing or data storage. In order to keep such a network operational, service control and resource allocation mechanisms are required. Performing such task with a centralized coordinator, however, has several difficulties: 1) A continuously updating mechanism would be needed to reflect the changes in the service demands and node connectivity; and 2) Long latencies for obtaining updates about the nodes at the edge of the network as the diameter of the network grows.
Exploring the Catallactic Coordination Approach for Peer-to-Peer Systems
1267
These drawbacks motivate the evaluation of a decentralized coordination concept, which is able to allocate services and resources without having a dedicated and centralized coordinator instance. The catallaxy coordination approach [3; 7] is a coordination mechanism for information systems consisting of autonomous network elements, which is based on constant negotiation and price signaling. Autonomous agents are able to adapt their heuristic strategies using machine learning mechanisms. This constant revision of prices leads to an evolution of the agent strategies, a stabilization of prices throughout the system and self-regulating coordination patterns [3]. The resulting patterns are comparable to those witnessed in human market negotiation experiments [10].
3 Experimental Framework 3.1 The Simulator for P2P Networks In order to evaluate the behavior of a P2P system with the catallactic coordination mechanism, we have used the CATNET network simulator [2]. CATNET is a simulator for an application layer network, which allows creating different types of agents to form a network. This simulator is implemented on top of the JavaSim network simulator [8]. JavaSim simulates a general TCP/IP network and provides substantial support for simulating real network topologies and application layer services, i.e. data and control messages among application network instances. We have implemented two main control mechanisms for the network coordination: the baseline and the catallactic control mechanism. The baseline mechanism computes the service/resource allocation decision in a centralized instance. In the catallactic mechanism, autonomous agents take their decisions in a decentralized way, having only local information about the environment. Each agent disposes of a strategy to take decisions, which targets to increase the agent´s own benefit. In the simulations, we consider a service as the functionality, which is exchanged among the peers in the network. The concept of service and the functions, or “personalities”, a peer can assume in the CATNET simulator, are the following:
•
• •
Service: a service encapsulates a general function performed in the P2P network. A service is the provision of a resource such as computing power, data storage, content, or bandwidth. The service provision includes the search for a resource and its reservation for availability. Client: A peer may act as a client or consumer of a service. As such it needs to access the service, use it for a defined time period, and then continues with its own program sequence. Resource: A peer, which is the owner of a required functionality. This functionality, for instance, may represent content, storage or processing power. The functionality, which is required by the clients or consuming peers, is encapsulated in a service.
1268
O. Ardaiz et al.
•
Service copy: A peer acting as a service copy offers a service as an intermediary, however it is not the owner of the components to provide the service. It must cooperate with the resource to be able to provide the service. Service copies offer the service to requesting clients.
3.2 Money and Message Flow In the simulator, the network activity is characterized by a continuous exchange of control messages and service provision. Different control messages in the two coordination mechanisms are used to accomplish the negotiation between peers. In Figure 1 we show the money and message flow used in the catallactic coordinated system. The requests from clients are broadcasted and forwarded to the service copies. Compared with the flooded requests model used in Gnutella [5], however, the numbers of hops a request can be forwarded in the simulator is limited. Service copies initiate negotiations with the resources they know to provide the service. Upon successful negotiations the service copies offer the service to the client. If the client accepts, then the service copy provides the service by means of a resource. Client
Resource
ServiceCopy
1: request_service
Client
Resource
ServiceCopy
Master SC
1: request_service 2: request_service
2: request_service 3: cfp
3: request_service
4: propose
4: accept
5: cfp
5: accept
6: propose 7: accept 8: accept
6: accept 7: transfer_money
9: transfer_money 10: transfer_money
Fig. 1. Money and message flows. Catallactic coordinated system.
8: transfer_money
Fig. 2. Money and message flows: Baseline approach.
In the centralized baseline system (Figure 2), the master service copy (MSC) receives the client requests with additional information through the resource/service copy pairs. Taking into account the distance and availability, it selects a resource/service copy pair and sends back an accept/reject message to the client. The resource allocates the required resource units and the service copy provides the service to the client.
Exploring the Catallactic Coordination Approach for Peer-to-Peer Systems
1269
4 Experimental Evaluation 4.1 Experimental Setup With experiments we wish to measure if a P2P network coordinated by the catallaxy paradigm is able to successfully provide service to requesting clients. A second goal is to compare qualitatively the obtained results with the centrally coordinated baseline system. In our experiments we explore as design space of the system the node density and node dynamics of the network (Figure 3). First, we simulate the P2P network with different densities of the service and resource providing agents in a high dynamics environment (Figure 1 experiments 1A-C). Then, we simulate the high node density network in environments with different values of the dynamics (Figure 3 experiments 2A-C). Catallactic Baseline
node density
Exp. 2A
Exp. 2B
Exp. 2C Exp. 1C
high
Exp. 1B
medium
Exp. 1A low
medium
high
node dynamics
Fig. 3. Design space of the system evaluated experimentally. Experiments 1A-C: Node density. Experiments 2A-C: Node dynamics.
In the simulations the input is a trace of client demands with requests for service. The service request specifies the amount of service, a price, and its duration. In all experiments the same demand trace is used. The physical network topology used in the experiments is organized in three levels of pentagons with leaves on the outer level, such as shown in Figure 4. Although other specific or random topologies of the nodes could be used as well, we applied this topology since it facilitates controlled experiments. On the top of the physical network an application layer network is build. On each node of the network, peers can be instantiated. Peers are instantiated having one of the previously described types of personalities, which can be a client, service copy or resource agent. Depending on the particular experiment, a node may contain several agents or no agent at all. In the second case, the node acts as a router.
1270
O. Ardaiz et al.
Fig. 4. Example of the network topology with approx. 100 nodes used in the experiments.
The relation of the experimental configuration to real world P2P systems is the following: A high value for the dynamics is interpreted to reflect the high level of connection and disconnection in P2P networks. A high level of node density represents the large number of machines with limited capability as found in P2P networks. In the simulations with high node density, we reduce the capacity of the resource agents, in order to represent small machines at the edge of the network. In the low node density scenario, on the other hand, the capacity of the service copy is increased such that the total amount of service available by the network is equal over all experiments. In Table 1, the configuration of the experiments is detailed. Table 1. Experiment description.
Input trace - 2000 service requests generated randomly by 75 clients over a time
Node topol. Node density
Node dynamics
interval of 100 s. - each request is for 2 service units. - each service has a duration of 5 s. - 106 physical nodes - 75 clients on the leaves of the physical network - different density of resource and service copy agents. Each Resource has one service copy associated. Exp 1A: low node density: 5 resources with capacity 60. Exp 1B: medium node density: 25 resources with capacity 12. Exp 1C: high node density: 75 resources with capacity 4. Dynamic behavior: On average 70% of the service copies are connected. Exp 2A: Service copies do not change its state (static network) Exp 2B: Each 200 ms every service copy can change its state (connected/disconnected) with a probability of 0.2. Exp 2C: Each 200 ms every service copy can change its state (connected/disconnected) with a probability of 0.4.
The main parameters we are interested to measure are the number of client requests the network is able to provide a service for in the different scenarios. The sce-
Exploring the Catallactic Coordination Approach for Peer-to-Peer Systems
1271
nario we are particularly interested in is the one with high node dynamics and high node density, as this configuration can be related to the conditions found in P2P networks.
4.2 Experimental Results In Figure 5 the service provision rate of a network with different node density in a highly dynamic environment is shown (experiments 1A – C). It can be observed that the network using catallactic coordination achieves a higher service provision rates than the baseline system with a smooth decrease for increasing node density. In Figure 6 the service provision rate of a network with high node density in different dynamic environment is shown (experiments 2A – C). It can be observed that the service provision rate of the catallactic system is rather independent to the dynamics. The baseline system, on the other hand, decreases with increasing dynamics. 100
CA BL
80
60
40
20
0
service provided [%]
service provided [%]
100
80
CA BL
60
40
20
0
low node density
medium node density
high node density
Fig. 5. Service provision in % in a highly dynamic network environment with different node density. CA = catallactic coordinated system. BL = baseline system.
low medium high dynamics dynamics dynamics
Fig. 6. Service provision in % in a high node density environment with different dynamics. CA = catallactic coordinated system. BL = baseline system.
Considering the achieved service provision rate, our experimental results indicate that service provision in networks with many small nodes in a highly dynamics environment could be coordinated successfully by the catallaxy paradigm. Exploring additional parameters of the system has the potential to provide more insight in the behavior of such a complex system. Currently, we examine the influence of other parameters on the performance tendencies discovered so far. One of the drawbacks we found of the catallactic approach is the time needed to establish a service provision, which is high due to the negotiation protocol carried out by agents. Other parameters we study is how scale affects the performance of the system.
1272
O. Ardaiz et al.
5 Conclusions We have first indicated the need for an intelligent decision mechanism for service provision in future P2P networks, which shall cost-consciously decide from where the content should be retrieved and on which path it should travel. Such mechanism should not only achieve the functionality, but also reduce the overall cost to provide this functionality. We have proposed the catallactic coordination as a decentralized economic approach for resource allocation in P2P networks. In this approach, the decisions of the peers are based on economic principles, being aware that resources like bandwidth, processing power and storage are limited. Autonomous peers negotiate with each other for service provision in order to optimize their own benefits. With simulations we have investigated if service provision in P2P networks can be achieved by the catallactic coordination approach. We compared the obtained results with a centralized baseline approach. We observed in the experiments that the service provision in the catallactic coordination is rather independent of the dynamics of the network. The service provision capability of the baseline approach appears to be sensitive to the dynamics, reducing its performance in highly dynamic environments.
References 1.
Buyya, R., D. Abramson, and J. Giddy, "A Case for Economy Grid Architecture for Service-Oriented Grid Computing". Proc. 10th IEEE International Heterogeneous Computing Workshop (HCW 2001). San Francisco, 2001. 2. CatNet project. “CATNET”. http://research.ac.upc.es/catnet 3. Eymann, T. and B. Padovan, "The Catallaxy as a new Paradigm for the Design of Information Systems". Proceedings of The World Computer Congress 2000 of the International Federation for Information Processing.2000. 4. Freenet. 2003. The Freenet home page. http://www.freenetproject.org 5. Gnutella. 2003. The Gnutella home page. http://www.gnutella.com 6. Grid Computing and Distributed Systems (GRIDS) Laboratory. "GRIDBUS Project". The University of Melbourne, Australia. http://www.gridbus.org/, 2002-11-28. 7. Hayek, F.A., W.W. Bartley, P.G. Klein, and B. Caldwell. The collected works of F.A. Hayek. University of Chicago Press, Chicago, 1989. 8. JavaSim Project. "JavaSim". Ohio State University EEng Dept. http://www.javasim.org/, 2 A.D.-11-29. 9. Napster. 2003. The Napster home page. http://opennap.sourceforge.net/ 10. Pruitt, D.G. Negotiation Behavior. Academic Press, New York, 1981. 11. SETI@HOME. http://setiathome.ssl.berkeley.edu/
Incentives for Combatting Freeriding on P2P Networks Sepandar D. Kamvar, Mario T. Schlosser, and Hector Garcia-Molina Stanford University
Abstract. We address the freerider problem on P2P networks. We first propose a specific participation metric, which we call a peer’s EigenTrust score. We show that EigenTrust scores accurately capture several different participation criteria. We then propose an incentive scheme that may be used in conjunction with any numerical participation metric. We show that, when these incentives are used in conjunction with EigenTrust scores, they reward participatory peers but don’t exclude less active peers.
1
Introduction
A notable problem with many of today’s P2P file-sharing networks is the abundance of freeriders on the network – peers who take advantage of the network without contributing to it. Up to 70% of Gnutella clients do not share any files, and nearly 50% of all responses are returned by 1% of the peers [1]. This abundance of freeriders, and the load imbalance it creates, punishes those peers who do actively contribute to the network by forcing them to overuse their resources (e.g. bandwidth). We address this problem by providing incentives for peers to make active contributions to the network. For example, active participators may get preference when they are competing for another peer’s resources, such as bandwidth. Our approach is simple: each peer gets a certain participation score, and it receives rewards based on its participation score. The challenges here lie in how to structure incentives that reward active participators without completely excluding peers that are less active. Previous work in this area has focused primarily on currency-based systems wherein peers gain currency for uploading files, and use currency when downloading files [2]. We take a different approach, rewarding peers with high participation scores with advanced services, such as faster download times or an increased view of the network. Our approach may be used in conjunction with currency-based approaches. In this work, we describe a scoring system that accurately quantifies participation, even in the presence of malicious peers trying to subvert the system, and we propose some incentives, and show empirically that these incentives benefit participatory peers in a fair manner.
2
EigenTrust
In the incentives that we propose in this paper, we assume the existence of some scoring system that measures the relative participation levels of peers in the system. One useful metric is a peer’s EigenTrust score [4]. EigenTrust was developed as a reputation metric H. Kosch, L. B¨osz¨orm´enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 1273–1279, 2003. c Springer-Verlag Berlin Heidelberg 2003
1274
S.D. Kamvar, M.T. Schlosser, and H. Garcia-Molina
for P2P systems. In this work, we show that EigenTrust is also a good measure of a peer’s relative participation level. To test how well a peer’s EigenTrust score reflects its participation level, we simulate a P2P network in the manner described in [3]. The simulator described in [3] is an eventdriven simulator that proceeds by query cycles. At each query cycle, peers submit and respond to queries according to certain distributions over peers’ interests and the files peers share, and download files from peers who respond to their queries. Freeriders and malicious peers sharing inauthentic files are modeled as well as active participatory peers. In Figure 1 we plot each peer in the network on a graph where the x-axis represents the EigenTrust score of the peer, and the y-axis represents the number of authentic uploads that the peer provides in a given timespan (15 query cycles). Notice that the EigenTrust score is correlated with the number of authentic uploads; those peers that provide many authentic uploads also have high EigenTrust score. The correlation coefficient is 0.97 indicating a close relationship between the number of authentic uploads and the EigenTrust score.
Fig. 1. Correlation between EigenTrust scores (y-axis) and the number of authentic uploads (xaxis). Each point represents a peer.
In order to give the reader further intuition on EigenTrust scores, we examine the following simple example users in our P2P simulation. These users, and their characteristics, are described below and summarized in Table 1. Angela is an active participator and shares many popular files across many content categories. Bob is an average user who shares a moderate number of files, many of them popular, some of them unpopular, from a couple of content categories. Corcoran is an occasional user who shares a few files, most of them popular. David is an eccentric user, sharing many obscure files that few people want. Ebeniezer is a freerider, who doesn’t share any files. And Forster is a malicious user, who shares many corrupt files. Based on these descriptions, Angela is the most active participator, followed by Bob, Corcoran, David, Ebeniezer, and Forster, in that order. Table 1 shows that EigenTrust
Incentives for Combatting Freeriding on P2P Networks
1275
captures this ranking. Notice that the EigenTrust scores of Ebeniezer the freerider and Forster the malicious peer are both 0. One may wish for a scoring system in which the malicious peer gets a lower score than a freerider. However, it should be noted that, due to the ease of entry in P2P networks, giving a malicious peer a score below that of a freerider is not very effective. A malicious user with a poor score may simply create a new peer and enter the network as a new freerider. Table 1. EigenTrust vs. other rankings for each of our seven sample users. Each of these users has the same uptime (20%), and the same number of connections (10).
Angela Bob Corcoran David Ebeniezer Forster
# Content Cat. 10 3 3 3 0 10
# Files Popularity ET Score ET Rank 1,000 Typical .02875 1 100 Typical .00462 2 50 Typical .00188 3 200 Unpopular .00115 4 0 0 5 (tie) 2,000 Malicious 0 5 (tie)
We also examine the following pairs of peers, where the first member of each pair is more participatory in a different way. Again, we use these examples to give the reader intuition on the behavior of EigenTrust scores. 1. The first scenario involves Tim and Jim. Tim and Jim share the exact same files, except Tim is always online while Jim turns his computer off at night. 2. The second scenario involves Stan and Jan. Stan and Jan share files in the same content categories. However, Stan shares twice as many files as Jan. 3. The third scenario involves Stephanie and Bethany. Stephanie and Bethany share five files each from the same content category. However, Stephanie shares five very popular files, and Bethany shares five unpopular files. 4. The final scenario involves Mario and Luigi. Mario and Luigi share the same number of files, but Mario has a diverse collection, sharing files in many content categories, while Luigi has narrow interests, and shares files from only one content category. Table 2 compares the EigenTrust scores of each of these pairs of peers, and shows that each of these characteristics we tested (uptime, number of shared files, popularity of shared files, and diversity of shared files) are reflected in the EigenTrust score of a peer. For example, Tim (who shares the same files as Jim, but has a greater uptime) has a greater number of authentic uploads, and a greater EigenTrust score, than Jim.
3
Incentives
Our goal is to provide incentives that reward participatory peers and punish freeriders and malicious peers. However, we do not want to punish freeriders so much that they are not able to download files and share them to become participators if they so choose. Two ways to reward participatory peers is to award them faster download times, and grant them a wider view of the network. In this section, we propose two score-based
1276
S.D. Kamvar, M.T. Schlosser, and H. Garcia-Molina Table 2. Comparing Pairs of Peers
Tim Jim Stan Jan Stephanie Bethany Mario Luigi
# Cont. Cat. 3 3 3 3 1 1 5 1
# Files Popularity # Links 1000 Typical 10 1000 Typical 10 500 Typical 10 1000 Typical 10 5 5 Most Pop. 10 5 5 Least Pop. 10 5000 Typical 10 5000 Typical 10
Uptime # Auth. Uploads ET Score 92% 965 0.045 35% 415 0.028 33% 325 0.011 33% 379 0.018 35% 15 0.00038 35% 0 0 25% 267 0.023 25% 81 0.0052
incentive schemes: a bandwidth incentive scheme and a TTL incentive scheme. Notice that these score-based schemes that we propose are completely general, and may be used with any scoring scheme that gives scores for which peers should be rewarded. Bandwidth. The first incentive that we propose is to give active participators preference when there is competition for bandwidth. More specifically, we propose that, if peer i and peer j are simultaneously downloading from another peer k, then the bandwidth of peer k is divided between peer i and peer j according to their participation scores. So if Tim and Jim are simultaneously downloading from some peer k, then Tim T will get STS+S ∗ 100% of peer k’s bandwidth (where ST represents Tim’s participation J J score, and SJ is Jim’s participation score), and Jim will get STS+S ∗ 100% of peer k’s J bandwidth. This also works when more than 2 peers are simultaneously downloading from the same peer. The protocol for this incentive is given in Algorithm 1.
Peer i with available bandwidth b, assigns bandwidth to each peer j downloading from it as follows: foreach peer j downloading from peer i do score(peer j) bandwidth(peer j) = score(peer j) b; j
end Algorithm 1: Bandwidth incentive
Notice that if a peer has a participation score of 0, it will get none of peer k’s bandwidth if it is competing against other peers for peer k’s bandwidth. However, this doesn’t exclude that peer from the network, since it is able to download from peers that are not servicing other peers. We show in Section 4 that freeriders are not excluded from the network when this incentive is implemented. TTL. Currently, Gnutella assigns each peer a time-to-live of 7 for each query. Our second incentive is to assign each peer a TTL based on its participation score, giving active participators a wider view of the network. There are many ways to do this, but one simple way would be to give each peer who has an above-average participation score a
Incentives for Combatting Freeriding on P2P Networks
1277
high TTL (for example, 10), and to give each peer who has a below-average participation score a low TTL (for example, 5). The protocol for this incentive is given in Algorithm 2.
if score(peer j) > mean score then T T L(peer j) = high-ttl; else T T L(peer j) = low-ttl; end Algorithm 2: TTL incentive
One problem here is that each peer must know the average participation score in the network, and explicitly computing this can be prohibitively costly in terms of message complexity, since it would require each peer to know the participation scores of every other peer in the network. However, if EigenTrust scores are used, this isn’t a problem. Since the EigenTrust scores in the network sum to 1, the average EigenTrust score for any network is 1/n, where n is the number of peers in the network. (We assume that a peer either knows or can approximate the number of peers in the network. If this is impractical for the given network, a peer can simply substitute its own EigenTrust score for 1/n.) Therefore, each peer can compare its own EigenTrust score to 1/n, and if its EigenTrust score is greater, the peer may issue a query with a TTL of 5. Otherwise, the peer may issue a query with a TTL of 3. A peer never needs to explicitly compute the average participation score in the network.
4
Experiments
We have shown in Section 2 that EigenTrust scores are a good measure of participation. Our task in this section is to show that the proposed incentives achieve their stated goals: to reward participatory peers with faster download times and a wider view of the network without completely excluding less active peers from the network. Again, we use our sample peers to give the reader intuition on how these incentives reward peers in the network. Bandwidth. The bandwidth incentive aims to reward peers with faster downloads. To test this, we again examine the sample peers Angela et. al., and measure their average download speeds in our simulations with and without the bandwidth incentive implemented. In Table 3, we show the average download speed for each of our sample users when the bandwidth incentive is implemented in conjunction with the EigenTrust scoring scheme, compared to the average download speed for each of our sample users when no incentive is implemented. To simulate a congested network, we employ the following congestion model: for each download that peer i begins to download from peer j, he competes for peer j’s bandwidth resources against somewhere between 0 to 4 other peers (chosen from a discrete uniform random distribution) who are also downloading files from peer j. Notice that, when the bandwidth incentive is implemented in this
1278
S.D. Kamvar, M.T. Schlosser, and H. Garcia-Molina
simulation in conjunction with EigenTrust scores, active participators are compensated for their participation, but not at too great an expense to the less active peers. Table 3. The average download speed (and percent bandwidth) for each peer using the bandwidth incentive in conjunction with the EigenTrust Score (left), and with no incentive (right) Bandwidth Incentive No Incentive ET Score % Bandwidth Download Speed % Bandwidth Download Speed Angela .02875 .6549 .4566 Bob .00462 .5307 .4566 Corcoran .00188 .5978 .4566 .00115 .4737 .4566 David Ebeniezer 0 .2 .4566 Forster 0 .2 .4566
TTL. Again, the important issue to investigate is whether this incentive compensates participators enough to be a useful incentive while giving nonparticipators enough resources so that they have the option of becoming participators if they so choose. To do this, we simulate 15 query cycles with the TTL incentive scheme activated as in Algorithm 2. Since this is a small network (100 peers), we define the default TTL to be 4 (peers can reach 75 other peers in this network on average with a TTL of 4). In this case, we define high-ttl to be 5, and low-ttl to be 3. For reporting our results, we split the peers into two groups: premium users (those peers with EigenTrust scores larger than the average EigenTrust score N1 ), and moderate users (those peers with EigenTrust scores less than the average EigenTrust score N1 ). Table 4 shows the number of premium users and moderate users and the average number of peers within their respective TTL range. Note that activating the TTL incentive scheme with these settings decreases the query load on the network while beefing up the service levels for premium users, which is a very desirable result: With the TTL scheme switched on, 77 ∗ 27 + 23 ∗ 98 = 4367 query messages will be generated throughout the network when all peers issue a query. With the TTL scheme switched off, 100 ∗ 75 = 7500 messages will be generated in the same process. Also, notice that even the freeriders will not be excluded from the network, as they receive a TTL of 3 for their queries. Table 4. The TTL, average number of peers reached (PR), and average number of responses per query (RPQ) for each peer using the TTL incentive in conjunction with the EigenTrust Score (left) and with no incentive (right). TTL Incentive No Incentive ET Score #Peers #Peers in TTL Range #Peers #Peers in TTL Range Moderate Users < N1 77 27 74 75 Premium Users > N1 23 98 26 75
Incentives for Combatting Freeriding on P2P Networks
5
1279
Conclusion
Two main results are presented in this paper: First, we show that a peer’s EigenTrust score is a good participation metric according to three natural participation criteria. Second, we present two incentive protocols based on these scores, and show that they reward participatory peers without completely excluding non-participatory peers.
References 1. E. Adar and B. Huberman. Free riding on gnutella. First Monday, 5(10), October 2000. 2. P. Golle, K. Leyton-Brown, I. Mironov, and M. Lillibridge. Incentives for sharing in peer-topeer networks. In WELCOM01, 2001. 3. S. Kamvar and M. Schlosser. Simulating a File-Sharing P2P Network. In SemPGRID03, 2003. 4. S. Kamvar, M. Schlosser, and H. Garcia-Molina. The EigenTrust Algorithm for Reputation Management in P2P Networks. In WWW 2003, 2003.
Topic 18 Demonstrations of Parallel and Distributed Computing Ron Perrott, Henk Sips, Jarek Nabrzyski, and Michael Kropfberger Topic Chairs
For many years Euro-Par has provided a forum for researchers from all over the world with the opportunity to present their research ideas and results to other colleagues; colleagues with similar interests as well as colleagues with related interests. This year for the first time Euro-par has enhanced this forum of international activities with a special session based solely on ‘Demonstrations’. The rationale is based on the premise that it is important not only to research new ideas but also to show their usage. Demonstrations are a means of showing the advantages of the underlying technologies within real-use environments and based on proof-of-concept applications. This new session is therefore designed as an incentive and catalyst for researchers to show the feasibility and necessity of their work. The focus of the reviewers was therefore not only on scientific contributions, but also on demonstrations which can illustrate proof of concepts and e.g. show the ease of use of certain new tools, or performance gains in comparison to commonly used algorithms or concepts. The number of contributions submitted to the “Demo Session” was 12, consisting of demonstrations from a wide range of areas relevant to Euro-Par. The reviewers of the “Demo Session” selected 6 of the submitted papers which hopefully will add to the portfolio of Euro-Par activities and will encourage more submissions in future years. Since this is the first time for this topic, the organisers would welcome feedback on the topic and suggestions for its promotion at future conferences. The reviewers are convinced of its necessity in the field and would like to thank all who submitted contributions.
H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, p. 1280, 2003. c Springer-Verlag Berlin Heidelberg 2003
Demonstration of P-GRADE Job-Mode for the Grid1 1
1
1
1
1
1
P. Kacsuk , R. Lovas , J. Kovács , F. Szalai , G. Gombás , N. Podhorszki , 2 2 3 4 4 Á. Horváth , A. Horányi , I. Szeberényi , T. Delaitre , G. Terstyánszky , and 4 A. Gourgoulis 1
Computer and Automation Research Institute (MTA SZTAKI) Hungarian Academy of Sciences, 1518 Budapest, P.O. Box 63, Hungary {kacsuk, rlovas, smith, szalai, gombasg, pnorbert}@sztaki.hu 2 Hungarian Meteorological Service, H-1525 Budapest, P. O. Box 38, Hungary {horvath.a, horanyi.a}@met.hu 3 Department of Control Engineering and Information Technology, Budapest University of Technology and Economics, H-1117 Budapest, XI., Pázmány Péter sétány 1/D, Hungary
[email protected] 4 Centre for Parallel Computing (CPC), Cavendish School of Computer Science, University of Westminster, 115 New Cavendish Street, London W1W 6UW, UK {delaitt, terstyg, agourg}@cpc.wmin.ac.uk
Abstract. The P-GRADE job execution mode will be demonstrated on a small Grid containing 3 clusters from Budapest and London. The first demonstration illustrates the Grid execution of a parallel meteorology application. The parallel program will be on-line monitored remotely in the Grid and locally visualized on the submitting machine. The second demonstration will use a parallel traffic simulation program developed in P-GRADE to show the usage of the PGRADE job mode for Grid execution. The parallel program will be checkpointed and migrated to another cluster of the Grid. On-line job and execution monitoring will be demonstrated.
1 Introduction MTA SZTAKI has developed a parallel program development environment (PGRADE) [1] for hiding the low-level PVM and MPI APIs from the users and providing a much higher-level graphical programming model with full support for parallel program debugging, monitoring, performance visualization, mapping, loadbalancing, etc. P-GRADE is now extended towards the Grid and it stands for Parallel Grid Run-time and Application Development Environment. This environment enables the Grid application programmer to develop a parallel program that can be executed as a Grid job on any parallel site (like a supercomputer, or a cluster) of a Grid in a transparent way.
1 This work was partially supported by the following grants: EU DataGrid IST-2000-25182, EU GridLab IST-2001-32133, Hungarian Scientific Research Fund (OTKA) no. T032226, Hungarian OMFB-02307/2000, and Hungarian IKTA-00075/2001. H. Kosch, L. Böszörményi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 1281–1286, 2003. © Springer-Verlag Berlin Heidelberg 2003
1282
P. Kacsuk et al.
2 Current Version of P-GRADE P-GRADE supports the interactive development of a parallel program as well as their job execution mode. The interactive execution can be on a single processor system, on a supercomputer or on a cluster. The recommendation is that the editing, compiling and debugging activities should be on a single processor system while mapping, and performance monitoring should take place on parallel systems like supercomputers or clusters. If the program is correct and delivers the expected performance on parallel systems, the user can switch to the job mode. Here the user should specify the resource requirements, input files, output files and error file of the job. Then P-GRADE automatically generates the appropriate job from the parallel program developed in the interactive working mode. Currently two job types can be generated: • Condor job • PERL-GRID job
3 Condor Job Mode of P-GRADE By integrating P-GRADE and Condor our goal was to provide a Grid-enabled runtime system for P-GRADE. In Condor mode, P-GRADE automatically constructs the necessary job description file containing the resource requirements of the parallel job. The mapping function of P-GRADE was changed according to the Condor needs. In Condor the user can define machine classes from which Condor can reserve as many machines as it is defined by the user in the job description file. When the user generates Condor job under P-GRADE this is the only Condor-related task. P-GRADE supports this activity by offering a default set of machine classes for the user.
Design/ Edit
Compile
Condor Map
Attach
P-GRADE batch mode
Detach
Submit job
Fig. 1. P-GRADE services in job mode
Finally, after submitting the Condor job the user can detach P-GRADE from the job. It means that the job does not need the supervision of P-GRADE when it is executed in the Grid (for that purpose we use Condor). Meanwhile the P-GRADE generated
Demonstration of P-GRADE Job-Mode for the Grid
1283
Condor job is running in the Grid, P-GRADE can be turned off or it can be used for developing other parallel applications. However, at any time the user can attach again P-GRADE to the job to watch the current status and results of the job. The program development and execution mechanism of P-GRADE for the Grid is shown in Fig. 1.
4 PERL-GRID Job Mode of P-GRADE The Condor job mode can be used only if the submit machine is part of a Condor pool. However, such a restriction is too strong in the Grid where there are many Condor pools as potential resources and these should be accessed from the submit machine. This is the case, for example, in the Hungarian ClusterGrid where the Grid itself is a collection of Condor pools (clusters) but the user’s submit machine is typically not part of the ClusterGrid. In such situation some new functionalities should be provided for the user. In order to provide these missing functionalities we have created a thin layer, called PERLGRID, between P-GRADE and Condor with the following tasks: • The selection of the high-performance computing site (Condor pool) is the task of a Grid resource broker. Currently a random selection algorithm is applied by PERL-GRID. (This will be replaced by a Grid resource broker in the future.) • PERL-GRID takes care of contacting the selected site and executing the mutual authentication based on the SSH technology. • PERL-GRID also takes care of file staging by transferring the input files and the code file into a temporary directory at the reserved Grid site. Finally, it passes the job to the local job manager. Currently it is Condor but soon others, like SGE will be supported. • If the selected Grid site becomes overloaded, the whole parallel application is check-pointed and migrated to another Grid site by PERL-GRID. An automatic parallel check-point mechanism has been elaborated for parallel programs developed under P-GRADE. By this check-point mechanism, all those features of Condor that are provided for sequential jobs in the Standard Universe (job migration, fault-tolerance, guaranteed execution) are supported even for parallel jobs under P-GRADE. Running a parallel job in the Grid requires the on-line monitoring and steering possibility of that job. Using the Grid resource and job monitoring techniques developed in the GridLab project we can on-line monitor and visualize the status of P-GRADE jobs. More than that, the processes of a parallel application can be monitored by integrating the GRM monitor of P-GRADE either with the GridLab Grid monitor or with the R-GMA monitoring infrastructure of the DataGrid project. In either case using PROVE of P-GRADE enables the on-line visualization of the process interactions of a parallel application running anywhere in the Grid.
1284
P. Kacsuk et al.
5 Description of the Demonstration All the features of the PERL-GRID job mode of P-GRADE described in the previous section will be demonstrated by two scenarios. During the demonstration we will use a demonstration Grid consisting of 3 clusters: SZTAKI cluster in Budapest, cluster of the Budapest University of Technology and Economics, cluster of the University of Westminster in London. 5.1 Scenario 1 Scenario 1 will demonstrate the Grid execution of the MEANDER nowcast program package of the Hungarian Meteorology Service. The goal of the MEANDER package is to provide ultra-short range (up to 1 hour) weather forecast of dangerous situations like storm and fog on a high resolution regular mesh (10km -> 1km). To achieve this goal members of OMSZ and SZTAKI have jointly parallelised the six most time consuming calculation module of MEANDER by P-GRADE [2].
Fig. 2. The P-GRADE demonstration version of the MEANDER program
We will use a simplified version of MEANDER containing only 4 algorithms for the demonstration. The P-GRADE view of this MEANDER version is shown in Fig. 2. It is clear on Fig. 2 that the four algorithms are realized by the processor farm concept. The numbers in the clouds represent the number of processors to be used in the dem-
Demonstration of P-GRADE Job-Mode for the Grid
1285
onstration. It can also be seen that these algorithms are connected like a workflow. First the delta algorithm should be executed, then in the second phase, the other three algorithms can be executed in parallel. At that time 40 processors will be used in parallel. Finally, a visualization process is applied to draw the weather map shown in the right bottom of Fig. 2. The parallel job will be generated by P-GRADE and passed to PERL-GRID which will transfer the executable code to the 58-processor Linux cluster of SZTAKI (shown in the right top of the picture) to execute the job. Then PERL-GRID collects the necessary meteorology database input file from the Hungarian Meteorology Service and passes the job with all the files to Condor at the SZTAKI cluster. Condor takes care of the parallel execution of the job at SZTAKI. When monitoring is requested, PERLGRID delivers the local monitor code, too and the collected trace file is sent back and visualized by PROVE on the submit machine at Klagenfurt whenever we request a trace collection by PROVE. The final trace file visualization picture is shown in Fig. 3.
Fig. 3. Process space-time diagram of the MEANDER program
The picture clearly shows that first the delta algorithm is running on 25 processors and when it is finished, it triggers the execution of the other three algorithms that are executed simultaneously. When the whole job is finished, PERL-GRID takes care of
1286
P. Kacsuk et al.
transferring the result file back to Klagenfurt and removing the temporary directory it created for the job at the SZTAKI cluster. Finally, the weather-forecast map of Hungary will be displayed at Klagenfurt. 5.2 Scenario 2 Scenario 2 will demonstrate the Grid execution of an urban traffic simulation program developed at the University of Westminster [3]. The program will be started as a PERL-GRID job under P-GRADE. PERL-GRID will select one of the three clusters of the demonstration Grid and transfers the necessary files to the selected cluster where it passes the job to Condor. When the program starts we will increase the load of the selected cluster by starting other higher priority jobs there. The application will be check-pointed and PERL-GRID will select another cluster and migrates the job there and passes it to the local Condor job manager. Using the GridLab Grid monitor system we will on-line monitor and visualize the job status as well as its processes and their interactions by PROVE.
6 Conclusions P-GRADE provides a high-level graphical environment to develop parallel applications both for parallel systems and the Grid. One of the main advantages of PGRADE is that the user has not to learn the different APIs for parallel systems and the Grid, simply by using the same environment will result in a parallel application transparently applicable either for supercomputers, clusters or the Grid. The current version of P-GRADE supports the interactive execution of parallel programs as well as the creation of a Condor or PERL-GRID job to execute the parallel program in the Grid. Remote monitoring and performance visualization of Grid applications are also supported by P-GRADE. P-GRADE will be used in the Hungarian ClusterGrid that connects the Condor pools of the Hungarian higher educational institutions into a high-performance, highthroughput Grid system. Though the Grid execution mode of P-GRADE is currently strongly connected to Condor, the use of PERL-GRID enables the easy connection to other local job managers like SGE and PBS. This work is planned as part of the Hungarian Grid activities.
References 1. P. Kacsuk: Visual Parallel Programming on SGI Machines, Invited paper, Proc. of the SGI Users’ Conference, Krakow, Poland, pp. 37–56, 2000 2. R. Lovas, et al: Application of P-GRADE Development Environment in Meteorology, Proc. of DAPSYS'2002, Linz, pp. 30–37, 2002 3. Gourgoulis, et al: Using Clusters for Traffic Simulation, Proc. of Mipro'2003, Opatija, 2003
Coupling Parallel Simulation and Multi-display Visualization on a PC Cluster J´er´emie Allard, Bruno Raffin, and Florence Zara Laboratoire Informatique et Distribution Projet APACHE ID-IMAG CNRS – INPG – INRIA - UJF 38330 Montbonnot, France Abstract. Recent developments make it possible for PC clusters to drive multi-display visualization environments. This paper shows how we coupled parallel codes with distributed 3D graphics rendering, to enable interactions with complex simulations in multi-display environments.
1
Introduction
Visualization appears as an efficient way to analyze results of complex simulations. Immersive environments, like CAVEs [1], enhance the visualization experience. They provide a high resolution and large surface display created by assembling multiple video projectors. Interactivity and stereoscopic visualization further improve the possibility of these workspaces. These environments are classically powered by dedicated graphics supercomputers, like SGI Onyx machines. Today, the anatomy of supercomputing is quickly and deeply changing. Clusters of commodity components are becoming the leading choice architecture. They are scalable and modular with a high performance/price ratio. Clusters have proved efficient for classical (non interactive) intensive computations. Recently the availability of low cost high performance graphics cards have foster researches to use these architectures to drive immersive environments [2,3]. The first goal was to harness the power of multiple graphics cards distributed on different PCs. But the scalability and performance of PC clusters allow to go beyond only distributed graphics rendering. While some cluster nodes have graphics cards to power the immersive environment, complex simulations can take advantage of extra nodes to decrease their execution time and reach an update rate suitable for interactivity. The demo we propose is based on two interactive applications running on a PC cluster driving a multi-display environment: a cloth simulation [4] and a fluid simulation [5]. Both applications involve a parallel simulation coupled with a distributed graphics rendering.
2
Softwares and Environments
The applications were developed on the following open source softwares: H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 1287–1290, 2003. c Springer-Verlag Berlin Heidelberg 2003
1288
J. Allard, B. Raffin, and F. Zara
– CLIC: Clic [6] is a Linux distribution dedicated to PC clusters. It includes several pre-configured free software tools to ease cluster installation, administration and monitoring. – Net Juggler: Net Juggler [3] enables an application to use the power of multiple graphics cards distributed on different PCs. It parallelizes graphics rendering computations for multi-display environments by replicating the application on each node and using MPI to ensure copies are coherent and displayed images are synchronized. Net Juggler is based on VR Juggler [7], a platform for virtual reality applications. – Athapascan: Athapascan [8,9] is a parallel programming language designed to develop portable applications and to enable efficient executions on PC clusters. An Athapascan program consists of a description of parallel tasks communicating through a shared memory. At runtime, Athapascan builds a data flow graph from the data dependencies between tasks. Based on this graph and on a cluster description, it controls task scheduling and data distribution.
3
Coupling Parallel Simulation with Parallel Rendering
We present two applications, a fluid and a cloth simulation. Both consist of a parallel simulation coupled with a distributed 3D graphics rendering. They use two different paradigms to parallelize the simulation part: the fluid simulation uses MPI while the cloth simulation uses Athapascan. 3.1
First Case Study: Fluid Simulation
The fluid simulation is based on the implementation of the Navier-Stokes equation solver proposed by Stam [10]. The space is discretized on a grid of cells, where the fluid can flow. Each cell holds a fluid velocity vector and a scalar fluid density characterizing the fluid present in a given cell. At each time step, the solver updates these data. These computations require simple matrix computations, a conjugate gradient and a Poisson solver. The solver is implemented with PETSC [11], a mathematical library distributing matrix-based operations on the cluster using MPI. As Net Juggler already sets up an MPI environment, it is easy to integrate PETSC in Net Juggler. More details can be found in [5]. The simulation involves 2 fluids flowing on a 2D grid (Fig. 1). The user can interactively disturb the fluid flow with a pointer that applies a force on the fluids. He can also add or remove obstacles. The simulation and the visualization are executed synchronously. With the solver executed on one node, the obtained frame rate is about 8 frames per second (fps). When the solver is parallelized on four nodes, the frame rate is close to 20 fps.
Coupling Parallel Simulation and Multi-display Visualization
3.2
1289
Second Case Study: Cloth Simulation
This application uses two different parallelization paradigms. The simulation is parallelized with the parallel programming environment Athapascan [9]. Rendering computations are distributed with Net Juggler [3] on graphics nodes. The cloth is modeled as a triangular mesh of particles linked up by springs. The object is block partitioned using Athapascan tasks [4]. At each time step, a particle position is computed based on two steps of a Newton’s equation integration. Data mapping, task scheduling and communications are managed at runtime by Athapascan. At the end of each time step, computed positions are sent in a socket to Net Juggler for graphics rendering (Fig. 2).
Fig. 1. Interactive simulation of two 2D fluids.
4
Fig. 2. Piece of cloth of 100 particles with two or one corner(s) fastened.
Demo Setup
The demo steup consists of a 6 nodes cluster using a fast Ethernet network and driving 3 video-projectors. We will alternatively show the fluid simulation demo and the cloth simulation demo. The attendees will be able to evaluate by them-self the quality and performance of commodity graphics cards, the synchronization level provided by Net Juggler to create a seamless 3 projector display, the interactivity level reached on complex simulations when parallelized. These demo will also give attendees the opportunity to have some insights about the CLIC distribution. Short videos of the demos are available at http://netjuggler.sourceforge.net/Gallery.php.
5
Conclusion
Interactive simulation execution coupled with advanced visualization environments can greatly help to analyze and understand complex data. We showed
1290
J. Allard, B. Raffin, and F. Zara
in this paper how PC clusters can be used to execute such applications. The fluid and cloth simulations take advantage of two levels of parallelism to enable the interactive execution of complex simulations in a multi-display environment. The first one distributes graphics rendering using Net Juggler, and the second one parallelizes the core of the simulation using MPI or Athapascan. Future works will focus on developing solutions for automatic coupling, targeting executions on large clusters and grid computing infrastructures. Interactive processing and visualization of large data sets will also be considered.
References 1. Cruz-Neira, C., Sandin, D.J., DeFanti, T.A., Kenyon, R.V., Hart, J.C.: The Cave Audio VIsual Experience Automatic Virtual Environement. Communication of the ACM 35 (1992) 64–72 2. Samanta, R., Funkhouser, T., Li, K., Singh, J.P.: Hybrid Sort-First and Sort-Last Parallel Rendering with a Cluster of PCs. In: SIGGRAPH/Eurographics Workshop on Graphics Hardware. (2000) 3. Allard, J., Gouranton, V., Lecointre, L., Melin, E., Raffin, B.: Net Juggler: Running VR Juggler with Multiple Displays on a Commodity Component Cluster. In: IEEE VR, Orlando, USA (2002) 275–276 4. Zara, F., Faure, F., Vincent, J.M.: Physical cloth simulation on a PC cluster. In D. Bartz, X.P., Reinhard, E., eds.: Fourth Eurographics Workshop on Parallel Graphics and Visualization 2002, Blaubeuren, Germany (2002) 5. Allard, J., Gouranton, V., Melin, E., Raffin, B.: Parallelizing pre-rendering computations on a net juggler PC cluster. In: Immersive Projection Technology Symposium, Orlando, USA (2002) 6. MandrakeSoft, ID, L., Bull: (The clic linux cluster distribution) http://clic.mandrakesoft.com. 7. Bierbaum, A., Just, C., Hartling, P., Meinert, K., Baker, A., Cruz-Neira, C.: VR Juggler: A Virtual Platform for Virtual Reality Application Development. In: IEEE VR 2001, Yokohama, Japan (2001) 8. Roch, J.L., et al.: Athapascan: Api for asynchronous parallel programming. Technical report, INRIA Rhˆ one-Alpes, projet APACHE (2003) 9. Galil´ee, F., Roch, J.L., Cavalheiro, G., Doreille, M.: Athapascan-1: On-line building data flow graph in a parallel language. In IEEE, ed.: Pact’98, Paris, France (1998) 10. Stam, J.: Stable Fluids. In: SIGGRAPH 99 Conference Proceedings. (1999) 121– 128 11. Balay, S., Gropp, W.D., McInnes, L.C., Smith, B.F.: PETSc 2.0 Users Manual. Technical Report ANL-95/11 - Revision 2.0.29, Argonne National Laboratory (2000)
Kerrighed: A Single System Image Cluster Operating System for High Performance Computing Christine Morin1 , Renaud Lottiaux1 , Geoffroy Vall´ee2 , Pascal Gallard1 , Ga¨el Utard1 , R. Badrinath1,3 , and Louis Rilling4 1
IRISA/INRIA – PARIS project-team 2 EDF 3 IIT Kharagpur 4 ENS-Cachan, antenne de Bretagne
Abstract. Kerrighed is a single system image operating system for clusters. Kerrighed aims at combining high performance, high availability and ease of use and programming. Kerrighed implements a set of global resource management services that aim at making resource distribution transparent to the applications, at managing resource sharing in and between applications and at taking benefit of the whole cluster resources for demanding applications. Kerrighed is implemented as a set of modules extending the Linux kernel. Legacy multi-threaded applications and message-passing based applications developed for an SMP PC running Linux can be executed without re-compilation on a Kerrighed cluster. The proposed demonstration presents a prototype of Kerrighed running on a cluster of four portable PCs. It shows the main features of Kerrighed in global memory, process and stream management by running multi-threaded and MPI applications on top of Kerrighed.
1
Topics
We propose to demonstrate a prototype of Kerrighed1 , a single system image operating system for clusters. The research work presented in this prototype has been carried out in the PARIS project-team at IRISA/INRIA and relates to several topics: Topic 01: Support Tools and Environments. A single system image operating system like Kerrighed can be considered as a tool or an environment to conveniently and efficiently execute parallel applications on clusters. Topic 03: Scheduling and Load Balancing. Kerrighed’s prototype implements a configurable global scheduler to balance the load on cluster nodes. Kerrighed’s global scheduler is based on novel efficient process management mechanisms. 1
Kerrighed (previously named Gobelins) has been filed as a community trademark.
H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 1291–1294, 2003. c Springer-Verlag Berlin Heidelberg 2003
1292
C. Morin et al.
Topic 09: Distributed Algorithms. The prototype of Kerrighed implements the container concept for efficient global memory management. Based on containers, Kerrighed provides shared virtual memory segments to threads executing on different cluster nodes, a cooperative file cache and remote memory paging. Topic 14: Routing and Communication in Interconnection Networks. Kerrighed’s prototype implements a portable high performance reliable communication system providing a kernel level interface for the implementation of Kerrighed’s distributed system services. The standard communication interface used by communicating Linux processes (pipe, sockets, etc. . . ) is also available in Kerrighed on top of this communication system. In Kerrighed, communicating processes can be transparently migrated and can still efficiently communicate with other processes after migration.
2
Originality of the Demonstrated Prototype
The main originality of the prototype we propose to present is that it is a single system image operating system for clusters. Kerrighed provides the same interface as the standard operating system running on each cluster node. The current prototype has been implemented as a set of Linux modules and a small patch to the kernel (less than 200 lines of code, mainly for exporting kernel functions). Hence, existing applications that have been developed for an SMP PC running Linux can be executed on a cluster without even being recompiled. Unix applications can be easily ported on Kerrighed by recompiling them. So, sequential processes requiring huge processing and/or memory resources, multi-threaded applications and parallel applications based on message passing can be easily and efficiently executed on a cluster running Kerrighed. Unlike other systems Kerrighed supports efficiently both the message-passing and the shared memory programming models on clusters. There are very few other research projects working on the design and implementation of a single system image operating system for clusters. Mosix [3] and Genesis [1] are examples of operating systems targeting the single system image properties. Mosix offers processor load balancing on top of a cluster. However, it does not support memory sharing between threads or processes executing on different cluster nodes unlike Kerrighed. A process which has migrated in Mosix cannot communicate efficiently with other processes after migration as messages are forwarded to it by the node on which it was created. Moreover, processes in Mosix cannot take benefit of the local file cache of their current execution node after migration, leading to poor performance for file accesses. Kerrighed has none of these drawbacks as it preserves direct communications between processes even after migration and as it implements a cooperative file cache. Genesis is a single system image operating system for clusters which, in contrast to Kerrighed, has been developed from scratch and is based on the microkernel technology. As Kerrighed it supports both the shared memory and the message-passing programming paradigms. However, it implements a distributed
Kerrighed: A Single System Image Cluster Operating System
1293
shared memory system which does not provide a standard interface. Thus, legacy shared memory parallel applications (such as Posix multi-threaded applications) cannot be executed on Genesis without a substantial porting effort. To our knowledge, Genesis only supports PVM for message-passing parallel applications.
3
Mechanisms for Demonstrating the Prototype
We will show a cluster of five portable PCs interconnected by a Fast Ethernet network. Four nodes of the cluster will run Linux and the Kerrighed modules. On this cluster, we will show the execution of instances of two parallel applications, one being a multi-threaded application, Volrend, the other one being an MPI application. Volrend is a multi-threaded application which implements ray-tracing to display 3D images. The fifth node executes a standard Linux system and is used to display the applications’ results and to show some performance values (processor load, memory occupation and network throughtput for instance) as well as the location of all applications’ threads and processes on the cluster’s nodes (graphical interface with different colors for the different applications). Kerrighed’s features that are demonstrated are the container concept for memory sharing between threads (Volrend), the process migration mechanism which allows to migrate any process or thread even if it communicates with other threads or processes by shared memory (Volrend) or by message-passing (Mandelbrot fractal), the configurable global scheduler in which the scheduling policy can be changed without stopping Kerrighed and the applications currently running on top of it. Note that a demonstration of an initial prototype of Kerrighed (not including process management and global scheduling features) has been successfully demonstrated at Supercomputing in Baltimore (USA) in November 2002 and at Linux Expo in Paris in February 2003 on the same cluster of portable PCs. The demonstration is stand-alone and does not require an Internet access.
4
Scientific Content
Our goal in designing Kerrighed is to combine high performance, high availability and ease of use [5]. Kerrighed performs global resource management. With a set of global resource management services, Kerrighed aims to make resource distribution transparent to the applications, to manage resource sharing in and between applications and to take benefit of the whole cluster resources for demanding applications. These services are distributed and each of them is in charge of the global management of a particular resource (memory, processor, disk). Global memory management in Kerrighed is based on the container concept [4]. In modern operating systems, block devices and memory are managed on page granularity. A container is a software object allowing to store and share pages cluster wide. Containers are managed in a software layer in between high level services of the node’s standard operating system (virtual memory, virtual
1294
C. Morin et al.
file system) and its device managers. Thus, the standard user interface is kept while Kerrighed offers virtual shared memory segments, a cooperative file cache and remote paging, all based on containers. Kerrighed’s global process management service consists in a global scheduler based on efficient process state management mechanisms. As characteristics of the workloads executed on clusters may differ from one environment to another, Kerrighed’s global scheduler has been designed to allow the specialization of the scheduling policy [6]. Changing the global scheduling policy can be done dynamically without rebooting the cluster nodes. The description of the scheduler to be used in a particular environment need only be described using XML configuration files. Kerrighed implements a basic primitive to extract the state of a process from the kernel. This primitive is exploited to perform process duplication, migration and check-pointing [7,2]. Processes exchanging messages can also be efficiently migrated as Kerrighed implements migrable sockets that ensure a direct communication between two communicating processes even after migration of one of them. These sockets are implemented on top of a dynamic stream service that allows processes to attach or detach from a stream. Kerrighed is an open source software distributed under the GNU GPL license. Scientific publications and a first (demonstration) version can be downloaded at http://www.kerrighed.org.
References 1. M.J. Hobbs A.M. Goscinski and J. Silock. Genesis : The operating system managing parallelism and providing single system image on cluster. Technical Report TR C00/03, School of Computing and Mathematics, Deakin University, February 2000. 2. Ramamurthy Badrinath and Christine Morin. Common mechanisms for supporting fault tolerance in DSM and message passing systems. Rapport de recherche 4613, INRIA, November 2002. 3. Amnon Barak, Shai Guday, and Richard G. Wheeler. The MOSIX Distributed Operating System, volume 672 of Lecture Notes in Computer Science. Springer, 1993. 4. Renaud Lottiaux and Christine Morin. Containers: A sound basis for a true single system image. In Proceeding of IEEE International Symposium on Cluster Computing and the Grid (CCGrid ’01), pages 66–73, Brisbane, Australia, May 2001. 5. Christine Morin, Pascal Gallard, Renaud Lottiaux, and Geoffroy Vall´ee. Towards an efficient single single system image cluster operating system. In ICA3PP, 2002. 6. Geoffroy Vall´ee, Christine Morin, Jean-Yves Berthou, and Louis Rilling. A new approach to configurable dynamic scheduling in clusters based on single system image technologies. In International Parallel and Distributed Processing Symposium, April 2003. 7. Geoffroy Vall´ee, Christine Morin, Jean-Yves Berthou, Ivan Dutka Malen, and Renaud Lottiaux. Process migration based on Gobelins distributed shared memory. In Proc. of the workshop on Distributed Shared Memory (DSM’02) in CCGRID 2002, pages 325–330, Berlin, Allemagne, May 2002. IEEE Computer Society.
ASSIST Demo: A High Level, High Performance, Portable, Structured Parallel Programming Environment at Work M. Aldinucci, S. Campa, P. Ciullo, M. Coppola, M. Danelutto, P. Pesciullesi, R. Ravazzolo, M. Torquati, M. Vanneschi, and C. Zoccolo Dept. of Computer Science – University of Pisa – Viale Buonarroti 2, 56127 Pisa
Abstract. This work summarizes the possibilities offered by parallel programming environment ASSIST by outlining some of the features that will be demonstrated at the conference demo session. We’ll substantially show how this environment can be deployed on a Linux workstation network/cluster, how applications can be compiled and run using ASSIST and eventually, we’ll discuss some ASSIST scalability and performance features. We’ll also outline how the ASSIST environment can be used to target GRID architectures. Keywords. Structured parallel programming, skeletons, coordination languages.
1
Demo Background
ASSIST (A Software development System based on Integrated Skeleton Technology) is a parallel programming environment based on skeleton and coordination language technology [8,9,3,2]. ASSIST provides the user/programmers with a structured parallel programming language (ASSISTcl), an integrated set of compiling tools (astCC) and a portable run time (the actual runtime CLAM, and the loader/runner assistrun). ASSIST is based on both skeleton and coordination languages technology, and comes after some other different experiences of our group related to skeleton based parallel programming [5,4]. It builds on the experience gained in these projects. The main goals in the design of ASSIST have been: high level programmability, rapid prototyping and suitability for complex multidisciplinary applications; functional and performance portability across a range of different target architectures; software reuse and interoperability. These goals have been achieved by taking a number of design choices and using several different implementation techniques.
This work has been partially supported by the National Research Council Coordinated Project CNRC0014B3 “Development environment for multiplatform and multilanguage high-performance applications, based upon the objects model and structured parallel programming (Agenzia2000)” and ASI-PQE2000 “Earth observation application development with High Performance Computing Tools” projects
H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 1295–1300, 2003. c Springer-Verlag Berlin Heidelberg 2003
1296
M. Aldinucci et al.
Programmability: the coordination language allows programmers to express both standard skeleton/coordination patterns as well as more complex parallelism exploitation patterns. This because a new1 , general purpose, highly configurable parallelism exploitation pattern has been included in the language (the parmod one) and because completely general graphs of parmods can be used to describe the parallel structure of applications. Furthermore, ASSISTcl allows a controlled usage of external objects (e.g. existing, possibly parallel, libraries) after programmer requests Performance: ASSIST environment design is highly layered. The source code is first translated into an intermediate “task code”. The task code, in turn, is compiled to an abstract machine built on top of ACE (Adaptive Coordination Environment [1]) and AssistLib. ACE provides communication and process framework. AssistLib is a C++ library implementing specific mechanisms needed to implement task code on top of ACE. The whole compiler design is based on OO design pattern technology, thus allowing easily replacement of compiler parts as well as introduction of new features without affecting the ASSIST support overall design. In addition, the whole compile process and both the task code and the AssistLib level have been carefully optimized in order to avoid any kind of bottlenecks as well as of overhead sources. Interoperability and software reuse: all the sequential portions of code needed to instantiate parmod parallelism exploitation patterns within an ASSISTcl program can be written in any of the C, C++ or F77 languages, and the details a programmer has usually to deal with when using such a programming language mix are handled by the ASSISTcl compiling tools (and further languages, such as Java, are being taken into account). Furthermore, sequential portions of code can invoke external CORBA object services, and a whole ASSISTcl application could be automatically wrapped into a CORBA object (IDL code generation is automatic), in such a way that its “parallel services” could be invoked from outside the ASSIST world. Existing, highly optimized, parallel scientific libraries can be integrated into ASSISTcl applications in such a way that they look like “normal” ASSISTcl parmods [7] Our group is currently working to ASSIST enhancement and evolution within several National research projects. Within a couple of National Research Council Strategy projects the ASSIST environment is currently being migrated on GRIDs [6]. Further ASSIST environment enhancements are planned within another, three year, large, National Research Council Project (GRID.it). The interested reader can refer to different papers describing ASSIST: [8,9,2,7,6,3].
2
ASSIST Framework Setup
ASSIST currently runs on cluster or networks of workstations. In order to install ASSIST the user needs to install some separate, public domain software packages. In particular, ACE, the Adaptive Communication Environment is needed, as well 1
with respect to previously developed skeleton based programming environments
ASSIST Demo
1297
as DVSA, the Distributed Virtual Shared Areas library. The former providing the basic functions of ASSIST abstract machine, the latter providing the support for data sharing across network connected processing elements. Both libraries assume a POSIX TCP/IP framework. ACE runs on both Linux and Windows or Mac (OS/X) boxes. DVSA originally runs on top of Linux, and is currently begin tested on the other environments by our research group. Once these libraries have been configured, the installation of ASSIST is a matter of a couple of make commands. The correct installation of the ASSIST package provides user with the astCC and assistrun commands, i.e. the compiler and run commands, respectively. Compiling ASSIST programs. Once an ASSIST program has been produced, using any available editor, it can be compiled using the astCC command. The compiler basically produces a set of C++ “object code” files, a kind of configuration file storing an XML representation of the resources needed to run the program, the makefiles needed to compile actual object code out of these sources, and eventually executes a sort of make all completing the generation of actual object code. Several parameters of the astCC command allow, for instance, the source files to be kept after the production of object code, in such a way the programmer may intervene directly at the “task code” or AssistLib level, i.e. at the level of the intermediate abstract machine, if needed. Running ASSIST programs. In order to run an ASSIST executable, the user must basically perform two steps: first, a CLAM (Coordination Language Abstract Machine, the one built out of ACE and AssistLib, basically) instance must be run on the processing elements of the target architecture. This can be done running by hand the CLAM process onto every node of the target architecture. ASSIST provides scripts that make this process automatic, once the (IP) names of the nodes are known. This step can be performed once and forall, as CLAM is an execution server and it can be invoked multiple times, with different object codes. Every time, CLAM loads object code, configuration info and provides to actually run the ASSIST object code. Second, we must issue an assistrun command. The command accepts as a parameter the XML configuration file produced by the compiler2 and consequently properly configures CLAM and actually starts computation.
3
Programmability
The time needed to develop running, scalable ASSISTcl programs is significantly smaller than the time needed to develop equivalent (both in the functional and in the performance sense) applications with different, more classical parallel programming tools, such as MPI, for instance. The programmer has handy ways to express simple as well as complex parallelism exploitation patterns. All the 2
and possibly manipulated by the programmer/user
1298
M. Aldinucci et al.
9000
1 support 1.5% support 1.5% (ideal) support 1% support 1.0% (ideal)
8000
support 1.5% support 1.0%
0.8
6000 0.6 5000
Efficiency
Execution time (secs)
7000
4000
0.4 3000
2000
0.2
1000
0
0 1
2
3
4
5
6
7
8
9
#PE
10
1
2
3
4
5
6
7
8
9
10
#PE
Fig. 1. Scalability and efficiency of Apriori data mining application
details needed to implement parallelism exploitation are handled by the compiling tools. Therefore, on the one hand the programmers may write the parallel structure of the application very quickly, while on the other hand performance exploitation is in charge of the tools, and again this consistently shortens the application development time.
4
Performance Results
We experimented different synthetic benchmarks as well as complete applications written in ASSISTcl. Typical performance numbers got out of Intel/Linux cluster are depicted in Figure 1 (left). Provided that computational grain is medium to coarse grain, ASSIST demonstrates good speedup and efficiency. The Figure plots values achieved running a data mining application exploiting an “a priori” algorithm.3 In the Apriori code execution, efficiency is constantly more than 80%, as shown in the Figure 1 (right).
5
Interoperability
ASSIST programs can invoke external services/code using several mechanism including typical CORBA ones. In particular, any portion of code included in an ASSIST program may call external CORBA objects methods and a whole ASSISTcl program can be wrapped into a CORBA object whose services/methods4 can be called from elsewhere. To demonstrate this feature, we prepared a Nbody program whose computational intensive part is performed in an ASSISTcl program but visualizing results accessing an X display via CORBA. The graphical output of the program is in Figure 2. The Nbody program is actually a code implementing the naive, n2 algorithm, as the goal there was only to demonstrate 3
4
The parameters of these runs can be summarized as follows: #DBtransactions = 1236000, average transaction len = 30, #items = 1000. Large item-sets: #patterns = 2000, average pattern len = 10, correlation between consecutive patterns = 0.5, average confidence in a rule = 0.75, variation in the confidence = 0.1 computation of the parallel program onto a given input data set, actually
ASSIST Demo
1299
Fig. 2. N-body ASSISTcl program interacting with graphic display via CORBA (left) and assistConf configuration tools (right)
interoperability via CORBA, even in case of small granularity ops (i.e. graphic display operations).
6
Heterogeneous Target Architecture & GRID
We are currently adapting the ASSISTcl compiling tools to produce object code for heterogeneous architectures (networks). Due to the structured layered implementation of the compiling tools of the ASSIST framework, in order to address heterogeneous architectures we simply have to perform two steps: first, we need to activate the possibility of delivering external data representation messages between hosts. ACE has a full support for such “processor neutral” data representation but at the moment the marshaling and unmarshaling routines do not use such feature. Second, we must arrange the makefile production in such a way that different DLLs (object code) are produced for the different machines in the target architectures. When all the needed different object codes are available, the assistrun command may exploit them accordingly to the contents of the XML configuration file. Both these tasks do not present any technical difficulty. They have been postponed only due to lack of human resources (programmers) in the project and we plan to have a working version of ASSISTcl compiler targeting heterogeneous architecture by the end of year 2003.5 In the meanwhile, we are moving the whole ASSIST framework to the GRID. As a first step, a tool has been built [6] that allows to manipulate (within a nice graphical interface) the XML configuration file according to information taken from GRID information services, in such a way that ASSISTcl programs can be eventually run on GRIDs. Figure 2 shows a snapshot of the tool. Actually, most of the work needed to run ASSISTcl programs on GRID is to be performed by hand by the programmer. In particular, most of the dynamic features of GRID6 5 6
development, experiments and debugging is performed on a mixed Linux/Pentium and MacOS/X/PowerPC machines e.g. resource lookup and reservation
1300
M. Aldinucci et al.
are completely in charge of the programmer interacting with the configuration tool.
7
Conclusion
We discussed some features of the ASSIST structured, parallel programming environment, that will be demonstrated during this conference in the new, for Europar, “demo session”. More precise information concerning ASSIST can be found in the other papers of our group. Acknowledgements. We wish to thank people that contributed in different, essential ways, to the development of ASSIST: R. Baraglia, D. Laforenza, M. Lettere, D. Guerri, S. Magini, S. Orlando, A. Paternesi, R. Perego, A. Petroccelli, E. Pistoletti, L. Potiti, N. Tonellotto, L. Vaglini, P. Vitale.
References 1. The Adaptive Communication Environment home page. http:// www.cs.wustl.edu/ ∼schmidt/ACE-papers.html, 2003. 2. M. Aldinucci, S. Campa, P. Ciullo, M. Coppola, M. Danelutto, P. Pesciullesi, R. Ravazzolo, M. Torquati, M. Vanneschi, and C. Zoccolo. A framework for experimenting with structured parallel programming environment design. In Proceedings of PARCO’03, 2003. to appear. 3. M. Aldinucci, S. Campa, P. Ciullo, M. Coppola, S. magini, P. Pesciullesi, L.Potiti, R. Ravazzolo, M. Torquati, M. Vanneschi, and C. Zoccolo. The implementation of ASSIST, an Environment for Parallel and Distributed Programmind. In Proceedings of Europar’03, 2003. to appear. 4. B. Bacci, M. Danelutto, S. Orlando, S. Pelagatti, and M. Vanneschi. P3 L: A Structured High level programming language and its structured support. Concurrency Practice and Experience, 7(3):225–255, May 1995. 5. B. Bacci, M. Danelutto, S. Pelagatti, and M. Vanneschi. SkIE: a heterogeneous environment for HPC applications. Parallel Computing, 25:1827–1852, Dec. 1999. 6. R. Baraglia, M. Danelutto, D. Laforenza, S. Orlando, P. Palmerini, R. Perego, P. Pesciullesi, and M. Vanneschi. AssistConf: A Grid Configuration Tool for the ASSIST Parallel Programming Environment. In Proceedings of the Eleventh Euromicro Conference on Parallel, Distributed and Network-Based Processing, pages 193–200. IEEE, February 2003. ISBN 0-7695-1875-3. 7. P. D’Ambra, M. Danelutto, D. di Serafino, and M. Lapegna. Integrating MPI-Based Numerical Software into an Advanced Parallel Computing Environment. In Proceedings of the Eleventh Euromicro Conference on Parallel, Distributed and NetworkBased Processing, pages 283–291. IEEE, February 2003. ISBN 0-7695-1875-3. 8. M. Vanneschi. ASSIST: an environment for parallel and distributed portable applications. Technical Report TR 02/07, Dept. Comp. Sc., Univ. of Pisa, May 2002. 9. M. Vanneschi. The programming model of ASSIST, an environment for parallel and distributed portable applications. Parallel Computing, 28(12):1709–1732, Dec. 2002.
KOJAK – A Tool Set for Automatic Performance Analysis of Parallel Programs Bernd Mohr1 and Felix Wolf2 1
Forschungszentrum J¨ulich, Zentralinstitut f¨ur Angewandte Mathematik, 52425 J¨ulich, Germany
[email protected] 2 Innovative Computing Laboratory, Computer Science Department, University of Tennessee, Knoxville, TN 37996
[email protected] Abstract. Today’s parallel computers with SMP nodes provide both multithreading and message passing as their modes of parallel execution. As a consequence, performance analysis and optimization becomes more difficult and creates a need for advanced performance tools that are custom made for this class of computing environments. Current state-of-the-art tools provide valuable assistance in analyzing the performance of mpi and Openmp programs by visualizing the run-time behavior and calculating statistics over the performance data. However, the developer of parallel programs is still required to filter out relevant parts from a huge amount of low-level information shown in numerous displays and map that information onto program abstractions without tool support. The kojak project (Kit for Objective Judgement and Knowledge-based Detection of Performance Bottlenecks) is aiming at the development of a generic automatic performance analysis environment for parallel programs. Performance problems are specified in terms of execution patterns that represent situations of inefficient behavior. These patterns are input for an analysis process that recognizes and quantifies the inefficient behavior in event traces. Mechanisms that hide the complex relationships within event pattern specifications allow a simple description of complex inefficient behavior on a high level of abstraction. The analysis process transforms the event traces into a three-dimensional representation of performance behavior. The first dimension is the kind of behavior. The second dimension describes the behavior’s source-code location and the execution phase during which it occurs. Finally, the third dimension gives information on the distribution of performance losses across different processes or threads. The hierarchical organization of each dimension enables the investigation of performance behavior on varying levels of granularity. Each point of the representation is uniformly mapped onto the corresponding fraction of execution time, allowing the convenient correlation of different behavior using only a single view. In addition, the set of predefined performance problems can be extended to meet individual (e.g., application-specific) needs.
H. Kosch, L. B¨osz¨orm´enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 1301–1304, 2003. c Springer-Verlag Berlin Heidelberg 2003
1302
1
B. Mohr and F. Wolf
Short Description of the KOJAK Tool Set
Figure 1 gives an overview about the architecture of the current prototype and its components. The kojak analysis process is composed of two parts: a semi-automatic multi-level instrumentation of the user application followed by an automatic analysis of the generated performance data. The first subprocess is called semi-automatic because it requires the user to slightly modify the makefile and execute the application manually.
Fig. 1. KOJAK tool architecture
To begin the process, the user supplies the application’s source code, written in either C, C++, or Fortran, to opari (Openmp Pragma And Region Instrumentor), which performs automatic instrumentation of Openmp constructs and redirection of Openmplibrary calls to instrumented wrapper functions on the source-code level based on the pomp api [6]. Instrumentation of user functions is done either on the source-code level using tau or using a compiler-supplied profiling interface. Instrumentation for mpi events is accomplished using a pmpi wrapper library, which generates mpi-specific events by intercepting calls to mpi functions. All mpi, Openmp, and user-function instrumentation call the epilog (Event Processing, Investigating and LOGging) run-time library, which provides mechanisms for buffering and trace-file creation. At the end of the instrumentation process the user has a fully instrumented executable. Running this executable generates a trace file in the epilog format. After program termination, the trace file is fed into the expert (Extensible Performance Tool) analyzer. The analyzer uses earl (Event Analysis and Recognition Language) to provide a highlevel view of the raw trace file. We call this view the enhanced event model, and it is where the actual analysis takes place. The analyzer generates an analysis report, which serves as input for the expert presenter. Figure 2 shows a screendump of the expert presenter.
KOJAK – A Tool Set for Automatic Performance Analysis
1303
Fig. 2. expert Presenter Example Screendump. Using the color scale shown on the bottom, the severity of performance problems found (left pane) and their distribution over the program’s call tree (middle pane) and machine locations (right pane) is displayed. By expanding or collapsing nodes n each of the three trees, the analysis can performed on different levels of ranularity.
In addition, it is possible to convert epilog traces into vtf3 format and analyze them manually with the vampir event trace analysis tool [8]. We are currently working on integrating both tools so that the instance with the highest severity of each performance problem found can be displayed by vampir on request. This would provide the ability to analyze the history of inefficient behavior in a time-line diagram or to do further statistical analysis using vampir’s powerful features. Currently, the measurement components are available for the following platforms: – – – – – –
Linux IA-32 cluster ibm power3 and Power4 cluster sgi mips cluster (Onyx, Challenge, Origin 2000, Origin 3000) sun Sun Fire cluster cray t3e hitachi sr8000-f1
On Linux clusters and hitachi sr8000-f1 systems the instrumentation of user functions is done automatically using the unpublished profiling interface of the pgi compiler or of the hitachi compiler, respectively. For IBM systems, EPILOG provides an automatic binary instrumentor, which has been implemented on top of DPCL. On all other systems,
1304
B. Mohr and F. Wolf
user function instrumentation must be carried out on the source-code level either manually or automatically utilising the TAU instrumententation facilities [4]. The analysis components are based on Python and run on any workstation or laptop.
2 Additional Information – The kojak tool suite including the source code can be downloaded from the kojak website. The website also provides a variety of technical papers, presentations, and screen dumps showing the analysis of example applications. → http://www.fz-juelich.de/zam/kojak/ – For more information on expert’s analysis and presentation features, see Felix Wolf’s Ph.D. thesis [2]. The theoretical aspects can also be found in [3]. A more detailed overview (than this short description) about kojak can be found in [1]. – Details on instrumentation of Openmp applications based on the pomp interface are described in [6] and [7]. – More information on the source-code instrumentation of user functions can be found on the homepages of the TAU [4] and PDT [5] projects. → http://www.cs.uoregon.edu/research/paracomp/tau/ → http://www.cs.uoregon.edu/research/paracomp/pdtoolkit/ – The kojak project is part of the European IST working group APART. → http://www.fz-juelich.de/apart/
References 1. F. Wolf, B. Mohr. Automatic Performance Analysis of Hybrid MPI/OpenMP Applications. 11th Euromicro Conference on Parallel, Distributed and Network Based Processing, 2003. 2. F. Wolf. Automatic Performance Analysis on Parallel Computers with SMP Nodes. Dissertation, NIC Series, Vol. 17, Forschunszentrum J¨ulich, 2002. 3. F. Wolf and B. Mohr. Specifying Performance Properties of Parallel Applications Using Compound Events. Parallel and Distributed Computing Practices (Special Issue on Monitoring Systems and Tool Interoperability), Vol. 4, No. 3. 4. S. Shende, A. D. Malony, J. Cuny, K. Lindlan, P. Beckman, and S. Karmesin. Portable Profiling and Tracing for Parallel Scientific Applications using C++. In Proc. of the SIGMETRICS Symposium on Parallel and Distributed Tools, pages 134–145. ACM, August 1998. 5. K. A. Lindlan, J. Cuny, A. Malony, S. Shende, B. Mohr, R. Rivenburgh, C. Rasmussen. A Tool Framework for Static and Dynamic Analysis of Object-Oriented Software with Templates. In Proc. of Supercomputing 2000, Dallas, TX, 2000. http://www.sc2000.org/techpapr/papers/pap.pap167.pdf 6. B. Mohr, A. Malony, S. Shende, and F. Wolf. Design and Prototype of a Performance Tool Interface for OpenMP. The Journal of Supercomputing, 23:105–128, 2002. 7. B. Mohr, A. Malony, H.-Ch. Hoppe, F. Schlimbach, G. Haab, J. Hoeflinger, S. Shah. A Performance Monitoring Interface for OpenMP. In Proc. of 4th European Workshop on OpenMP (EWOMP 2002), Rome, Italy, 2002. http://www.caspur.it/ewomp2002/prog.html 8. Pallas GmbH. Visualization and Analysis of MPI Programs http://www.pallas.de/e/products/vampir/
Visual System for Developing of Parallel Programs O.G. Monakhov Institute of Computational Mathematics and Mathematical Geophysics SB RAS, Novosibirsk, 630090, Russia
[email protected] Abstract. A program system for manipulating and translating of multimedia program skeletons (”films”) into parallel programs is considered. The main goal of this system is to make easier to create parallel programs for various parallel computing systems.
In this paper a programming environment for visual developing of parallel programs is presented. The environment allows writing parallel programs in abstract mode without consideration of particular computing system. Also the environment uses multimedia tools (graphics, text, animation, sound) for presentation of parallel algorithms to make developing of parallel programs more clear and understandable. The environment has a flexible GUI (Graphical User Interface) written in Java. The system is based on idea of program skeletons. A program skeleton is a multimedia, hierarchical, tree-like representation of parallel program in ”film” format [1],[3]. The film describes in animation mode (by sequence of multimedia stills) the dynamic activity of the algorithm in time-space coordinates. The film includes a parameterized template code of parallel computational algorithms. On the basis of the template code it is possible to create parallel program code after definition of parameters, variables and formulas in time-space points of algorithm activity. The film database includes the parallel algorithmic skeletons, proposed in [4], and can be extended by any user-defined program skeletons. The programming environment for visual developing of parallel programs consists of the following main parts: – Graph and film editors for creating, editing, manipulating, composition and viewing of program multimedia skeletons. – Icon language based on an open set of intuitively understandable small images for representing computational schemes, operations and formulas in points of algorithmic activity. – Modules for attachment of parameters, variables and formulas to skeletons and two translators for generating of code for sequential C-program and for parallel ANSI C-program with using MPI library for execution on many parallel and distributed systems and clusters. H. Kosch, L. B¨ osz¨ orm´ enyi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 1305–1308, 2003. c Springer-Verlag Berlin Heidelberg 2003
1306
O.G. Monakhov
Fig. 1. Structures examples for films
Fig. 2. Still editing interface of the film
The programming environment for visual developing of parallel programs is inspired by Active Knowledge Studio (AKS)[2], a framework supporting the development, the acquisition and the use of self-explanatory components related to methods of computational mathematics, physics and other natural science. The system provides seamless workflow from visual film format to automatic executable code generation and visualization of results. It includes film compo-
Visual System for Developing of Parallel Programs
1307
Fig. 3. Watching interface of the system.
Fig. 4. A panel of the node specifying mode: input/edit formulas
sition and manipulation part, attaching formula/sound and program synthesis part, program execution and result visualisation part. The film composition and manipulation part includes ”Structure Editor” (depicted by Fig.1) and ”2D Rendering Engine” modules for film creation and ma-
1308
O.G. Monakhov
nipulations with self-explanatory components. This part provides a user interface for watching (Fig.2), editing (Fig.3), and composing modes for film management system. The attaching formulas and program synthesis part includes ”Formula Editor” (Fig.4) and ”Program Synthesis/Compiling” modules for formulas attachment and executable code generation. This part provides a library of template programs and corresponding template control files representing basic algorithmic skeletons. This part also includes executable component codes being created. The program execution and result visualisation part provides execution of generated program and user interface for visualisation of input/output data for the program.
References 1. N. Mirenkov,: VIM Language Paradigm, in: Parallel Processing: CONPAR’94VAPP’IV, Lecture Notes in Computer Science, Vol.854, B.Buchberger, J.Volkert (Eds.), Springer-Verlag, pp. 569–580, 1994. 2. N. Mirenkov, A. Vazhenin, R. Yoshioka, T. Ebihara, T. Hirotomi, and T. Mirenkova,: Self-explanatory components: a new programming paradigm, International Journal of Software Engineering and Knowledge Engineering, Vol.11, No. 1, World Scientific, 2001, 5–36. 3. N. Mirenkov, O. Monakhov, R. Yoshioka,: Self-explanatory components: visualization of graph algorithms, in: Proc. of the 8th Internat. Conference on Distributed Multimedia Systems (DMS’02), (Workshop on Visual Computing), 2002, San Francisco, CA, USA, IEEE Press, p. 562–567. 4. M. Cole,: Algorithmic Skeletons: Structured Management of Parallel Computation, The MIT Press, 1989.
Peer-to-Peer Communication through the Design and Implementation of Xiangqi Abdulmotaleb El Saddik and Andre Dufour
Multimedia Communications Research Lab (MCRLab) University of Ottawa, Ottawa, Canada, K1N 6N5
[email protected] Abstract. In this work, we conducted a case study by implementing the Xiangqi game using a P2P networking technology. In so doing, we gained a deeper understanding of P2P communication in general and a more in-depth knowledge of the specific technology used. We encountered a number of issues while implementing our game using the JXTA P2P framework, but we were nevertheless able to create a working prototype that functioned satisfactorily.
1 Introduction Peer-to-peer communication is emerging as one of the most potentially disruptive technologies in the networking sector. If the interest in such technologies as Napster, Morpheus and Gnutella is any indication, peer-to-peer networks will be a major component in the future of computer communications and the Internet. Indeed, the Morpheus P2P network had, on average, 300,000 simultaneous users in 2001 and now regularly exceeds 3,000,000 users [1]. These numbers are too large to be ignored and the rapid growth of P2P is a testimonial to its usefulness. When we add the host of users using instant messaging clients or SETI@Home, to name just those two, it becomes clear that P2P networking technologies have proven their value and have made their way into mainstream use. In this work, we propose to investigate the suitability of P2P networking technologies for multimedia gaming applications through a case study. There were two objectives to the experiment: we wanted to gain a deeper understanding of peerto-peer communication in general and of one technology in particular, and we wished to evaluate the suitability of P2P networks for gaming applications of a particular type.
2 Application Characteristics Given that Xiangqi is a turn-based game somewhat similar to western chess, it has very low bandwidth requirements. Other than initial players discovery and board setup, messages are only exchanged when a player makes a move, in order to notify the opponent, and their content is very compact: they simply encapsulate a starting point and an ending point. Since time between moves can be of the order of several tens of seconds or more, latency and jitter are not major concerns. Indeed, the time for H. Kosch, L. Böszörményi, H. Hellwagner (Eds.): Euro-Par 2003, LNCS 2790, pp. 1309–1313, 2003. © Springer-Verlag Berlin Heidelberg 2003
1310
A. El Saddik and A. Dufour
a message to reach a player should be of the order of milliseconds or seconds at most, which is negligible when compared to the time that a user may legitimately expect to wait for the other player’s move. Likewise, even relatively high jitter (with respect to message transit time) would have negligible impact on the gaming experience because players expect certain variability in the time between moves. We do require, however, that messages be sent reliably, which is to say that they arrive in sequence and intact, otherwise the game will enter an undefined state. Overall, we have created a fairly undemanding application. This was done intentionally because the gaming application is little more than an interesting test harness in this connection. Since we were mostly concerned with learning about P2P networks it seemed appropriate to select the characteristics of the test application such that they could be easily supported by the network architecture. That way, we improved our chances of achieving a successful implementation.
3 Software Architecture Our Xiangqi implementation is based on the Java binding of the JXTA protocol suite [5]. Given that the low-level communication functions were handled using Java, it seemed natural to use Java/Swing for the graphical user interface. We strove to design our application such that the GUI and game logic were decoupled from the communications functionality. This increased code modularity and clarity and avoided the propagation of JXTA’s idiosyncrasies into all parts of the code. The following figure gives a summary of the software architecture for the Xiangqi application. Fig. 1. Xiangqi Application Architecture
Graphical User Interface and Game Logic Communications Infrastructure JXTA Communications Interface JXTA Services and Core (taken from the JXTA project) As we can see, the GUI and application logic are not separated. This is because they are closely related. For example, the game board, a graphical component, is responsible for determining if moves requested by the user are valid or not. It seemed most logical to proceed in this way because the information required to make this determination is contained within the board. The communications infrastructure mediates between the GUI and the JXTA-specific primitives in the lower layers of the architecture, providing simple send and receive primitives that do not don’t reveal the complexity of interacting with JXTA.
4 Game Implementation Issues In order to gauge the suitability of P2P networks, and in particular JXTA, to our gaming application, it is necessary to review the difficulties encountered in implementing the solution with the given technology.
Peer-to-Peer Communication through the Design and Implementation of Xiangqi
1311
4.1 Reliability One of the most salient issues we had to contend with during the implementation of Xiangqi was the fact that JXTA does not provide reliable messaging. Indeed, a significant amount of effort was required to devise an acknowledgement scheme using sequence numbers in order to confirm that messages arrived at their destination. It was also necessary to include a CRC in the messages to ensure the integrity of their content. While we didn’t fully implement a scheme to resend missing or erroneous messages, it would have been useful to do so. Also, we noted that JXTA’s internal messages, such as service and pipe advertisements, did not include any protection against lost or corrupted messages. This caused us significant problems because the pipe advertisements we embedded in the Xiangqi service advertisements were frequently corrupted in transit. The following is part of a JXTA pipe advertisement:
uuid-2CD0915F41244B19B341AB9F49204F4 3174A4A3EF51B4A9785C296532AFCC01404 As we can see, there are 66 hexadecimal numbers in this UUID. If only one of these 264 bits is corrupted, the peer on the receiving end will attempt to bind to an inexistent pipe. This situation occurred fairly frequently. It would have been necessary to implement a checksum or CRC algorithm in a class derived from JXTA’s advertisement classes in order to ensure the reliability of these advertisements. We can see that a significant amount of developer effort is required in order to ensure true reliability when using JXTA. Unfortunately, any application layer reliability scheme is redundant in the case where JXTA is running over a protocol such as TCP, which already provides for reliable transport. This is a significant failing in JXTA. 4.2 Network Reliability The issue with the messaging was not the only difficulty we encountered with respect to reliability in JXTA. We also found the JXTA P2P network to be somewhat unreliable. JXTA uses so-called rendezvous peers to propagate discovery requests throughout the network. Sun also defines rendezvous peers as “well-known peers that have agreed to cache a large number of advertisements in support of a peer group ”. Any peer can be configured to be a rendezvous peer, but in reality, relatively few are. If no rendezvous peers are present on the network or if they are configured such that they don’t form an uninterrupted connection between two new peers wishing to play Xiangqi, it will not be possible for the game to take place: the players will never be able to find each other. In this situation, the network is effectively “down”. We observed this several times during our experimentation. This is similar to the failure of a central server and quite an unexpected situation for a P2P network! However, it is important to note that the JXTA network is still in its infancy and that if the technology is generally accepted and becomes popular, it should be very rare to see the network fail for lack of rendezvous peers. Nevertheless, this did occasionally present a problem for our prototype application.
1312
A. El Saddik and A. Dufour
4.3 Ability to Serialize Advertisements One further problem we encountered was the fact that JXTA advertisements are not serializable. This makes it impossible to directly convert the objects to a form suitable for transmission, such as an octet stream. The issue first arose when we wanted to embed a pipe advertisement in GameRequestMessage objects and found that we couldn’t. Creating a subclass of the advertisement and making it serializable would not have helped because every class in the hierarchy must be serializable for the operation to succeed. Ultimately, we were forced to construct a serializable wrapper class that, given an advertisement, extracted the values of the instance variables and stored them before transmission. Once the wrapper object was reconstructed on the receiving side we could create a new advertisement using the stored values and work with that object as though it had itself been transmitted. This strategy was quite effective, but it seems a strange gaucherie on the part of JXTA’s designers to not have implemented the Serializable interface in the first place.
5 Conclusion and Future Work JXTA’s flaws notwithstanding, we were able to successfully implement the Xiangqi application using P2P technology. The implementation did require a significant amount of effort in order to ensure some reliability and more would doubtless be required to guarantee complete dependability of the inter-peer messaging, but we are confident that the objective is attainable. We feel that the idea of using peer-to-peer communication for gaming is sound and that the advantages far outweigh the drawbacks and difficulties associated with our particular implementation. Indeed, the fact that our application functions satisfactorily proves the concept. While successful, our implementation is still only a prototype and as such has a number of limitations and a number of areas could be improved upon. In particular, it would be desirable to complete the work required in order to make the P2P communication fully reliable. This might even go so far as to introduce a “keepalive” mechanism to periodically inspect the integrity of the pipe connection. From a feature standpoint, it would be attractive to users if they could see what players are online and select the opponent they wish to play with. A chat facility would also make the game play more enjoyable. There is much work to do and while this work describes a particular frozen version of the project, we intend to make the source code available on the Internet so that others may add to it and keep the project very much alive.
References [1] [2] [3]
http://www.openp2p.com/pub/a/p2p/2001/07/11/numbers.html, "Gnutella Blown Away? Not exactly. [July 10, 2001]", visited Dec. 4, 2002 Sun’s Java Technology and Web services Documentation: http://www.uo.com/newplayer/newplay_0.html, "ORIGIN – Ultima Online – Main", visited Dec. 4, 2002 http://www.tiaranetworks.com/news/010919pr_mfr.html, "Tasman News", visited Dec. 4, 2002
Peer-to-Peer Communication through the Design and Implementation of Xiangqi [4] [5] [6] [7] [8]
1313
http://www.openp2p.com/pub/a/p2p/2000/11/24/shirky1-whatisp2p.html?page=2, "What is P2P… and What Isn’t [Nov. 24, 2000]", visited Dec. 4, 2002 GONG, Li, Project JXTA: A Technology Overview, Sun Microsystems inc., Palo Alto, CA, USA, 2001. http://www.openp2p.com/pub/a/p2p/2001/05/30/hailstorm.html?page=2, OpenP2P.com:HailStorm : Open Web Services Controled by Microsoft [May 30, 2001]", visited Dec. 5, 2002 http://www.microsoft.com/netservices/passport/license.asp, "Licensing .NET Passport",visited Dec. 5, 2002 http://wwws.sun.com/software/jini/faqs/index.html#1q15, "Jini Network Technology", visited Dec. 5, 2002
Author Index
Agarwal, Amit 230 Agarwal, Saurabh 160 Agarwal, Tarun 230 Aguilar-Saborit, Josep 328 Akutagawa, Tomoyoshi 1082 Aldinucci, Marco 712, 1295 Aliagas, Carles 616 Allard, J´er´emie 1287 Alm´ asi, George 543 Aloisio, Giovanni 421 Alt, Martin 427, 742 Alves, Albano 969 Amorim, Claudio L. de 1207 An, Sunshin 1153 Anshus, Otto J. 47 Anvik, John 81 Ardaiz, Oscar 1265 Artigas, Pau 1265 Bad´ıa-Contelles, Jos´e M. 310 Badrinath, R. 1291 Bailey, Mark 261 Bal, Henri E. 4 Balaton, Zolt´ an 404 Barchet-Estefanel, Luiz Angelo 632 Barli, Niko Demus 603 Batista, Rodolfo 517 Baydal, Elvira 958 Beckett, Duan H. 533 Beer, Wolfgang 1064 Beletskyy, Volodymyr 297 Bell, Robert 17 Bellofatto, Ralph 543 Bellotti, Francesco 1129 Benthin, Carsten 499 Bergen, Benjamin 840 Berger, Michael 1109 Berta, Riccardo 1129 Berthold, Jost 732 Biersack, E.W. 1230 Bigonha, Mariza Andrade da Silva 1074 Bigonha, Roberto da Silva 1074 Bischof, Holger 682 Bister, Michel 624 Bjørndalen, John Markus 47
Blanco, Vicente 704 Blochinger, Wolfgang 722 Bochmann, Gregor v. 342, 1046 Boissier, Olivier 1091 Bongo, Lars Ailo 47 Bosschere, Koen De 556 Boukerche, Azzedine 1099 Boyle, Eamonn 640 Brand, Per 470, 675 Brasileiro, Francisco Vilar 169 Br¨ uning, Ulrich 481 Bruhn, Andr´es 481 Brunheroto, Jos´e 543 Brunheroto, Jos´e R. 109 Brunie, Lionel 374 Bunimov, Viktor 923 Byun, Ilsoo 1170 Cafaro, Massimo 421 Campa, Sonia 712, 1295 Carabelea, Cosmin 1091 Ca¸scaval, C˘ alin 543 Casta˜ nos, Jos´e G. 543 Cesar, Eduardo 141 Ceze, Luis 543 Chan, C.Y. 870 Chan, Tim K.T. 1017 Chatzigiannakis, I. 1003 Chen, G. 271 Cheung, Wai-leung 537 Choi, Gyu Sang 160 Choi, Jin-Seok 661 Choi, Jong-Mu 661 Chopra, Sumit 230 Choudhary, A. 271, 279 Christian, Volker 1064 Chu, Kenneth M.K. 1017 Chung, Sang-Hwa 995 Chunlin, Li 980 Cirne, Walfredo 169 Ciullo, Pierpaolo 712, 1295 Cl´ audio, Ana Paula 57 Clint, Maurice 640 Coppola, Massimo 712, 1295 Cordasco, Gennaro 911
1316
Author Index
Cores, Fernando 859 Coulon, C´edric 318 Cox, Simon J. 357, 412,525, 533, 1148 Crumley, Paul 543 Cunha, Jo˜ ao Duarte 57 Danelutto, M. 1295 Daoudi, El Mostafa 844 Das, Chita R. 160 Decker, Keith 384 Delaitre, T. 1281 DeRose, Luiz 7, 109 Dessloch, Stefan 3 Devillers, Sylvain 1216 Dickens, Phillip M. 938 Dietrich, Andreas 499 Dimitriou, T. 1003 Dong, Ligang 236 Dongarra, Jack 394 Downton, Andy 750 Duato, J. 1190 Dufour, Andre 1309 Dutra, Inˆes 509 El-Khatib, K. 1046 El Saddik, Abdulmotaleb 1309 Engelen, Robert van 261, 421 Ercan, Muhammet F. 537 Eres, Hakki 525, 1148 Erway, C. Christopher 543 Exposto, Jos´e 969 Eymann, Torsten 1265 Faber, Peter 303 F` abrega, Josep 989 Fahringer, Thomas 27 Fairman, Matthew J. 357 Felber, P.A. 1230 Feldmann, Anja 230 Fern´ andez, Juan C. 491 Fern´ andez, M. 1190 Ferreto, Tiago C. 7 Ferretti, Edmondo 1129 Ferscha, Alois 1064 Finney, Sean 1160 Fischer, Markus 481 Flammini, Michele 1056 Fleury, Martin 750 Flich, Jos´e 947 Fonseca, Tiago 517
Fortes, Jos´e A.B 98 Franchetti, Franz 251 Freitag, Felix 1265 F¨ urlinger, Karl 127 Fujie, Tetsuya 451 Fujita, Satoshi 201 Fung, Yu-fai 537 Gabarr´ o, Joaquim 640 Gagliano, Joseph 543 Gallard, Pascal 930, 1291 Gambosi, Giorgio 1056 Ganchev, Kuzman 1160 Garc´es-Erice, L. 1230 Garcia, Montse 616 Garcia-Molina, Hector 1273 Gasparini, Alessandro 1056 Gaudiot, Jean-Luc 576 Gautama, Hasyim 88 Gemund, Arjan J.C. van 88 Gerndt, Michael 127 Gil-Garc´ıa, Reynaldo 310 Gijzen, Martin van 820 Gin´e, Francesc 212 Gloria, Alessandro De 1129 Goldschmidt, Bal´ azs 1199 Gomb´ as, G´ abor 404, 1281 Gonzalez, Antonio 616 Gonz´ alez, Jes´ us A. 704 Gorlatch, Sergei 427, 682, 742 Gourgoulis, A. 1281 Goussevskaia, Olga 1118 Granat, Robert 800 Griebl, Martin 303 Guilloud, Cyril 38 Guirado, Fernando 218 Guo, H-F. 694 Gupta, G. 694 Hadibi, N. 1046 H¨ am¨ al¨ ainen, Timo 1141 H¨ annik¨ ainen, Marko 1141 Hagersten, Erik 760 Hagihara, Kenichi 135 Haridi, Seif 470, 675 Hasegawa, Atsushi 1082 He, Ligang 195 Heinemann, Andreas 1038 Hern´ andez, Porfidio 212, 859 Hern´ andez, Vicente 491
Author Index Herrero, Jos´e R. 461 Hiett, Ben 533 Hiraki, Kei 609 Ho, Tin-kin 537 Hoare, C.A.R. 1 Holmgren, Fredrik 470, 675 Hor´ anyi, A. 1281 Horspool, Nigel 242 ´ Horv´ ath, A. 1281 Housni, Ahmed 669 Hsiao, Hung-Chang 1248 Hu, Zhenjiang 789 Huedo, Eduardo 366 H¨ ulsemann, Frank 840 Hung, Luong Dinh 603 ´ ad 1137 Husz´ ak, Arp´
Kitzelmann, Emanuel 682 Klusik, Ulrike 732 Ko, Young-Bae 661 Kohn, Emil-Dan 1180 Kolcu, I. 271 Korch, Matthias 830 Kounoike, Yuusuke 451 Kov´ acs, J. 1281 Kowarschik, Markus 441 Kral, Stefan 251 Kranzlm¨ uller, Dieter 74 Kreahling, William 261 Krysta, Piotr 230 K¨ uchlin, Wolfgang 722 Kwon, Hyuk-Chul 995 Kwon, Jin B. 851
Iba˜ nez, Pablo 586 Ibe, Akihiro 1082 Imre, S´ andor 1137 Ino, Fumihiko 135 Ishar, Syarraieni 624 Iwama, Chitaka 603
Lacroix, Michel 669 Laforest, Christian 903 Lakhouaja, Abdelhak 844 Larriba-Pey, Josep-L. 328 L´ aszl´ o, Zolt´ an 417, 1199 Lawley, Richard 384 Layuan, Li 980 Lee, Ben 995 Lee, Jack Y.B. 870 Lee, Kee Cheol 1224 Lee, Seong-Won 576 Lehner, Wolfgang 348 Lengauer, Christian 303 Le´ on, Coromoto 704 Leonardi, Letizia 1027 Leung, Karl R.P.H. 1017 Lezzi, Daniele 421 Li, Chun Hung 1017 Li, Xiaolin 181 Lieber, Derek 543 Lim, Kyungsoo 1153 Lin, Chuan-Mao 1248 Liu, Hua 66 Llaber´ıa, Jose Maria 586 Llorente, Ignacio M. 366 Lobosco, Marcelo 1207 L¨ of, Henrik 760 Logie, Hans 556 Loogen, Rita 732 L´ opez, Pedro 947, 958 Loques, Orlando 1207 Lorenz, Juergen 251 Lottiaux, Renaud 1291
Jang, Si Woong 880 Jarvis, Stephen A. 195 Jiang, Hong 224 Jiao, Zhuoan 412 Johnston, David 750 Jonsson, Isak 810 Kacsuk, Peter 119, 1281 Kadayif, I. 279 K˚ agstr¨ om, Bo 800, 810 Kalogeropulos, Spiros 597 Kammenhuber, Nils 230 Kamvar, Sepandar D. 1273 Kandemir, M. 271, 279 Kangasharju, Jussi 1038 Karakoy, M. 279 Karasti, Olavi 1141 Karatza, Helen D. 1257 Keane, Andy 525 Keen, Aaron W. 770 Kergommeaux, Jacques Chassin de Kerherv´e, Brigitte 342 Kim, Han-gyoo 1224 Kim, Jai-Hoon 661 Kim, Jin-Ha 160 King, Chung-Ta 1248
38
1317
1318
Author Index
Loulergue, Fr´ed´eric 781 Loureiro, Antonio A.F. 1118 Lovas, R. 1281 Luck, Michael 384 Luque, Emilio 141, 212, 218, 566, 859 Lyardet, Fernando 1038 Magini, Silvia 712 Malony, Allen D. 17 Mamei, Marco 1027 Margarone, Massimiliano 1129 Mart´ın, Mar´ıa J. 287 Mar´ın, Mauricio 338 Martins, Thelmo 517 Martorell, Xavier 543 Masukawa, Masayuki 201 Mateus, Geraldo Robson 1118 Matossian, Vincent 1240 Matsuzaki, Kiminori 789 Mavronicolas, M. 1003 Mehofer, Eduard 242 Mehrmann, Lars 1064 Melo, Alba Cristina Magalhaes A. Melo, Renata Cristina F. 517 Mesa, J.G. 141 Messeguer, Roc 1265 Michael, Maged M. 651 Miguel-Alonso, Jos´e 98 Miller, Jim 2 Mindlin, Pedro 109 Miura, Hideyuki 603 Mizutani, Yasuharu 135 Mohr, Bernd 1301 Molina, Carlos 616 Molinari, Marc 412 Monakhov, O.G. 1305 Montero, Rub´en S. 366 Moreau, Luc 384 Moreira, Jos´e E. 109, 543 Morillo, Pedro 1190 Morin, Christine 930, 1291 Moure, Juan C. 566 M¨ uhlh¨ auser, Max 1038 Mu˜ noz, Xavier 989 Munt´es-Mulero, Victor 328 Nakajima, Tatsuo 1082 Nardelli, Marcelo 517 Navarra, Alfredo 1056 Navarro, Gonzalo 338
Navarro, Juan J. 461 Navarro, Leandro 1265 Negro, Alberto 911 Newhall, Tia 1160 Ng, Joseph Kee-Yin 1017 Ngoh, Lek Heng 236 Nikoletseas, S. 1003 Nudd, Graham R. 195 ¨ Ozsu, M. Tamer 318 Olsson, Ronald A. 770 Ordu˜ na, Juan M. 1190 Outada, Halima 844
517
Pacitti, Esther 318 Page, David 509 Parashar, Manish 66, 181, 1240 Pardines, I. 206 Park, Taesoon 1170 Park, Woojin 1153 Park, Yong Woon 880 Payne, Terry 384 Pe˜ nalver, Lourdes 491 Pereira, Fernando Magno 1074 Pesciullesi, Paolo 712, 1295 Pierson, Jean-Marc 374 Pina, Ant´ onio 969 Podhorszki, Norbert 119, 1281 Pohl, Thomas 441 Pons-Porrata, Aurora 310 Pontelli, E. 694 Popov, Konstantin 470, 675 Poromaa, Peter 800 Potiti, Laura 712 Pound, Graeme E. 357, 525 Priebe, Steffen 732 Qazzaz, Bahjat 859 Qin, Xiao 224 Radovi´c, Zoran 760 Rafea, Mahmoud 470, 675 Raffin, Bruno 1287 Ramparany, Fano 1091 Rantanen, Tapio 1141 Rauber, Thomas 830 Ravazzolo, Roberto 712, 1295 Reinicke, Michael 1265 Renard, H´el`ene 148 Rexachs, Dolores I. 566
Author Index Rilling, Louis 1291 Ripoll, Ana 218, 859 Rivera, Francisco F. 206, 287 Robert, Yves 148 Robles, Antonio 947 Rodr´ıguez, Casiano 704 Rodr´ıguez, Germ´ an 704 Roig, Concepci´ o 218 Rose, C´esar A.F. De 7 Rosenberg, Arnold L. 911 Ross, K.W. 1230 R¨ ude, Ulrich 441, 840 Rufino, Jos´e 969 Sakai, Shuichi 603 Sakellariou, Rizos 189 Sanomiya, Alda 543 Santos Costa, Vitor 509 Scarano, Vittorio 911 Scarpa, Michael 74 Schaeffer, Jonathan 81 Schimmler, Manfred 923 Schlesinger, Lutz 348 Schlosser, Mario T. 1273 Schmollinger, Martin 885 Scholz, Bernhard 242 Schulz, Martin 433 Schuster, Assaf 1180 Seitz, Christian 1109 Seitz, Ludwig 374 Shavlik, Jude 509 Shende, Sameer 17 Shinano, Yuji 451 Sibeyn, Jop F. 894 Siedlecki, Krzysztof 297 Siki¨ o, Janne 1141 Silva, Anderson 1207 Silva, Daniel Paranhos da 169 Simon, Vilmos 1137 Singh, David E. 287 Slogsnat, David 481 Slusallek, Philipp 499 Solsona, Francesc 212 Somogyi, Csongor 417 Song, Ha Yoon 1224 Song, Wenbin 525 Sorribes, Joan 141 Spiegel, Michael 1160 Spirakis, P. 1003 Spooner, Daniel P. 195
Stein, B. de Oliveira 38 Stewart, Alan 640 Strauss, Karin 543 Subramanyan, Rajesh 98 Suppi, Remo 859 Swanson, David R. 224 Szab´ o, S´ andor 1137 Szafron, Duane 81 Szalai, F. 1281 Szeber´enyi, I. Szeber´enyi, Imre 417, 1281 Tagashira, Shigeaki 201 Takagi, Masamichi 609 Takeichi, Masato 789 Takesue, Masaru 917 Tan, Joo Geok 236 Tan, Kai 81 Tanaka, Hidehiko 603 Tashiro, Daisuke 603 Tersty´ anszky, G. 1281 Thomas, Ken S. 533 Torquati, Massimo 712, 1295 Torres, Enrique F. 586 Trehel, Michel 669 Trinitis, Carsten 433 Truong, Hong-Linh 27 Tubella, Jordi 616 Ueberhuber, Christoph W. Uh, Gang-Ryung 261 Urvoy-Keller, G. 1230 Utard, Ga¨el 1291
251
Vadhiyar, Sathish 394 Valente, Marco Tulio 1074 Vall´ee, Geoffroy 1291 Vallejo, Isabel 640 Vandierendonck, Hans 556 Vanhatupa, Timo 1141 Vanneschi, Marco 712, 1295 Villaverde, K. 694 Vi˜ nals, Victor 586 Vivien, Fr´ed´eric 148 Vlassov, Vladimir 470 V¨ ocking, Berthold 230 Volkert, Jens 74 Waddell, Michael Wald, Ingo 499
509
1319
1320
Author Index
Walter, Maria Em´ılia Telles Wason, Jasmin 412 Weickert, Joachim 481 Weskamp, Nils 732 Whalley, David 261 Wheelhouse, Ed 433 Wilke, Jens 441 Wolf, Felix 1301 Woo, Sinam 1153 Xu, Fenglian 1148 Xue, Gang 357 Yamamoto, Kouji 1082 Yang, Xiaoyuan 859
517
YarKhan, Asim 394 Ye, Haiwei 342 Yeom, Heon Y. 851 Yoo, Anydy B. 160 Yoon, In-Su 995 Yuan, Xiao 218 Yuan, Xin 261 Zambonelli, Franco 1027 Zara, Florence 1287 Zerfiridis, Konstantinos G. 1257 Zhao, Henan 189 Zhu, Yifeng 224 Zoccolo, Corrado 712, 1295
