Proceedings of the International Conference ParCo2001
PARALLEL COMPUTING Advances and Current Issues
*++ Editors
G. R. Joubert A. Murli F. J. Peters M. Vanneschi
Imperial College Press
PARALLEL COMPUTING Advances and Current Issues
This page is intentionally left blank
Proceedings of the International Conference ParCo2001
PARALLEL COMPUTING Advances a n d C u r r e n t Issues Naples, Italy
4-7 September 2001
Editors
G. R. Joubert Clausthal University of Technology, Clausthal, Germany
A. Murli University of Naples "Federico II", Naples, Italy
F. J. Peters Philips Research, Eindhoven, The Netherlands
M. Vanneschi University of Pisa, Italy
Imperial College Press
Published by Imperial College Press 57 Shelton Street Covent Garden London WC2H 9HE Distributed by World Scientific Publishing Co. Pte. Ltd. P O Box 128, Farrer Road, Singapore 912805 USA office: Suite IB, 1060 Main Street, River Edge, NJ 07661 UK office: 57 Shelton Street, Covent Garden, London WC2H 9HE
British Library Cataloguing-in-Publication Data A catalogue record for this book is available from the British Library.
PARALLEL COMPUTING: ADVANCES AND CURRENT ISSUES Proceedings of the International Conference ParCo2001 Copyright © 2002 by Imperial College Press All rights reserved. This book, or parts thereof, may not be reproduced in any form or by any means, electronic or mechanical, including photocopying, recording or any information storage and retrieval system now known or to be invented, without written permission from the Publisher.
For photocopying of material in this volume, please pay a copying fee through the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, USA. In this case permission to photocopy is not required from the publisher.
ISBN
1-86094-315-2
Printed in Singapore by Uto-Print
CONFERENCE COMMITTEE Gerhard R. Joubert (Germany/Netherlands) (Conference Chair) Almerico Murli (Italy) Frans J. Peters (Netherlands) Roberto Vaccaro (Italy) Marco Vanneschi (Italy)
STEERING COMMITTEE Frans Peters (Netherlands) (Chair) Bob Hiromoto (USA) Masaaki Shimasaki (Japan) Dennis Trystram (France) Marian Vajtersic (Slovakia)
ORGANISING COMMITTEE Almerico Murli (Italy) (Chair) Roberto Vaccaro (Italy) (Vice-Chair) Pasqua D'Ambra (Italy) Luisa D'Amore (Italy) Riccardo Simino (Italy) (Exhibition)
FINANCE COMMITTEE Frans J. Peters (Netherlands) (Chair)
SPONSORS Center for Research on Parallel Computing & Supercomputers (CPS) - CNR, Naples and University of Naples "Federico II" v
INTERNATIONAL PROGRAM COMMITTEE Marco Vanneschi (Italy) (Program Committee Chair) Almerico Murli (Italy) (Program Committee Vice-Chair) HojjatAdeli(USA)
Lucio Grandinetti (Italy)
Andreas Reuter (Germany)
Giovanni Aloisio (Italy)
Rolf Hempel (Germany)
Dirk Roose (Belgium)
Hamid Arabnia (USA)
Rudolf Kober (Germany)
Domenico Sacca (Italy)
Farhad Arbab (Netherlands)
Norbert Kroll (Germany)
Giuseppe Serazzi (Italy)
Lidia Arcipiani (Italy)
Giuliano Laccetti (Italy)
Henk Sips (Netherlands)
Fabrizio Baiardi (Italy)
Domenico Laforenza (Italy)
Antonio Sgamellotti (Italy)
Arndt Bode (Germany)
Antonio Lagana (Italy)
Giandomenico Spezzano (Italy)
Alan Chalmers (UK)
Keqin Li (USA)
Vaidy Sunderam (USA)
Jacques Chassin de Kergommeaux (France)
Alberto Machi (Italy)
Domenico Talia (Italy)
Andrea Clematis (Italy)
Enzo Marinari (Italy)
Marco Tomassini (Switzerland)
Pasqua D'Ambra (Italy)
Valero Mateo (Spain)
Theo Ungerer (Germany)
Luisa D'Amore (Italy)
John Mellor-Crummey (USA)
Dennis Trystram (France)
Marco Danelutto (Italy)
Paul Messina (USA)
Marian Vajtersic (Slovakia)
Erik D'Hollander (Belgium)
Hermann Mierendorff (Germany)
Henk van der Vorst (Netherlands)
Koen De Bosschere (Belgium)
Giovanni Milillo (Italy)
Arjan van Gemund (Netherlands)
Giuseppe De Pietro (Italy)
Wolfgang Nagel (Germany)
Nicola Veneziani (Italy)
Jack Dongarra (USA)
Salvatore Orlando (Italy)
Heinrich Voss (Germany)
Ian Duff (UK)
Yoshio Oyanagi (Japan)
Helmut Weberpals (Germany)
Salvatore Gaglio (Italy)
Nikolay Petkov (Netherlands)
David Womble (USA)
Wolfgang Gentzsch (USA)
Wilfried Philips (Belgium)
Andrea Zavanella (Italy)
Giulio Giunta (Italy)
Erhard Rahm (Germany)
Hans Zima (Austria)
VI
VII
ADDITIONAL REFEREES Marco Aldinucci (Italy)
Yves Denneulin (France)
Hans-Peter Kersken (Germany)
Paolo Palmerini (Italy)
Alberto Apostolico (Italy)
Daniela di Serafino (Italy)
Jochen Kreuzinger (Germany)
Enric Pastor (Spain)
Eduard Ayguade (Spain)
J. Diaz
Norbert Kroll (Germany)
Matthias Pfeffer (Germany)
Kristof Beyls (Belgium)
Jose Duato (Spain)
Uwe Lehmann (Germany)
Francesco Scarcello (Italy)
Steeve Champagneux (France)
Bernardo Favini (Italy)
Giuseppe Lo Re (Italy)
R. Sommerhalder (Netherlands)
Guilhem Chevalier (France)
Uwe Fladrich (Germany)
Maria Lucka (Austria)
Gerardo Toraldo (Italy)
Massimo Coppola (Italy)
Ilio Galligani (Italy)
Giuseppe Manco (Italy)
Salvatore Vitabile (Italy)
Benedicte Cuenot (France)
Serge Gratton (France)
Eduard Mehofer (Austria)
Manuela Winkler (Germany)
Marco D'Apuzzo (Italy)
Mario Guarracino (Italy)
Lorenzo Muttoni (Italy)
Yijun Yu (Belgium)
Monica De Martino (Italy)
Juan GuillenDaniel Ortega Scholten (Holland)
Peter Zoeteweij (Netherlands)
This page is intentionally left blank
PREFACE The international conference ParCo2001 was held in September 2001 in Naples, Italy. This biannual conference, which is the longest running and most prestigious international conference on all aspects of parallel computing in Europe, again formed a milestone in assessing the status quo and highlighting future trends. Whereas many aspects of parallel computing have already become part of main stream computing, challenging new application areas are opening up. These aspects were highlighted by the invited speakers and the panel discussion. Together with the contributed papers and the mini-symposia an overall scenario of on the one hand consolidation of parallel computing technologies and on the other emerging new areas of research and development was presented. New areas in which parallel computing is fast becoming a strategic technology are image and video processing, multimedia applications, financial modelling, data warehousing and mining, to name but a few. New definitions of the parallel computing paradigm in the form of cluster and grid computing are gradually reaching the stage where their widespread application to a multitude of problem areas will become a viable option. With the ParCo conferences the emphasis has always been on quality rather than quantity. This approach resulted in the decision to run these conferences on a biannual basis, such that noticeable strides in technology development can be highlighted. Furthermore, all contributions were in the past reviewed during and after the conference. This latter approach had the disadvantage that proceedings were only available many months after the conference. In an attempt to shorten the time to publication in the case of ParCo2001, authors were requested to submit full versions of papers for the first review and selection process. Electronic versions of revised papers were made accessible to all registered delegates before the start of the conference. A final revision during and after the conference resulted in the papers incuded in these proceedings. It should be noted that papers presented as part of the various mini-symposia are not included in the printed proceedings. Their inclusion would have seriously delayed publication. Such papers will be considered for publication in Special Issues of Parallel Computing journal.
IX
X
The editors are indebted to the members of the International Program Committee, the Steering Committee and the Organising Committee for the time they spent in making this conference such a successful event. Special thanks are due to the staff of the Center for Research on Parallel Computing & Supercomputers (CPS) - CNR in Naples for their enthusiastic support. In this regard particular mention should be made of the key roles played by Almerico Murli (Organising Committee Chair), Pasqua D'Ambra and Luisa D'Amore in making this event such a great success. Gerhard Joubert Germany November 2001
Almerico Murli Italy
Frans Peters Netherlands
Marco Vanneschi Italy
CONTENTS
Committees
v
Preface
ix
Invited Papers
1
Deploying Parallel Numerical Library Routines to Cluster Computing in a Self Adapting Fashion J. J. Dongarra andK. J. Roche
3
GRID: Earth and Space Science Applications Perspective L. Fusco
31
e-Science, e-Business and the Grid T.Hey
33
Graph Partitioning for Dynamic, Adaptive and Multi-Phase Scientific Simulations V. Kumar, K. Schloegel and G. Karypis
34
Challenges and Opportunities in Data-Intensive Grid Computing P. Messina
40
Applications
41
Giant Eigenproblems from Lattice Gauge Theory on CRAY T3E Systems N. Attig, Th. Lippert, H. Neff, J. Negele andK. Schilling
43
Parallel Consistency Checking of Automotive Product Data W. Blochinger, C. Sinz and W. Kiichlin
50
Implementation of an Integrated Efficient Parallel Multiblock Flow Solver T. Bonisch and R. Ruhle
58
Distributed Simulation Environment for Coupled Cellular Automata in Java M. Briesen and J. Weimar
66
XI
XII
Human Exposure in the Near-Field of a Radiobase-Station Antenna: A Numerical Solution Using Massively Parallel Systems L. Catarinucci, P. Palazzari and L. Tarricone Tranquillity Mapping Using a Network of Heterogeneous PC A. Clematis, M. De Martino, G. Alessio, S. Bini, and S. Feltri Parallel Skeletons and Computational Grain in Quantum Reactive Scattering Calculations S. Crocchianti, A. Lagana, L. Pacifici and V. Piermarini Parallel Image Reconstruction Using ENO Interpolation J. Czerwinska and W. E. Nagel Training On-Line Radial Basis Function Networks on a MIMD Parallel Computer A. D'Acierno Parallel Simulation of a Cellular Landslide Model Using Camelot G. Dattilo and G. Spezzano Parallel Numerical Simulation of Pyroclastic Flow Dynamics at Vesuvius T. Esposti Ongaro, C. Cavazzoni, G. Erbacci, A. Neri and G. Macedonio
75
83
91
101
109
117
125
A Fast Domain Decomposition Algorithm for the Simulation of Turbomachinery Flows P. Giangiacomo, V. Michelassi and G. Chiatti
13 3
Massively Parallel Image Restoration with Spatially Varying Point-Spread-Functions G. Gorman, A. Shearer, N. Wilson, T. O'Doherty andR. Butler
141
Asynchronous Algorithms for Problem of Reconstruction from Total Image JV. M Gubareni
149
Parallel Flood Modeling L. Hluchy, G. T. Nguyen, L. Halada and V. D. Tran
157
XIII
The Xyce Parallel Electronic Simulator — An Overview S. Hutchinson, E. Keiter, R. Hoekstra, H. Watts, A. Waters, T. Russo, R. Schells, S. Wix and C. Bogdan
165
A Breakthrough in Parallel Solutions of MSC. Software L. Komzsik, S. Mayer, P. Poschmann, P. Vanderwalt, R. Sadeghi, C. Bruzzo and V. Giorgis
173
Implementation of a Parallel Car-Parrinello Code on High Performance Linux-Based Clusters 5. Letardi, M. Celino, F. Cleri, V. Rosato, A. De Vita and M. Stengel
181
PRAN: Special Purpose Parallel Architecture for Protein Analysis A. Marongiu, P. Palazzari and V. Rosato
189
Design of a Parallel and Distributed Web Search Engine S. Orlando, R. Per ego andF. Silvestri
197
Active Contour Based Image Segmentation: A Parallel Computing Approach V. Positano, M. F. Santarelli, A. Benassi, C. Pietra andL. Landini
205
Parallelization of an Unstructured Finite Volume Solver for the Maxwell Equations J. Rantakokko and F. Edelvik
213
An Hybrid OpenMP-MPI Parallelization of the Princeton Ocean Model G. Sannino, V. Artale and P. Lanucara
222
A SIMD Solution to Biosequence Database Scanning B. Schmidt, H. Schroder and T. Srikanthan
230
Parallel Program Package for 3D Unsteady Flows Simulation E. Shilnikov andM. A. Shoomkov
238
Parallel Lossless Compression Algorithm for Medical Images by Using Wavefront and Subframe Approaches A. Wakatani Adaptivity and Parallelism in Semiconductor Device Simulation H. Weberpals and S. Thomas
246
254
XIV
Algorithms
263
Analysis of Communication Overhead in Parallel Clustering of Large Data Sets with P-AutoClass S. Basta and D. Talia
265
A Coarse-Grain Parallel Solver for Periodic Riccati Equations P. Benner, R. Mayo, E. S. Quintana-Orti and V. Hernandez
274
Preconditioning of Sequential and Parallel Jacobi-Davidson Method L. Bergamaschi, G. Pini and F. Sartoretto
282
Parallel Checkpointing Facility on a Metasystem Y. Cardinale andE. Hernandez
290
An Efficient Dynamic Programming Parallel Algorithm for the 0-1 Knapsack Problem M. Elkihel andD. El Baz
298
Parallel Algorithms to Obtain the Slotine-Li Adaptive Control Linear Relationship J. C. Fernandez, L. Pehalver and V. Hernandez
306
A High-Performance Gepp-Based Sparse Solver A. Gupta
314
System Model for Image Retrieval on Symmetric Multiprocessors O.Kao
322
Granularity and Programming Paradigms in Parallel MPP Image Coding R. Norcen and A. Uhl
330
Parallel Quasi-Newton Optimization on Distributed Memory Multiprocessors /. Pardines and F. F. Rivera
338
A Parallel Condensation-Based Method for the Structural Dynamic Reanalysis Problem A. Pelzer andH. Voss
346
Combined MPI/OpenMP Implementations for a Stochastic Programming Solver D. Rotiroti, C. TrikiandL. Grandinetti
354
XV
The Same PDE Code on Many Different Parallel Computers W. Schonauer, T. Adolph and H. Hafner What Do We Gain from Hyper-Systolic Algorithms on Cluster Computers? W. Schroers, Th. Lippert andK. Schilling
362
370
Leader Election in Wrapped Butterfly Networks W. Shi, A. Bouabdallah, D. Talia and P. KSrimani
382
Software Technology and Architectures
391
Assembling Dynamic Components for Metacomputing Using CORBA A. Amar, P. Boulet and J. Dekeyser
393
Simulation-Based Assessment of Parallel Architectures for Image Databases T. Bretschneider, S. Geisler and O. Kao
401
Structured Parallel Programming and Shared Objects: Experiences in Data Mining Classifiers G. Carletti and M. Coppola
409
Access Histories Versus Segment Histories for Datarace Detection M. Christiaens, M. Ronsse andK. De Bosschere
417
On Skeletons & Design Patterns M. Danelutto
425
A Portable Middleware for Building High Performance Metacomputers M. Di Santo, F. Frattolillo, E. Zimeo and W. Russo
433
Using a Parallel Library of Sparse Linear Algebra in a Fluid Dynamics Application Code on Linux Clusters S. Filippone, P. D'Ambra andM. Colajanni Performance Evaluation of a Graphic Accelerators Cluster M. R. Guarracino, G. Laccetti and D. Romano
441
449
xvi Analysis and Improvement of Data Locality for the Transposition of a Sparse Matrix D. B. Heras, J. C. Cabaleiro andF. F. Rivera
457
Automatic Multithreaded Parallel Program Generation for Message Passing Multiprocessors Using Parameterized Task Graphs E. Jeannot
465
Parallel Launcher for Cluster of PC C. Martin and O. Richard
473
Parallel Program Development Using the MetaPL Notation System ./V. Mazzocca, M. Rak and U. Villano
481
PAVIS: A Parallel Virtual Environment for Solving Large Mathematical Problems D. Petcu and D. Gheorchiu
490
Development of Parallel Paradigms Templates for Semi-Automatic Digital Film Restoration Algorithms G. Sardisco and A. Machl
498
Exploiting the Data-Level Parallelism in Modern Microprocessors for Neural Network Simulation A. Strey andM. Bange
510
Extending the Applicability of Software DSM by Adding User Redefinable Memory Semantics B. Vinter, O. J. Anshus, T. Larsen and J. M. Bjorndalen
518
Industrial Perspective
527
Porting and Optimizing Cray T3E Applications to Compaq AlphaServer SC Series J. Pareti
529
Author Index
541
INVITED PAPERS
This page is intentionally left blank
DEPLOYING PARALLEL N U M E R I C A L LIBRARY R O U T I N E S TO CLUSTER C O M P U T I N G LN A SELF A D A P T I N G FASHION K E N N E T H J. R O C H E , J A C K J . D O N G A R R A Department
of Computer Science, The University 203 Claxton Complex, Knoxville, Tennessee 37996-3450 roche,
[email protected] of
Tennessee,
This paper discusses middleware under development which couples cluster system information with the specifics of a user problem to launch cluster based applications on the best set of resources available. The user is responsible for stating his numerical problem and is assumed to be working in a serial environment. He calls the middleware to execute his application. The middleware assesses the possibility of solving the problem faster on some subset of available resources based on information describing the state of the system. If so, the user's data is redistributed over the subset of processors, the problem is executed in parallel, and the solution is returned to the user. If it is no faster, or slower, the user's problem is solved with the best appropriate serial routine. The feasibility of this approach is empirically investigated on a typical target system and results reported which validate the method.
1
Overview
On multi-user computing systems in which the resources are dedicated but the allocation of resources is controlled by a scheduler, one expects a job, once allocated, to finish execution in a predictable amount of time. In fact, the user on such a system usually submits a job through a batch script which requires an upper bound on the predicted runtime of the task being queued. The user is billed for the entire time requested and thus has a responsibility to himself and other users to understand the behavior of his code. Scheduling schemes on such systems attempt to order the computations in a fair manner so that user jobs show progress towards completion in a timely fashion while the overall system throughput is maximized. One problem with this approach is that such a scheduling scheme is easier to talk about than to implement.(ref. [1]) It is a fact of life that the multi-processor scheduling problem is iVP-complete in the strong sense." Thus, developers must look for algorithms which are "Theoretically, a decision problem II is NP-complete in the strong sense if (II 6 NP)A(3Tlp which is JVP-complete). For decision problem II and polynomial p, defined over the integers, lip is the restriction of II t o instances / 3 Max[i] < p(Length[I]). If II is solvable by a pseudo-polynomial time algorithm, then n p is solvable in polynomial time. Consider the
3
4
efficient. This is a difficult and time consuming task (which can't really be avoided on production scale or commodity clusters such as those at national laboratories and supercomputing centers). The large variance in the average duration and demand of user jobs, for instance, further complicates the task. It's complicated. Even though there's no provably optimal way of addressing this problem, people do it all the time because it has to be done. We don't make further considerations of such systems in this paper. In shared, multi-user computing environments, such as clusters of workstations in a local area network, the notion of determinism in computations can be lost due to resource contention. Thus, successive runs of the same linear algebraic kernel with the same problem parameters, for instance, may result in grossly different wall clock times for completion due to the variability of the work load on the CPUs from competing tasks. If users compute responsibly and in coordination with one another such systems can be and are successful. (Administrators intervene otherwise to mediate serious contention problems.) This approach is often more efficient, for instance, in the developmental stage of parallel application codes due to the constant testing and debugging of software, or in groups where the average user job tends not to saturate the system resources and runs to completion on a relatively short time scale (e.g. minutes or even hours). One way for the user to possibly make better use of the available resources in such an environment is to employ the information describing the state of the computational system at runtime to select an appropriate subset of resources for the specific kernel at hand. It is acknowledged that making low risk predictions in such a system when multiple users are sharing the resources cannot be done with certainty. There is nothing to preclude the event that the demand on the resources may change dramatically in the time which transpires between deciding on a set of resources and getting one's problem set up and ready to go on the resources. Nonetheless, it seems negligent not to try to use available system related data at runtime. In the very least a user can identify saturated resources in the system and avoid allocating them for his/her run. In the event that the overall system behavior fluctuates about some time sensitive normal level of activity, then a statistical analysis of the collected data can be used as the basis for a predictive model of the system's behavior at some future time. Software, such as NWS, the multi-processor scheduling problem TIMS'- given a finite set J of jobs, a length l(j) e Z+Vj e J , a number, m e Z+, of processors, and deadline D e Z+ , is there a partition J = Ji U Ji U • • • U Jm of J into m disjoint sets such that m a i [ J ] . - j . l(j) : 1 < i < m] < Dl This problem is NP-complete in the strong sense and thus cannot be solved by a pseudo-polynomial time algorithm unless P = NP. (For proof see reference [2], for related information see references [3,4,5,6,7,8].)
5
Network Weather Service, operates sensors in a distributed computing environment and periodically (in time) collects measured data from them.(ref. [9]) NWS includes sensors for end-to-end T C P / I P performance (bandwidth and latency), available CPU percentage, and available non-paged memory. The collected data is kept and analyzed as a time series which attempts to forecast the future behavior of the system through low order ARM A, autoregressive moving averages, methods. This paper discusses software being developed which couples system information with information specifically related to the numerical kernel of interest. The model being used is that a user is assumed to contact the middleware through a library function call during a serial run. The middleware assesses the possibility of solving the problem faster on some subset of available resources based on information describing the state of the system. If so, the user's data is redistributed over the subset of processors, the problem is executed in parallel, and the solution is returned to the user. If it is no faster, or slower, the user's problem is solved with the best appropriate serial routine. It is conjectured that if the underlying application software is scalable then there will be a problem size which marks a turning point, Ntp, for which the time saved because of the parallel run (as opposed to the best serial runs for the same problem) will be greater than the time lost moving the user's data around. At this value, such software is deemed useful in the sense that it provides an answer to the user's numerical question faster than had the user done as well as an expert working in the same serial environment. That is, it benefits even the expert user working on a single node of the shared system to use the proposed software for problem sizes in which Nuaer > Ntp. As a case study we consider the problem of solving a system of dense, linear equations on a shared cluster of workstations using the ScaLAPACK software, (ref. [10]) A discussion of some specific implementations tested is made and results of selected experiments are presented. It is observed that even with naive data handling the conjecture is validated in a finite ensemble of test cases. Thus there is motivation for future studies in this area. It is also observed that the expert user in the parallel environment can always complete the dense, algebraic task at hand faster than the proposed software. (There are no clear winners for small problem sizes since each approach solves the problem serially with the best available version of the library routine.) This is no surprise since even in the most ideal cases, the proposed software has to touch the user's data at least enough to impart the relevant data structure expected by the logical process grid. The parallel expert, on the other hand, is assumed to be able to generate relevant data structures in-core, in-parallel at the time of the distributed run. This user also knows how to initialize the
6
numerical library routine, and make compilation time optimizations. He/she is probably not the typical scientist who has likely already labored just to reduce their problem to linear algebra. There are, in fact, many details to account for by any user before the parallel kernel runs correctly. Reporting the results of this expert user provides a basis for comparison to the other projected users and scenarios. 2
2.1
Numerical libraries in shared, homogeneous, distributed environments The computing
environment
In the development of the current investigation heterogeneous grid computing systems have not been the central focus. (See references [11,12,13,14,15,16,17].) However, it is noteworthy that one of the goals in resource selection when considering a pool of heterogeneous (and potentially geographically distributed) candidate resources is to achieve as much homogeneity in the allocated resources possible. Here's at least one complication of scheduling in a shared distributed system which is general to both heterogeneous and homogeneous systems: the scheduler of resources for a task in question has to try and allocate resources which not only look homogeneous at the instant of inquiry, but remain as homogeneous as possible for the duration of time that the task is in (parallel) execution. In short, even if we could solve the general multi-processor scheduling problem at some specific instant in time, we can't count on this partitioning to assist us in forecasting the state of the system resources at some time in the future. This is due to the fluctuating properties of system resources which one can observe in a shared environment. This brief subsection intends to describe the notion of homogeneity in the context of the current study. Some sample results of timing various operations in one of the systems tested demonstrates the notion as it is observed empirically. In complex mechanical systems the notion of homogeneity usually implies that the system behaves in a predictable manner when performing a specific task only in the absence of external influences. If this definition applies to computational systems (see Figure 1 for a sample computing environment), then it cannot be that a shared set of resources alone, such as a cluster of workstations in a local area network, is homogeneous. Usually such a system is only meaningful when responding to a user's requests. Users' requests are developed externally and then serviced by the system at runtime. Since there is no way to know when a user intends to make requests in such an open
7 Users, etc.
100 Mbit
5 Network File System Sun's NFS (RPC/UDP)
100 Gbit Switch, (fully connected)
100 Mbit Switch, (fully connected)
Remote memory server, e.g. IBP(TCP)
QQ Figure 1. The figure is a diagram of part of the local area net work in which many of the current investigations were made. It is noteworthy for the purposes of interpreting some of the results presented in this paper that the memory depot and network file server are separate machines in reality sitting on a shared network. The clusters on which we have developed the current study are removed from the shared network through one of two switches through which the cluster of workstations is said to be fully connected. It is a factor for the types of studies we have made that there is only one, 100Mbit line for all of the network flow to and from the network disk or the memory server.
system, any specific task, such as solving a set of linear equations, is likely to exhibit different total wall times for completion on subsequent runs. So what does one mean by a shared homogeneous, distributed computing environment? Naively, it is assumed that the hardware specifications and available software are duplicated between compute nodes on such a system. This is not enough however (and may not be necessary). The notion of homogeneity has meaning only in terms of some specific system state engaged in some specified activity as observed in an ensemble of test cases. Let us elaborate on this thought a little. Physical system parameters, when observed at equidistant time intervals and kept in collections of ordered data, comprise a time series, (see ref-
8
erences[18,19,20,21]) Because of the inherent fluctuations in the system parameters, CPU loads on a shared cluster for instance, the time series of the state of the shared system is a non-deterministic function. (For instance, the activity level of system resources often reflects the underlying nature of the humans which use the system. At lunch time, dinner time, or bed time one often observes a decrease in overall system activity. In the morning, one often observes some adiabatic growth of activity in the system as users arrive and begin working. This growth continues until some normal level of system activity is achieved. For some duration of time, one may expect the system resource activity levels to fluctuate around this normal. But the point is that the activity norm is time of day dependent more often than not.) Non-deterministic time series can only be described by statistical laws or models. To study such systems formally one assumes that a time series can only be described at a given instant in time, t, by a (discrete) random variable, Xt, and its associated probability distribution, fxtThus, an observed time series of the system parameters can be regarded as one realization of an infinite ensemble of functions which might have been generated by a stochastic process -in this case multi-users sharing a common set of resources as a function of time. Stochastic processes are strictly stationary when the joint probability distribution of any set of observations is invariant in time and are particularly useful for modeling processes whose parameters tend to remain in equilibrium about a stationary mean. Any stationary stochastic process can be described by estimating the statistical mean p (x = n _ 1 £)"=i xt), its variance a2 (s% = n _ 1 J2t=i(x* ~ ^)2)> * n e autocovariance function {cxx(k) = n _ 1 S£Ti (xt — x)ixt-k — x) e.g. the extent to which two random variables are linearly independent), and the sample autocorrelation function which is a kind of correlation coefficient (rXx(k) — Cxa(fc)(cx»(0))-1,fc = 0,...,n - l ) . A discrete random process for which all the observed random variables are independent is the simplest form of a stationary stochastic process. For this process, the autocovariance is zero (for all lags not zero) and thus such a process is referred to as purely random, or white noise. Usually, the observable system parameters such as CPU loads and available memory are not independent. Thus, the notion of homogeneity is manifest only in observing a specific task on the system, such as data I/O or multiplying matrices in-core, repeatedly under normal system activity on each of the computing nodes said to comprise the system. An expectation value (fi) for the specified task will be formed for each unit only in this time tested man-
9 ner. 6 One can subsequently compare the results from each of the units and determine a level of similarity between them. Ideally, the time to complete any serial task would be the same within some standard deviation (approximated by the square root of the variance) regardless of the compute node executing it. Further, the error bars should ideally tend to zero as the number of observations tends to infinity. This is not achievable in practice, clearly. (For non-stationary processes, one filters the available data sets thus transforming the problem into a stationary form.) Figures 2, 3, 4, and 5 illustrate the results of portions of an empirical study on one of the systems used to develop the current investigation. In each of the figures, successive runs of a task are time stamped and recorded to file. The results presented were analyzed statistically and only the mean and root of the variance are reported. The runs were conducted over the course of weeks and, as much as possible, during the same hours of the day (10am until 5pm,EST). No attempt was made to push users off the shared system accept during development of some test cases. In Figure 2, an assessment of the CPU of each node when executing (serial runs) a numerical kernel rich in floating point operations as a function of the numerical problem size is made. In this case the time to solution and performance are reported and there is a clear correlation in the expected behavior of each node. Figure 3 is composed of three plots. The plots look at the read and write I/O times per node as a function of bytes on each node's local disk, on the local network users' disk (in our case operating under Sun's NFS utilizing RPCs, UDP/IP), and on a memory server (running IBPC (ref. [23]), TCP/IP) on the same local network but with a different IP address from any node on the cluster or the NFS server itself. In plot one, the local disk accesses, one can only imagine that sporadic user activity is responsible for the larger variances in some of the reported results. To within the error bars 6
If observational data is to be of use in developing advanced software, a standard metric has to be agreed upon and needs to be reliable across different platforms. PAP I (ref. [22]), which accesses hardware counters as well as the system clock, provides such a tool and has been used throughout this investigation when recording observations. C IBP, the Internet Backplane Protocol, is software for managing and using memory on a distribution of disks across a network. Its design is really intended for large scale, logistical networking. However, because it was designed with a client /server model in mind, is also useful for our purposes (as will be discussed further in this report.) The client accesses remote resources through function calls in C, for instance. The multi-threaded servers have their own IP addresses. The client has to know this address in advance as well as to which port the server listens. Usually one sets up his own IBP depot and can choose the port. The IBP group also manages some public domain (free) depots across the country which one can use.
10 TORC(Homogeneity(CPlJ)) Performance of solve routine _GESV() in Gflops
TORC(Homogeneity(CPU)) Time to Solve Ax=b:: _GESVQ (ATLAS)
50
|
T
i
|
i
|
i
|
i
|
ns*!
0.40
/ CPU homogeneity on TORC::
/
/ 40
Time to solve Ax=b::_GESV()
30
• Morel * — »torc2
/
CPU homogeneity on TORC:: 0.38 -
Performance of _GESV() in GFLOPS
Tim
*
«
?0
"»•
| 100
*
UDP/IP, Network
torc3 torc4 torcS
• * *
(pfl) = (U) — - (Pfl) = (6.L)
j
t
//SM -
•
S00O
TOCO
N(pnUantlM)
Figure 6. The developer is responsible for understanding how to get the most out of the kernel being implemented in the proposed library. The plot on the left demonstrates a time range of over lOQseconds for the 8 processor run due to either a very poor choice of logical grid layouts, or the best possible. The plot on the right compares (a ratio of execution times is formed) the best run of the ScaLAPACK routine pdgesvQ versus the High Performance Linpack benchmark over the same number of nodes on the system. HPL is highly tuned and is really hard to beat in practice, even for experts. The results demonstrate the reliability of ScaLAPACK to perform well provided the correct system parameters are identified. Tuning the major kernels in ScaLAPACK has been a major concern on the side of the application routine developers.
to model the kernel as a function of the problem parameters as well as the state of the system. The basic communication model assumes a linear relation between any two nodes i, j such that the time to send a message of size X bytes is tcommii^ibandwidthij,latencyij) ~ latencyij + bandwidthj~j • X. Fora,pxq process grid, the factorization of a m x n panel, again, (see the documentation regarding HPL (ref. [31])) occurs within a single process column (p processors). Since the algorithm recurses within a panel, the assumption is made that any of the p processors within the panel perform at level 3 BLAS (ref. [32]) rates. Here is where we employ the notion of homogeneity in our target system as regards both communication and computational expectations. In the kernel of interest we are always bound by the slowest machine in the broadcast and update phases. Thus we select p processors which reflect homogeneity in the sense defined in this paper. If one only considers the time the nearest neighboring process columns will spend in the broadcast phase (after the fac-
23
torization), then the time to factor and broadcast a block panel is estimated for a collection of homogeneous resources by tfactor(m, n) + hroadcaat(m, n) ~ fdgemmi— - *jf) + latency{l + nlgip)) + (bandwidth)'1 (2n21 g(p) + nst). The Schur update of the remaining n x n matrix is approximated for homogeneous systems by tup(iate — Sq'1 (bandwidth)'1 n • nb + latency(p + hip) — 1) + fdgemm • n • nb(*f + 2-£). ^ere "^ *s *ke block size, m,n are the panel dimensions (not global) for the factorization and fdgemm approximates the performance of matrix multiply on an arbitrary node of our homogeneous system. Naturally, the system constraints have to be considered despite the assumption of homogeneity. The total time to perform this factorization in a homogeneous system may be approximated as ~ (start,i = 0) £"+=n6(*/actor(H ~ *>«&) + hTOadcast(n ~ *,nb) + tupdate(n
- t -
nb,nb)).
Quite frankly, one really requires a separate paper simply to work through the details of how we arrive at this time function and how it is used in the selection process. Clearly, we wish to minimize this function. Scheduling schemes are truly difficult however. People try simulated annealing, genetic algorithms, apply low order time series analyses, etc. The problem is of interest to the community in general. We will address the issue at this time. Future papers will describe our efforts in this area in detail. For now, suffice it to say, we crudley minimize this function based on the available system resources through an adhoc means and observe how well the allocated resources complete the task relative to the time we predict it should take based on analyzing the time function. The plots in Figure 7 reveal the faithfulness of our model in its current form. 3.2
Data movement scenarios
Moving and mapping m • n double precision matrix elements is informally discussed. The major consideration is on getting the data, matrices A e 3J m,n t-> (m*n*sizeof(double))bytes and vectors b € 5ft"1 •-> (m*sizeof(double))bytes , from the user and in place for the parallel solve routine. d It is assumed that the data is generated in a natural ordering, which is to say, in the language C for instance, that *(A + i + j*n) references the value of the matrix element in row i and column j of matrix A when 0 < i < m and 0 < j < n. This mapping is of general importance because the scalability of numerical calculations in a distributed computing environment is dependent upon spending more time computing than moving a user's data around. In particular, if numerical libraries such as that being investigated here are to be successful, it is necessary d
In the test cases m = n a s w e consider square matrices in the parallel solve routine.
24 Busy TORC Rons •: Total Wall Clock Time :PDGESVKandOnly
Busy TORC Runs:: PDGESV Kernel Only
Perfonnance Model:: Predicted vs. Measured Results
Ratio of Measured Tune vs. Tune Predicted by the Performance Model
; _s*> i™a jrt*^
; A—i. Ideal
1 '•* P
1.2
||.IS
^ /
f>
i ; - ^
(
/
/
State
• K
-
"
•—•• (n 2 * sizeof(dauble))bytes]-^ problem size = n), the block sizes (nbjrow, nb-column, or nb), the number of process columns in the logical rectangular process grid (npcols), the number of process rows in the logical rectangular process grid (nprows), and the matrix elements, A. In fact, the mapping is quite easy to implement naively but it is found that real improvements on such a mapping are more difficult to achieve than expected. In the remainder of this section we outline the general scenarios that we have considered. In each instance, the corresponding read of the data, from the perspective of the parallel application routine, is also discussed. In scenario one, the user first generates the data in-core. There are n2 +n
IIOOO
25
elements and each requires sizeof (double) bytes of memory. This user then passes a pointer to the middleware which in turn (after deciding on the process grid) writes (nprows x npcols) 2d block cyclically pre-mapped work files to the network disk (NFS) as well as nprows work files for the vector elements. In total, of course, only (n 2 + n) * sizeof (double) bytes are written to disk. However, it is useful to compare the time it takes to simply write natural A to a single file on the network disk versus the time to write the elements of A to multiple files in an order imposed by the 2d block cyclic mapping, (see Figure ) In this scenario, the processors involved in the parallel application routine have only to read their matrix elements from disk before executing the parallel application. It is noted that no extra elements are read per processor as the mapping has already taken place in the middleware and is exact. The second general scenario begins with the user's data having been written to a file on either the local network disk or on the memory depot. After the middleware determines a set of processes for the application routine, the dataJiandle is passed to the application routine from the middleware. The natural data is brought in-core by a chosen processor (root) from the logical process grid. The data may be distributed by root in a manner which imposes the mapping during point to point communication with the other nodes(2a). Alternatively, root may proceed to broadcast the data as is and let the mapping be done by each process locally(2b). (We label these scenarios here so the reader can make sense of the plots.) In the final scenario we consider the case where the natural data is brought in-core in parallel by each processor in the logical process grid. The mapping may be imposed exactly during the load phase through random access to the file (3a). Alternatively, bulk data can be brought over the network by each process and the mapping carried out in-core (3b). It is noted that one may contend with network congestion in this scenario. In reality, the developer has to experiment with these approaches before truly understanding the best approach for a given system. 4
Results and conclusions
Figure 8 is interesting to think about. For one thing, we clearly see Ntp exists for each of the scenarios reported on the graph. Notice that the scale of the graph is linear-log. Thus, small separations on the graph suggest large separations in time. Not all the scenarios are reported but the point is clearly made -the user will benefit from interfacing with the proposed software. Plot two of Figure 8 demonstrates at once the strength ScaLAPACK for the dense algebraic kernel, and the reason we pursued this idea in the first place
26 Total Wall Times Solving Ax=b:: TORC (data_generation (and movement)) + (solve)
Total Wall Time for kernel .pdgesvQ:: TORC
10000
10000
1000
1000
NPROOCS-.
^
100
Tolal_wail_time[expen/distr] Tolal_walljime[NFS_scenario(l)] » — « Toul_wall_timetIBP_scenario(2a)] «! ToIal_Wall_Time[IBP_scenalio(3b)|l' ^ Tota]_wall_time(expen/seria!j
4000 6000 N (problem_size)
.pgesv[expert/dislr] * _pgesv[NFS_scenario(l)] » * •s
• _pgesv[lBP_scenario(2a)] ^ _gesv[expert/serial] -s _pgesv[IBP_scenario(3b)]
I i I 4000 6000 N (problem_size)
I 8000
Figure 8. The total time is reported for multiple approaches to solving the linear problem Ax — b. The serial and parallel expert cases proceed without the intervention of additional software. All other runs reported invoke the middleware just as desribed in the current work.
-scalability. For each run on the graph the number of processors allocated for the parallel runs is provided. The expert user in the serial environment can no longer conduct his normal activities in a timely manner. His task is over lOOOseconds behind the proposed software when we stop counting. Figure 9 is also interesting because it displays the overheads involved in simply porting the user's data into a form amenable to the parallel application routine. The times reported for the top two plots are simply the time it takes for the expert user (serial, or parallel) to generate his data before calling the library routine. Next, if we look at the local network disk data (NFS), handling there are multiple plots. The original time it took for the middleware to generate the block cyclic data files for each process in the logical process grid was due to a careless mapping. This is an easy mistake for user's -even developers- to make. Simply compare the time it takes to write some set number of bytes to the network disk with these maps. The only difference in the poorly implemented case (which also affects the numbers in Figure 9) and the faster mappings is in buffering. Similarly, in the load of the workfiles in scenario (2a) attempts to first map the user's natural data and then communicate it. It is simply faster to let multiple requests engage the memory server. Of course, this is only true until such traffic generates
27
2000
T"
4000
6000
8000
10000
~T
T~
0
2000
4000
6000
8000
10000
1000
IBP.DATA
NFS DATA
100
geiKraie_A,bJiicore[nst/seiial] « wntc_wo4_Files[mw/serial] •* load_A',b'_wrile_x[appl/disci] (1) - seiial_wriie_no_map D writtj»oikJilesj3i]f[n]w/serial]
I generate, A,b_remotely[usr+mw/serial] I load_A,b_write_x[appl/distr] (2a) s load_A,b_writc_x[appl/distr] (3b)
_l_ 2000
4000
6000
8000
10000
N
0
2000
4000
6000
8000
10000
N
Figure 9.
congestion on the network. We don't see that here but easily could as larger problems and methods are investigated. There is much left that could be discussed about such next generation software which we have not had time to even mention. For instance, the issue of how to build fault tolerance into such a set of library routines, or assembling contracts which attempt to monitor the progress of a user's task and act if necessary. The general problem of redistributing the user's data is also important.
28
In closing, we have attempted to define the target system through identifying a set of criterion relevant to the numerical library routines we are investigating. Homogeneity is an important criterion and as has been discussed, cannot be taken for granted. We have denned our target user and provided an interface for this user into the middleware which has been designed to utilize existing, scalable library routines. We have reserved a detailed conversation for scheduling in shared, homogeneous system for another time. This subject is of great importance however and cannot be avoided in practice. It is possible, if not likely, that techniques from time sensitive statistical investigations will play a major role in coming to terms with large scale systems -especially as they only become larger. Motivation to further pursue such approaches to numerical libraries on shared, homogeneous systems was provided. We are proposing a more thorough study at this point which attempts to ferret out many of the outstanding mysteries. References 1. See documentation on IBM's LoadLeveler batch system. For instance http :: //uagibm.nerscgov/docs/LoadL/index.html provides documentation of a specific implementation on NERSC's IBM SP/RS6000, currently known as seaborg.nersc.gov. 2. Garey, M. and Johnson, D.,Computers and Intractability, a Guide to the Theory of NP-Completeness,Bel\ Telephone Laboratories, 1979 3. Hopcroft, J. E. and Ullman, J. D.,Introduction to Automata Theory, Languages, and Computation, Addison-Wesley, 1979 4. Ullman, J. T).,"NP-complete scheduling problems,",Journal of Computer and Systems Sciences 10:3, pp. 384-393 5. Knuth, D. E.,Fundamental Algorithms, volume 1 of The Art of Computer Programming, Addison-Wesley, 1968 6. Knuth, D. E.,Seminumerical Algorithms, volume 2 of The Art of Computer Programming, Addison-Wesley, 1969 7. Knuth, D. E.,Sorting and Searching, volume 3 of The Art of Computer Programming, Addison-Wesley, 1973 8. Cormen, T. H., Leiserson, C. E, Rivest, R. L., Stein, C,Introduction to Algorithms, 2nd edition, MIT Press, 2001 9. NWS, The Network Weather Service, http://nws.cs.ucsb.edu/ 10. Blackford, L. S., Choi, J., Cleary, A., D'Azevedo, E., Demmel, J., Dhillon, I., Dongarra, J., Hammarling, S., Henry, G., Petitet, A., Stanley, K., Walker, D., Whaley, R. C.,ScaLAPACK Users' Guide, SIAM, 1997 11. Berman, F., Chien, A., Cooper, K., Dongarra, J., Foster, I., Gannon, D.,
29
12.
13.
14. 15.
16.
17.
18. 19. 20. 21. 22. 23. 24.
25.
Johnsson, L., Kennedy, K., Kesselman, C , Mellor-Crummey, J., Reed, D., Torczon, L., Wolski, R.,"The GrADS Project: Software Support for High-Level Grid Application Development," 2001, Rice University, Houston, Texas; see also http://www.hipersoft.rice.edu/grads Foster, I. and Kesselman, C,"GLOBUS: A metacomputing infrastructure toolkit ^International Journal of High Performance Computing Applications, vol. 1 1 , pp. 115-128 Petitet, A., Blackford, S., Dongarra, J., Ellis, B., Fagg, G., Roche, K., Vadhiyar, S.,"Numerical Libraries and the Grid, "International Journal of High Performance Computing Applications, vol. 15, pp. 359-374 Foster, I. and Kesselman, C,The Grid: Blueprint for a New Computing Infrastructure, Morgan Kaufmann, San Francisco, 1999 Petitet, A., Blackford, S., Dongarra, J., Ellis, B., Fagg, G., Roche, K., Vadhiyar, S.,Numerical Libraries and the Grid: The GrADS Experiments with ScaLAPACK, Computer Science Department Technical Report UTCS-01-460, University of Tennessee, Knoxville, Tennessee 37996-3450 Boulet, P., Dongarra, J., Rastello, F., Robert, Y., Vivien, F.,"Algorithmic issues on heterogenous computing platforms," Parallel Processing Letters, 9:2, pp. 197-213, 1999 Kalinov, A. and Lastovetsky, A.,"Heterogeneous Distribution of Computations While Solving Linear Algebra Problems on Networks of Heterogenous Computers," Journal of Parallel and Distributed Computing, 6 1 , 4, pp. 520-535, 2001 Oksendal, B.,Stochastic Differential Equations, 3rd edition, SpringerVerlag (Berlin), 1992 Bremaud, P.,Point Processes and Queues, Martingale Dynamics, Springer-Verlag (New York), p.181 , 1981 Lipster, R. and Shiryayev, A.,Statistics of Random Processes: General Theory, Springer-Verlag (New York), 1977 Lipster, R. and Shiryayev, A..Statistics of Random Processes: Applications, Springer-Verlag (New York), 1978 PAP I, Performance API, http://icl.cs.utk.edu/projects/papi IBP, Internet Backplane Protocol, http://loci.cs.utk.edu/ibp Whaley, R. C,Basic linear algebra communication subprograms: Analysis and implementation across multiple parallel architectures, Computer Science Department Technical Report UT-CS-94-234, University of Tennessee, Knoxville, Tennessee 37996-3450 Whaley, R. C , Petitet, A., Dongarra, J. J., "Automated empirical optimizations of software and the ATLAS project," Parallel Computing, 27, 1,2, pp.3-35, 2001
30
26. Peterson, L. L. and Davie, B. S., Computer Networks: A Systems Approach, Morgan Kaufmann (San Francisco), 1996 27. Stevens, W. R., Unix Network Programming, Prentice-Hall, 1990 28. Balay, S., Gropp, W., Curfman Mclnnes, L., Smith, B.,PETSc Users Manual, rev. 2.1.0, Argonne National Laboratory, 9700 South Cass Avenue, Argonne, Illinois 60439 29. Golub, G. H. and Van Loan, C. F.,Afatrix Computations, 3rd edition, John's Hopkins University Press, 1996 30. Choi, J., Dongarra, J., Ostrouchov, S., Petitet, A., Walker, D., Whaley, R. C.,A Proposal for a Set of Parallel Basic Linear Algebra Subprograms, Computer Science Department Technical Report UT-CS-95-292, University of Tennessee, Knoxville, Tennessee 37996-3450 31. HPL, High Performance Linpack benchmaxk,http://www. netlib. org/benchmark/hpl 32. Dongarra, J., Du Croz, J., Duff, I. S., Hammarling, S.,"A Set of Level 3 Basic Linear Algebra Subprograms," ACM Trans. Math. Soft., 14, pp. 1-17, 1988 33. Lichtenstein and Johnson, "Block-cyclic dense linear algebra," SI AM J. Sci. Stat. Compt., 14, pp.1259-1288,1993 34. Petitet, A. Algorithmic Redistribution Methods for Block Cyclic Decompositions, PhD thesis, Department of Computer Science, University of Tennessee, Knoxville, Tennessee 37996-3450 35. Beaumont, O., Legrand, A., Rastello, F., Robert, Y., "Dense linear algebra kernels on heterogeneous platforms: Redistribution issues,", Parallel Computing, 28, (issue 2), pp. 155-185, 2002 36. Kernighan, B. W. and Ritchie, D. M.,The C Programming Language, 2nd edition,Bei\ Telephone Laboratories, Inc., 1988 37. Snir, M. , Otto, S. W., Huss-Lederman S., Walker, D. W., and Dongarra, J. 3.,MPI: The Complete #e/erence,MIT Press,1996
GRID: Earth and Space Science Applications Perspective Luigi Fusco Senior Advisor for EO Applications ESA - ESRIN, 00044 Frascati, Italy
[email protected] The integration of space and information technologies is an asset for many science and operational applications in various disciplines. For example, limiting to the case of Earth Observation, many environmental satellites have been and are operated to provide large and repetitive volumes of geophysical measurements for natural resource management and environmental monitoring of our planet, which require sophisticated computing resources. The worldwide scientific user community is relatively large, in the order of many thousands, and the International programmes (e.g. IGBP, ICSU, WCRP, IOC) are the key consumers of EO data and its derived information. At the same time operational institutional and commercial users have real time and off-line access to data and services. At present, the provision of user information services is far from being optimal, due to the complexity of product format, algorithms and processing required to meet the specific user needs (e.g. immediate access to integrated EO and other information, for the specific application, in the end user system environment). To meet the user communities' aims it is required the effective co-operation of all involved actors and the sharing of experience, methods, resources, etc. Many obstacles have to be removed, for example: • access to large data archives (which today are mainly in close operational environment); • common definition of metadata and data formats (interoperability across various data providers); • access to relevant ancillary information; • adequate network capacity across space data handling dedicated facilities, value adding, science and service industries; • access to adequate computing resources; • development of real time services; • development of effective user tools for integrating different data sets; • agreements on common or compatible data policies for data access. The GRID technology (in terms of network, distributed and high performing computing power, collaborative environment and large data archive resources) can and should help
31
32 the process of supporting the user community in their scientific and operational applications. The European Space Agency' has plans for handling the dedicated archives, operational and future missions, and to support their scientific, operational and commercial exploitation. Efforts in demonstrating the potential of the GRID technology elements and services (e.g. middleware for distributed processing and access to distributed archives) are on going. The plans to encourage a European strong presence in this domain are essential actions to provide the necessary support and infrastructure as required by complex environmental applications.
e-Science, e-Business and the Grid
Tony Hey University of Southampton, UK
The talk begins with a rapid survey of the technological drivers behind the IT revolution. Increasingly, many areas of science involve access to distributed computing and data resources, remote access to specialized and expensive facilities and world-wide collaborations of scientists. There are many examples of such 'e-Science' endeavours ranging from bioinformatics and proteomics to collaborative engineering and earth observation. To enable scientists to maximize the science derived in such a fashion we will see the emergence of a new IT infrastructure called the Grid. The Web gave us easy access to html pages and information: the Grid will give us seamless access to a much wider range of distributed resources. It will enable us to form transient 'Virtual Organisations' without compromising security or privacy. The Grid infrastructure developed to allow interoperability and scalability of such heterogeneous and dynamic resources has obvious interest for industry. The talk concludes with some examples of Grid technology in an industrial context. The UK e-Science Programme is also briefly described.
33
G R A P H PARTITIONING FOR D Y N A M I C , A D A P T I V E A N D MULTI-PHASE SCIENTIFIC SIMULATIONS KIRK SCHLOEGEL, GEORGE ~KARYPIS, AND VIPIN KUMAR Dept. of Computer Science and Engineering University of Minnesota E-mail: kirk, karypis, kumarQcs.umn.edu The efficient execution of scientific simulations on HPC systems requires a partitioning of the underlying mesh among the processors such that the load is balanced and the inter-processor communication is minimized. Graph partitioning algorithms have been applied with much success for this purpose. However, the parallelization of multi-phase and multi-physics computations poses new challenges that require fundamental advances in graph partitioning technology. In addition, most existing graph partitioning algorithms are not suited for the newer heterogeneous high-performance computing platforms. This talk will describe research efforts in our group that are focused on developing novel multi-constraint and multi-objective graph partitioning algorithms that can support the advancing state-of-the-art in numerical simulation technologies. In addition, we will present our preliminary work on new partitioning algorithms that are well suited for heterogeneous architectures.
1
Introduction
Algorithms that find good partitionings of unstructured and irregular graphs are critical for the efficient execution of scientific simulations on highperformance parallel computers. In these simulations, computation is performed iteratively on each element (and/or node) of a physical two- or threedimensional mesh. Information is then exchanged between adjacent mesh elements. Efficient execution of such simulations on distributed-memory machines requires a mapping of the computational mesh onto the processors that equalizes the number of mesh elements assigned to each processor and minimizes the interprocessor communication required to perform the information exchange between adjacent elements 1 0 . Such a mapping is commonly found by solving a graph partitioning problem 3 ' 4 . Simulations performed on shared-memory multiprocessors also benefit from partitioning, as this increases data locality, and so, leads to better cache performance. Although the graph partitioning problem is NP-complete, good heuristic solutions for instances arising in scientific simulation can be found using multilevel algorithms. Many of these algorithms are incorporated in software packages such as Chaco 2 , METIS 6 , and SCOTCH 7 . However, the parallelization of multi-phase and multi-physics computa-
34
35
tions poses new challenges that require fundamental advances in graph partitioning technology. In addition, most existing graph partitioning algorithms are not suited for the newer heterogeneous high-performance computing platforms. This talk will describe research efforts in our group that are focused on developing novel multi-constraint and multi-objective graph partitioning algorithms that can support the advancing state-of-the-art in numerical simulation technologies. In addition, we will present our preliminary work on new partitioning algorithms that are well suited for heterogeneous architectures. 2
Multi-Constraint and Multi-Objective Partitioning Algorithms
The traditional graph-partitioning problem is a single objective optimization problem subject to a single balance constraint. Our research has focused on generalizing this problem to allow for multiple optimization objectives as well as multiple balance constraints by assigning weight vectors to the vertices and edges of the graph 5 ' 8 . The resulting generalized algorithms have enabled effective partitioning for a variety of applications such as weapon-target interaction simulations involving multiple computational meshes and particlein-cells computations that underlie diesel combustion engine simulations. As an example, consider a multi-physics simulation in which a variety of materials and/or processes are simulated together. The result is a class of problems in which the computation as well as the memory requirements are not uniform across the mesh. Existing partitioning schemes can be used to divide the mesh among the processors such that either the amount of computation or the amount of memory required is balanced across the processors. However, they cannot be used to compute a partitioning that simultaneously balances both of these quantities. Our inability to do so can either lead to significant computational imbalances, limiting efficiency, or significant memory imbalances, limiting the size of problems that can be solved using parallel computers. Figure 1 illustrates this problem. It shows three possible partitionings of a graph in which the amount of computation and memory associated with a vertex can be different throughout the graph. The partitioning in Figure 1(b) balances the computation among the subdomains, but creates a serious imbalance for memory requirements. The partitioning in Figure 1(c) balances the memory requirement, while leaving the computation unbalanced. The partitioning in Figure 1(d), that balances both of these, is the desired solution. In general, multi-physics simulations require the partitioning to satisfy not just one, but a multiple number of balance constraints. (In this case, the partitioning must balance two constraints, computation and memory).
36
Figure 1. An example of a computation with nonuniform memory requirements. Each vertex in the graph is split into two amounts. The size of the lightly-shaded portion represents the amount of computation associated with the vertex, while the size of the dark portion represents the amount of memory associated with the vertex. The bisection in (b) balances the computation. The bisection in (c) balances the memory, but only the bisection in (d) balances both of these.
Other examples of scientific simulations that require partitionings with multiple constraints and multiple objectives are presented in s , 1 ° . Serial and parallel algorithm for solving such multi-objective, multi-constraint graph partitioning problems are described in 5 ' 8 . 9 > n . New Challenges As numerical simulation technologies continue to become more sophisticated and as the number of processors routinely used increases into the thousands and tens of thousands, partitionings are required to satisfy more and more generalized constraints and optimize many different types of objectives in order to ensure good parallel efficiencies. Many of these objectives cannot be defined on the vertices and the edges of the graph, but are instead defined in terms of the overall structure of the partitioning. As an example, many types of the parallel indirect solvers require partitionings that, in addition to minimizing the inter-processor communication, are also composed of well-shaped subdomains (i.e., the subdomains have good aspect ratios) x. As another example, domain-decomposition-based numerical simulations, such as those proposed for computational structural mechanics, require that the resulting partition-
37
ing simultaneously balances (i) the amount of time required to factorize the local subproblem using direct factorization, (ii) the size of the interface problem assigned to each processor, and (iii) the number of subdomains to which each subdomain is adjacent. In both of these examples, the various objectives and constraints cannot be modeled by assigning appropriate weights to the vertices and/or edges of the graph, as they depend on the structure of the partitioning. Developing algorithms for solving such problems is particularly challenging, as it requires that the partitioning algorithm balance quantities that can be measured only after a partitioning has been computed. Also, in many time dependent computations the physics or subdomain meshes change as a function of time. For such computations to remain load balanced, the mesh must be redistributed periodically. This requires an adaptive repartitioning algorithm that has yet an additional optimization objective on top of any others specified by the user (i.e., the minimization of the amount of data that needs to be redistributed during load balancing). In our research, we are continuing to develop a general partitioning framework that allows constraints and objectives to be specified in terms of the structure of the desired partitioning as well as the development of algorithms suitable for this framework. 3
Partitioning for Heterogeneous Computing Platforms
Most existing scalable parallel multi-disciplinary simulation codes can be easily ported to a wide range of parallel architectures as they employ a standard messaging layer such as MPI. However, their performance on these depends to a large degree on the architecture of the parallel platform. In particular, many core parallel algorithms were designed under the assumption that the target architecture is flat and homogeneous. However, the emergence of parallel computing platforms built using commercial-off-the-shelf components has resulted in high-performance machines becoming more and more heterogeneous. This trend is also influenced by the geographically distributed nature of computing grids as well as the effects of increasingly complex memory hierarchies. This heterogeneity presents real challenges to the scalable execution of scientific and engineering simulation codes. A promising approach for addressing this problem is to develop a new class of architecture-aware graph partitioning algorithms that optimally decomposes computations given the architecture of the parallel platform. Ideally, such as intelligent partitioning capability could alleviate the need for major restructuring of scientific codes. We are in the process of developing graph-partitioning algorithms that take into account the heterogeneity of the underlying parallel computing ar-
38
chitecture, and hence, compute partitionings that will allow existing scientific codes to achieve the highest levels of performance on a wide range of platforms. The intent is to develop an extensible hierarchical framework for describing the various aspects of the target platform that captures the underlying network topologies, inter-connection network bandwidths and latencies, processor speeds, memory capacities, and the various levels of the memory hierarchy. New graph-partitioning algorithms will then be designed that can use information from this framework to optimize partitionings with respect to the specified architecture. Acknowledgments This work was supported by DOE contract number LLNL B347881, by NSF grants CCR-9972519, EIA-9986042, and ACI-9982274, by Army Research Office contracts DA/DAAG55-98-1-0014, by Army High Performance Computing Research Center cooperative agreement number DAAD19-01-2-0014, the content of which does not necessarily reflect the position or the policy of the government, and no official endorsement should be inferred. Additional support was provided by the IBM Partnership Award, and by the IBM SUR equipment grant. Access to computing facilities was provided by AHPCRC and the Minnesota Supercomputer Institute. References [1] R. Diekmann, R. Preis, F. Schlimbach and C. Walshaw. Shape-Optimized Mesh Partitioning and Load Balancing for Parallel Adaptive FEM. Parallel Computing 26, 12 (2000). [2] B. Hendrickson and R. Leland. The Chaco User's Guide, Version 2.0. Technical Report Sandia National Laboratories SAND94-2692, 1994. [3] B. Hendrickson and R. Leland. A Multilevel Algorithm for Partitioning Graphs. Proceedings Supercomputing '95, 1995. [4] G. Karypis and V. Kumar. A Fast and High Quality Multilevel Scheme for Partitioning Irregular Graphs. SIAM Journal on Scientific Computing 20, 1 (1998). [5] G. Karypis and V. Kumar. Multilevel Algorithms for Multi-constraint Graph Partitioning. Proceedings Supercomputing '98, 1998. [6] G. Karypis, K. Schloegel and V. Kumar. ParMETIS: Parallel Graph Partitioning and Sparse Matrix Ordering Library, Version 3.0, 2002. [7] F. Pellegrini and J. Roman. Scotch: A Software Package for Static Mapping by Dual Recursive Bipartitioning of Process and Architecture
39
Graphs. HPCN-Europe, Springer LNCS 1067, 1996. [8] K. Schloegel, G. Karypis and V. Kumar. A New Algorithm for Multiobjective Graph Partitioning. Proceedings EuroPar '99, pages 322-331, 1999. [9] K. Schloegel, G. Karypis and V. Kumar. Parallel Multilevel Algorithms for Multi-constraint Graph Partitioning. Proceedings EuroPar2000, pages 296-310, 2000. [10] K. Schloegel, G. Karypis and V. Kumar. Graph Partitioning for High Performance Scientific Simulations. In CRPC Parallel Computing Handbook, Morgan Kaufmann. To appear. [11] K. Schloegel, G. Karypis and V. Kumar. Parallel Static and Dynamic Multi-constraint Graph Partitioning. Concurrency: Practice and Experience, To appear.
Panel Session on Challenges and Opportunities in Data -Intensive Grid Computing Paul Messina Center for Advanced Computing Research, CalTech
[email protected] The panel will discuss software and hardware technology issues related to very large, geographically distributed scientific data sets: archiving, accessing, and analyzing them. Discussion topics would include distributed, on-demand computing for analysis, federation of databases for simultaneous access to various databases, data mining, network middleware, scalability, and distributed visualization, among others.
40
APPLICATIONS
This page is intentionally left blank
G I A N T EIGENPROBLEMS F R O M LATTICE G A U G E THEORY ON CRAY T3E SYSTEMS N. ATTIG 1 , TH. LIPPERT 2 , H. NEFF 1 , J. NEGELE 3 , AND K. SCHILLING 1 ' 2 John von Neumann Institute for Computing Research Center Mlich, 52425 Jiilich, Germany 2 Dept. of Physics, University of Wuppertal, 42097 Wuppertal, Germany 3 Center for Theoretical Physics, MIT, Cambridge, USA The determination of physical properties of flavor singlet objects like the if meson by computer simulation requires the computation of functionals of the inverse fermionic matrix A/ - 1 . So far, only stochastic methods could cope with the enormous size of M. In this paper, we introduce an alternative approach which is based on the computation of a subset of low-lying eigenmodes of the fermionic matrix. The high quality of this 'truncated eigenmode approximation' (TEA) is demonstrated by comparison with the pion correlator, a flavor octet quantity, which is readily computable through a linear system of equations. We show that TEA can sue- cessfully approximate the flavor singlet rf correlator. We find that the systematic error of the method is tolerable. As the determination of the chosen subset of 300 eigenmodes requires about 3.5 Tflops-hours CPU-time per canonical ensemble and at least 15 GBytes of memory, the power of high-end supercomputers like the CRAY T3E is indispensable.
1
Introduction
A major goal of non-perturbative lattice quantum chromodynamics (LQCD) is the determination of hadronic mass states which are characterized by nonvalence contributions, such as flavor singlet mesons. Their correlation functions, CV'(ti - t 2 ) , contain so-called 'disconnected diagrams', i.e. correlators between closed virtual fermion loops. The reliable determination of these disconnected diagrams has been a long- standing issue ever since the early days of lattice gauge theory. It can be reduced to the numerical problem of how to achieve information about functionals of the inverse fermionic matrix M _ 1 . The first attempts in this direction have been started only a few years ago, using the so-called stochastic estimator method (SE) * to compute the trace of M _ 1 . This approach requires to solve the linear system Mx = £ on some hundred source vectors £, with £ being Z2 or Gaussian noise vectors. Meanwhile, substantial progress could be achieved for the determination of the T]1 by application of refined smearing methods 2 , where for the first time a proper signal-to-noise-ratio could be established. However, SE introduces stochastic noise, in addition to the stochastics already inherited from the
43
44
Monte Carlo process. In the following, we describe the determination of the 77' mass based on the computation of a set of low-lying eigenmodes of Q = 75 M, the hermitian form of M. We use the implicitly restarted Arnoldi method, a generalization of the standard Lanczos procedure. A crucial ingredient is the Chebyshev acceleration technique to achieve a transformation of the spectrum to a form suitable for the Arnoldi eigenvalue determination. The low-lying modes given, it is possible to estimate the entire matrix Q~l and those matrix functionals or functions of Q and M which are sensitive to long-range physics. In section 1, we introduce the meson correlators and in section 2, we shortly review their computation by conventional means. Section 3 is devoted to TEA and the organization of the computation on the CRAY T3E by use of the parallel Arnoldi package (PARPACK). In section 4, we assess the viability of TEA by comparing the correlator of the TT meson as computed from TEA with the result from the conventional approach. As the IT is a flavor octet quantity it can easily be computed through the solution of a linear system of equations by iterative Krylov subspace algorithms 3 . Finally, we apply TEA to the computation of the 77' meson correlator and compare with results from SE computations. 2
Meson Correlators
In LQCD, hadronic masses are extracted from the large-time behavior of correlation functions. The correlator of the flavor octet n meson is defined as C»(t = t 1 - t 2 ) = / ^ T r [ g - 1 ( n 1 t 1 ; T n , * a ) g - 1 ( m l t 2 ; n > t i ) ] \ \n,m
,
(1)
I y
while the flavor singlet 77' meson correlator is composed of two terms, one being connected and equivalent to the pion correlator, the second being the disconnected contribution from the correlation of virtual quark loops: C„-(ti- z
Determine the time step At (eq. 3); Partition the whole rectangular domain D=[Ni x Nj x N J into P= P; x Pj x Pk rectangular subdomains D'= [N'i x N'j x N ' J , being Pi( Pj, and P k the number of processors along dimension ij and k, and N. N; Nk N; = — - , N : = and N ^ = the dimension (expressed as number of "i
"j
"k
grid points) of the generic subdomain D'. for^OjKT^t^+At) in all the processors do compute the new values of H communicate the H values on the boundary of each subdomain to its neighbor compute the new values of E enddo in all put the correct value in the feed point in the processors containing the source;
80
in the boundary processors do compute the absorbing boundary conditions; in all the processors communicate die E values on the boundary of each subdomain to its neighbor; endfor end FD-TD algorithm. When implementing me algorithm, particular care has been put in the vectorization of memory accesses (to reduce the severe memory start-up penalty), the unrolling of critical loops (to minimize pipeline stalls and die waiting for the consequent initialization latency), the using of the fast internal multiport register file.
5
The Numerical Phantom
A relevant role in the accurate solution of the addressed problem is played by die numerical technique utilized to represent die electromagnetic properties of die exposed human subject. The use of an appropriate numerical phantom, in feet, is the attractive issue to overcome die real problem of several proposed experimental techniques, using homogenous representations of die human body, with a consequent approximation error not adequate to die goals of a radio-protection analysis. The story of die development of accurate numerical phantoms is rich and long, and we address to die specialized literature me interested reader [14 and references tiierein]. Here we recall diat a generical numerical phantom is an organized archive to build up a detailed millimeter-resolution human model so diat die dielectric and conductive properties are fixed for each working frequency in each part of the sample body. In mis work we refer to one of die most appreciated phantoms, die one proposed by die Visible Human Project at Yale University, whose development is supported by [15]. It is also important referring to [16] for a sensitivity analysis of die predicted values witii respect to frequency and voxel size. 6
Results
Fig. 1 report results for a human phantom exposed to a real radiobase antenna (Katiirein 730678). The distance between die operator and die antenna is around 60 cm. The domain is partitioned into 320x320x256 cubic cells, witii a 4 mm edge. The operating frequency is 902 MHz, for die well known GSM cellular system. At this frequency, die wavelength in vacuo is 33.2 cm. As die maximum relative dielectric constant in the phantom is around 60, die use of a 4mm cubic edge is in accordance witii a standard convergence criterion to be used witii FDTD codes, requiring a
81
minimum ratio of 10 between the wavelength and the cell's edge. Nonetheless, it is quite obvious that an adaptive meshing is nearly mandatory, and is currently under development
50
100
150
200
250
50
100
150
200
250
Fig la,b: Human phantom exposure to a real radiobase antenna in two different configurations. Red colour is for high-intensity E fields, blue is for low amplitudes.
As for computing times and performance, for the sake of brevity, we omit here details: an efficiency TJ=—2-- — = 6.68E-01 is achieved for a 26214400 cell tpar
P
domain, thus proving that the FD-TD integration scheme is very well suited to be implemented on massively parallel systems. The accuracy in the solution of the problem allows the correct evaluation of E and H fields wherevere inside the human phantom, as depicted in Fig. la,b. It is just worth noting here that the maximum observed percentage of absorbed E field inside the phantom is around 40 % in the nearby of the chin. 7
Conclusions
In this paper we have proposed an FDTD implementation on massively parallel systems for the evaluation of radiofrequency hazards in humans exposed to the nearfield of radiobase station antennas. The approach is proved to be appropriate, and the FDTD strategy extremely amenable to such a parallel implementation. E and H fields are accurately evaluated inside a numerical phantom, thus solving in a
82 rigorous and efficient fashion a relevant problem in a wide class of real industrial applications.
References 1 2 3
4
5
6 7 8
9
10 11
12 13 14 15 16
G. Wolfle, R. Hoppe, F. M. Landstorfer: 'Radio network planning with ray-optical propagation models for urban, indoor and hybrid scenarios', Proc. of Wireless 99, 1999. G. Wolfle, F. M. Landstorfer: 'Prediction of the field strength inside buildings with empirical, nerual and ray-optical prediction models', COST 259 TD 98-008. R. Hoppe, G. Wolfle, F. M. Landstorfer: 'Fast 3D Ray-tracing for the planning of microcells by intelligent preprocessing of the database', Proc. of COST 259 Workshop, 1998. G. Wolfle, A. J. Rohatscheck, H. Korner, F. M. Landstorfer: 'High resolution measurement equipment for the determination of channel impulse responses for indoor mobile communications', Proc. of PIERS 98,1998. J. Zimmermann, R. Hons, H. Mulhenbein: "The antenna placement problem for mobile radionetworks: an evolutionary approach', Proc. 8th Conf. Tel. Systems, pp. 358-364, 2000. A. Bahr, D. Manteuffel, D. Heberling: 'Occupational safety in the near field of a GSM base station', Proc. of AP2000, Davos, April 2000, 3A6.7. J.D. Kraus: 'Antennas'. Mc Graw Hill - 1988. K. S. Yee: 'Numerical solution of initial boundary value problems involving Maxwell's equations in isotropic media', IEEE Transactions on Antennas and Propagation, AP-14, 4, pp. 302-307, 1996. A. Taflove and M.E. Brodwin: 'Numerical solution of steady-state electromagnetic scattering problems using the time-dependent Maxwell's equations', IEEE Transactions on Microwave Theory and Techniques, MTT-23, 8, pp. 623-630,1975. A. Taflove: 'Computational Electrodynamics: The Finite-Difference Time-Domain Method', Norwood, MA, Artech House, 1995. G.Mur: 'Absorbing boundary conditions for the Finite-Difference approximation of the Time-Domain Electromagnetic-Field equations', IEEE Transactions on Electromagnetic Compability, vol. EMC-23,4, pp. 377-382, 1981. J.-P. Berenger: 'A perfectly matched layer for the absorption of electromagnetic waves', J. Computational Physics, vol. 114, 1994. J.-P. Berenger: 'A perfectly matched layer for the FD-TD solution of wave-structure interaction problems', IEEE Antennas Propag. Symp., vol. 44, n.l, 1996. C. Gabriel, S. Gabriel, E. Corthout, "The dielectric properties of biological tissues: I Literature Survey', Phys. Med. Biol., vol. 41, pp.2231-2249, 1996. C. Gabriel, 'Compilation of the Dielectric Properties of Body Tissues at RF and Microwave Frequancies', Brooks Air Force Technical Report AL/OE-TR-1996-0037. P. A. Mason et al., 'Effects of Frequency, Permittivity and Voxel Size on Predicted SAR values in biological tissues during EMF Exposure', IEEE Transactions on Microwave Theory and Techniques, MTT-48, 11, pp. 2050-2061, 2000.
Tranquillity Mapping Using a Network of Heterogeneous PC Andrea Clematis, Monica De Martino, Giulia Alessio Istituto per la Matematica Applicata - Consiglio Nazionale delle Ricerche Via De Marini 6, 16149 Genova, Italy Sara Bird, Susanna Feltri Amministrazione Provinciate di Savona ViaSormano 6, 17100 Savona, Italy
The use of parallel processing to speed-up Geographic Information System (GIS) algorithms and applications has been widely considered and documented. In this paper we present our experience and early results about the use of parallel computing to speed-up tranquillity mapping, a methodology which is finalised to provide support to the landscape assessment process, and which is of interest for public administrations. The parallelisation of tranquillity mapping algorithm is described. Since the final goal is to make it possible to use the parallel algorithm in a public administration, aspects related with the use of an heterogeneous network of Personal Computers are addressed.
1 Introduction The usefulness of parallel processing for Geographic Information System (GIS) applications is widely documented in several reports and research papers (Healey98). Experiences are available about the development of parallel algorithms which improve the performances of different costly GIS functions like raster to vector data conversion, Digital Terrain Model (DTM) and Triangulated Irregular Network (TIN) construction, Drainage Basins identification, morphological analysis and other. In this paper we are interested in the use of GIS for tranquillity mapping (Bell99), a process aimed to produce a thematic map, which shows the levels of disturbance or tranquillity of a region. This process may be considered as a part of landscape assessment (Cudlip99), and it is becoming of interest for public administration as well as for other subjects like tourist operators or estate agents. The reported experience is part of the PLAINS project (Prototype Landscape Assessment Information System), a research project sponsored by European Union as a Centre for Earth Observation (CEO) activity. The parallelisation of the tranquillity mapping process for an heterogeneous network of PC is described. 2 Tranquillity mapping Tranquillity mapping concerns the idea of mapping areas of landscape that could be regarded as tranquil: away from noise and visual intrusion. This concept was introduced and developed by the ASH Consulting Group in England in 1991, first
83
84
for a study for the Department of Transport, and then for the Countryside Commission and Council for the Protection of Rural England (Bell 99). If a scale or spectrum of tranquillity is drawn up, a "complete" tranquillity would correspond to the absence of unnatural noise and visual intrusion, while at the other end an absolute lack of tranquillity would exist where noise is greatest and/or visual intrusion is significant. A degree of tranquillity depends on the proportion and combination of effects between these two extremes. Moreover it is necessary to consider diat die terrain morphology may affect or improve tranquillity. Here a tranquillity mapping methodology is proposed which takes into account the effect of the presence of disturbance sources. The methodology is characterised by two main activities: 1. 2.
Input data identification and acquisition; Spatial and attribute data processing for tranquillity map generation.
Parallel computing will be used to speed-up the tranquillity map generation step. 2.1 Input data identification and acquisition In our approach the basic input data is the vector cartography of the investigated region. The region we have considered is part of Liguria in North- West of Italy, and it is characterised by a complex terrain morphology. For this reason we adopted a 1:5000 scale for the digital cartography. For other region with a more regular terrain morphology a smaller scale could be adopted. The other input data are disturbance values, represented by attributes attached to each geo-referenced object, which is a source of disturbance. In the present version of our Tranquillity mapping process we have considered factors concerning noise and visual intrusion generated by three categories of unnatural sources, which are road and railroad networks, industrial sites, and urban centres. Each object belonging to one of these categories must be contained in the spatial database as a single feature or as a set of features according wim the appropriate geometry: linear features for road and railroad networks and area features for industrial and urban centres. A disturbance level is attached to each single feature. The whole set of features of a category is identified as a theme. We have defined 8 disturbance levels: 1 is for highly negative influence and 8 is low influence. The definition of a tranquillity spectrum extends the effect of the disturbance level to the influenced area using an appropriate set of distance intervals to identify different areas ranging from very disturbed to very tranquil or remote. See (AlessioOl) for a table, which defines a tranquillity spectrum schema.
85
2.2 Spatial and attribute data processing for tranquillity map generation This activity may be split in two phases: • Calculus of tranquillity map for each theme or category of features • Generation of a final map as a synthesis of all the generated tranquillity maps. For each theme, tranquillity mapping process is characterised by the following steps: • Group features in distinct classes in accordance with their disturbance level; • For each class generate a buffer layer using the table of distance range; • Combine buffer layers generated by different classes of disturbance; • Map optimisation. Considering that we group features according with the disturbance levels, which are eighth, we generate a map with eight buffer layers. The buffer layers are then analysed and synthesised: buffers are combined to generate a unique map by using overlay analysis operation. The resulting map is characterised by a lot of polygons obtained by combining all buffers in such a way that the minimum tranquillity scale (corresponding to a maximum disturbance) of the combined buffers is associated to polygons. The tranquillity map is optimised, by merging adjacent areas with the same tranquillity scale value, in order to reduce the number of distinct polygons. The tranquillity maps derived from the different themes are finally merged together by overlay and a final optimisation is done. xsaaEssa^H^mmmmmmm^m^^^mmmmmmmmmmi^^^^m^m
Figure 1: A tranquillity map
86
Figure 1 shows the tranquillity map for the Letimbro drainage basin in Savona District ( Liguria Region, North West of Italy). 3 Parallelising TM: outline of a strategy for an heterogeneous environment The tranquillity mapping algorithm exhibits an high computational cost and its execution time on an unloaded workstation equipped with a professional GIS environment is of about 12 hours for an area of 400 Km2 characterised by a complex morphology and by a medium level density of disturbance features. Thus to speed-up the mapping algorithm will lead to important improvements especially for simulation purposes. 3.1 The Parallel Program Outline The Tranquillity Mapping algorithm permits to process in an independent way tasks, which may be obtained by partitioning the data domain using a logical approach (different themes) or a spatial approach (different sub-regions) or a mix of the two. To put together the results of the processing of two or more independent tasks requires to properly overlay maps, if the tasks correspond to different themes of the same sub-region, or join maps if tasks correspond to neighbour sub-regions and finally to execute a map optimisation. Note that the overlay operation is the same used in the sequential algorithm, while join does not present particular problems. In the present implementation, if a feature lies over a boundary between two sub-regions it is assigned to one of the two regions. This is sufficient to ensure the correctness of the parallel algorithm. Figure 2 provides an outline of the parallel program for tranquillity mapping. The parallel algorithm is based on master worker paradigm and it is divided in three distinct phases. In the first phase the master provides to data distribution and schedules a task for each worker. In the second phase each worker processes tasks and it notifies the master after each task completion. Results of computation are kept in local database of each worker during this phase. When all tasks have been completed the third phase, dedicated to overlay and join operation may start. These operations are executed as much as possible in parallel, trying also to minimise data transfer among worker. A tree like approach is used to overlay themes of each subregion, and to join maps of different sub-regions. The number of independent tasks progressively decreases during this phase of computation, and workers will leave computation following master instructions. 3.2 Load Balancing and data partitioning Load balancing is the key point to make the parallel algorithm efficient, and to obtain load balancing we have to consider the computational cost of each task,
87
Phase 1: data distribution and start up Master: { copy dataset to each worker local disk; schedule the first set of tasks; send TaskActivation message to each worker; }
Phase 2: Single task processing Master: while WorkToDo { receive TaskDone message; schedule NextTask; send TaskActivation message; } Worker: while WorkToDo { receive TaskActivation message; compute task; send TaskDone message; }
Phase 3: Maps overlay and join Master: { co-ordinate workers to properly overlay thematic maps for each drainage basin; co-ordinate workers to properly join drainage basin maps; accept and merge final dataset } Worker: { receive information from master about WhatToDo; select WhatToDo: Overlay: overlay themes and optimise map following master directives; Join: join and optimise regions following master directives; AcceptData: merge in your local database data provided by another worker; ProvideData: copy your local database to indicated worker location; GiveUp: exit computation }
Figure 2: Parallel tranquillity mapping which depends on the number of features contained in the task, and for tasks with a similar number of features on their density. In fact a higher density of features will lead to an higher fragmentation for the intermediate maps, thus making the optimisation phase more costly. Domain partitioning is primary obtained using the different themes. Since the different themes contain a different number of features we have to find a way to further subdivide more populated themes in appropriate
88
sub-themes. In the considered case this has been done by splitting the transportation infrastructure theme into four sub-themes, namely rail-roads, highways, primary roads and secondary roads. In mis way we obtained a limited number (six in the considered case) of independent tasks with a similar computational cost. To have a small number of tasks with the same computational cost ensures load balancing using few homogeneous resources. Actually we are interested in the use of heterogeneous resources. In diis case we may further divide sub-domain so that to generate a sufficient number of independent tasks with a finer granularity, which permits to exploit the self balancing properties of master - worker programs (Bacci°9, Schmidt94). The thematic sub-domains have been further divided using a spatial approach and individuating independent sub-regions. The sub-regions have been derived using the natural subdivision represented by drainage basins. In the studied region we have eight main drainage basins, which combined witii the six themes lead to a set of 48 tasks of sufficiently fine granularity to ensure an acceptable load balancing for the available set of heterogeneous nodes we have considered (see Section 4). Using a mix of thematic and spatial subdivision we obtain tasks with different computational cost. It is simple to predict the approximate relative computational cost of each task by considering the number of features and their density, this latter obtained using the area of the basin. In the present implementation we based scheduling mainly on a locality principle, and the master tries to assign the set of tasks corresponding to a sub-region to a single worker as much as possible, but still keeping load balancing. The master assigns tasks to processors considering the relative cost of sub-regions, the relative speed of processors, and a simple round robin strategy. This scheduling policy permits to reduce communication of large data set during the third phase of the algorithm, since the overlay operations may be executed using data available on each computing node. 4 Experimental results and conclusions We experienced the parallel tranquillity mapping algorithm on six personal computers running Windows NT and connected on a local area network, based on switched and mixed Ethernet / Fast-Ethernet. In this implementation the data warehouse is hold by an office like database, and the software is implemented using an object based proprietary library for GIS functions and PVM for communications. The PC are equipped with a Pentium III processor, with different clock frequency (from 350 to 700 Mhz), and 128 Mbyte of RAM. The measured network bandwith is around 1.7Mbyte/sec. The relative speeds of die computing nodes are 1, 1.3, 1.5, 2.1, 2.3, 3.0, and are calculated assuming the GIS workstation, used to run the sequential version of the algorithm, as the reference node with speed equal to 1. The relative speed has been measured executing the same task on the unloaded nodes.
89
The total available relative computing power is 11.2. The program has been tested by running it on the dataset of the region described in Section 2. We recall that to run the sequential code for this region on the GIS workstation takes about 12 hours. To discuss performance of the parallel program it is useful to consider its structure as depicted in Figure 2. Phase 1 represents an overhead with respect to the sequential algorithm. The main cost of this phase is due to dataset transfer to the local disk of each node. The input dataset for the considered region is around 80 Mbytes. The measured time for this phase is around 5 minutes. Phase 2 is dedicated to tasks processing, and its execution corresponds with a good approximation to themes processing for the sequential algorithm. This part of the sequential algorithm represents about the 70% of the total execution time, which is around 8 hours and 30 minutes. The measured time for this phase of the parallel program is about 61 minutes, which leads to a speed-up of 8.4, and a weighted efficiency around 75%, considering the available computational power. During this phase speed-up is mainly limited by not balanced computation, since the communication is limited to the exchange of few short messages between master and workers. Monitoring the load of each worker we note an over-scheduling of the node with relative speed equal to 2.3. Indeed this node computes two complete drainage basins (sub-regions) with a quite large number of features, and its relative load overwhelms its relative speed. Phase 3 is dedicated to map overlay and join. The overlay operation corresponds at a large extent to the map optimisation process in the sequential algorithm. The join operation, the dataset transfer and merging represent overheads of the parallel algorithm. The measured execution time for this phase is around 42 minutes. The whole parallel algorithm takes about 110 minutes, thus leading to a speed up around 6.5 and a weighted efficiency around 54%, again considering the available computational power, which is quite satisfactory as a first result. Up to now we do not have the opportunity of experimenting the algorithm on different data sets, because of lack of availability of suitable input data collection. However we made some simulation to better assess the performances of the algorithm. The simulation considers as input parameters the number of processing nodes, their relative speeds, the number of tasks and their costs, the communication cost, and the tree structure to build up the complete solution by properly combining single tasks results by overlay and join operations. The simulation is based on a simple analytical model of the computation derived from (Pasetto97, Clematis98). The simulation results show that the algorithm should maintain the observed performances in most cases, provided that data partitioning is properly supervised.
90 In future works we plan to modify the simple round robin strategy for sub-region assignment to processors with more sophisticated list scheduling heuristics. The reported experience shows that it is possible to use parallel computing for GIS applications developed using proprietary GIS, and to make it available in the public administration domain. The use object based technology seems to provide an effective possibility, while waiting for more advanced solutions which will be able to exploit the computing power provide by large, distributed, and truly heterogeneous GIS systems (Dowers 00 ). Acknowledgements Part of this work has been supported by the PLAINS project founded by European Union (Project ENV-4CT98-0753). The cartographic data used to produce Tranquillity Mapping are properly of Regione Liguria. The traffic data and other data used to derive disturbance levels have been provided by Savona Province. References (AlessioOl) G. Alessio, S. Bini, A. Clematis, M. De Martino, "Landscape characterisation", Tech. Rep. PLAINS-IMA-DE2, Marzo 2001 (Bacci99) Bacci B., Danelutto M., Pelagatti S., and Vanneschi M. "SklE: a heterogeneous environment for HPC applications", Parallel Computing., 25 (1999) 1827-1852. (Bell99) Bell S., "Tranquillity Mapping as an Aid to Forest Planning", Information Notes, March 1999, Forestry Commission, ( www.forestrv.gov.uk/publications/index.html) (Clematis98) A. Clematis, A. Corana, "Performance Analysis of SPMD algorithms on a network of workstation with virtual shared memory", in "Parallel Computing Fundamentals, Applications and New Directions", E.H. D'Hollander et al. Eds., 1998, pp.65 7-664 , North Holland (Cudlip99) Cudlip W. et al, "A new information system in support of landscape assessment: PLAINS", Computers, Environment and Urban Systems, 23 (1999) 459-467 (Dowers99) Dowers S., "Towards a framework for high-performance geocomputation: handling vector-topology within a distributed service environment", GeoComputation 99 (www.geovista.psu.edu/geocomD/geocomp99 ) (Healey98) Healey R. et al, "Parallel processing algorithms for GIS", Taylor and Francis, 1998 (Pasetto97) Pasetto D., Vanneschi M., "Machine independent analytical models for cost evaluation of template-based programs", Fifth Euromicro Workshop on Parallel and Distributed Processing, IEEE Computer Society, Jan. 1997, pp. 485-492 (Schmidt94) Schmidt B.K., Sunderam V.S., "Empirical Analysis of Overheads in cluster Environments", Concurrency Practice & Experience, 6(1994) 1-32
91 Parallel skeletons and computational grain in quantum reactive scattering calculations Stefano Crocchianti, Antonio Lagana, Leonardo Pacifici, Valentina Piermarini Department of Chemistry, University of Perugia, Via Elce di Sotto, 8, 06123 Perugia, Italy Abstract Two quantum reactive scattering computational approaches have been analyzed for parallelization. The parallel structuring of the two codes has been carried out using both the constructs based on directives of the MPI library and the skeletons defined by the SklE coordination language. 1
Introduction
In order to design efficient parallel computational procedures for dealing with complex chemical applications one has first to map the physical problem into a suitable mathematical model and then to transform the mathematical problem into an appropriate algorithmic structure. This process is not simple since there is not a unique correspondence relating physical, mathematical, and algorithmic parameters. A way of learning how these parameters are linked together is to single out of the existing codes the structures suitable for parallelization : . In this paper we examine two computational procedures taken from the problem solving environment SIMBEX 2 (an a priori simulation of molecular beam experiments). Typically, these procedures consist of a central computationally high demanding section. This central section can be schematized as one (or more than one) block(s) 3 having the structure given in Scheme 1: LOOP on a LOOP on b LOOP on z G = g(a,b, END the z loop
,z;a,/3,
w)
END the b loop END the a loop Scheme 1: The scheme of the typical central section of a reactive scattering code. where latin letters are used to indicate parameters for which the calculation of G is separated into uncoupled (albeit, eventually, nested) loops while greek
92
letters are used to indicate parameters for which the calculation of G is not decoupled into independent computations. 2
The computational procedures
The G block of the two quantum reactive scattering programs considered here is devoted to the integration of the Schrodinger equation for reactive processes 4 . The first of these programs is ABM. This program constructs the coupling matrix of the reactive scattering differential equations 5 obtained in quantum time independent approaches. The second program is TIDEP. This program integrates in time the quantum time dependent Schrodinger equation after collocating the system wavepacket on a proper multidimensional grid 6 . Both programs store calculated quantities that, owing to their large size, cannot be kept in memory. This allows the use of these data by the other programs in which the complex computational procedure is articulated. 2.1
ABM
At a given value of the total angular momentum quantum number J (in some approaches this is performed only at a given reference value of J ) , the G block of the ABM program 7 calculates for every point of a grid (the grid along the reaction coordinate that is divided into sectors) the eigenfunctions (given as a combination of some primitive functions) of a proper portion of the Hamiltonian. The program calculates also related eigenvalues and the overlap integrals between the eigenfunctions of adjacent sectors. The structure of ABM is given in Scheme 2: LOOP on the reaction coordinate grid-points LOOP on A Build the local primitive functions Evaluate local eigenfunctions and eigenvalues Store on disk the eigenvalues IF(not first grid-point) THEN Calculate overlaps with previous grid-point eigenfunctions Store on disk the overlap matrix END IF Calculate the coupling matrix END the A loop Store on disk the coupling matrix END the reaction coordinate grid-point loop Scheme 2: Scheme of the G block of the ABM program.
93 where the two nested loops run over the grid-points of the reaction coordinate and the 2 J + 1 projections A of the total angular momentum J on the z axis of a body fixed frame. The central section of the G block calculates the local eigenvalues and surface functions at each allowed value of the projection A. In the same section, overlaps between surface functions calculated at neighboor grid-points and the various contributions to the coupling matrix are calculated. This tightly couples each cycle of the loop on the grid-points to the previous one. Overlaps and eigenvalues are stored on disk for use by subsequent programs. 2.2
TIDEP
The G block of the time-dependent program TIDEP propagates in time t the real part of the wavepacket describing the system 6 . At the beginning (t = 0), for a given pair of values of J and A, the system wavefunction, expressed in a proper functional form using a suitable set of coordinates, and provided with a given amount of energy 6 , is collocated on a sufficiently fine grid of the spatial coordinates. Then the integration on time is performed by repeatedly applying, for about 104 -j- 105 times, the propagator which involves some time consuming matrix operations. After each time step propagation, the wavepacket is expanded, in terms of product states, at an analysis line placed in the product region. The coefficients of the expansion are stored on disk for use by subsequent programs. The structure of TIDEP is given in Scheme 3: LOOP on t LOOP on A Perform time step integration Perform the asymptotic analysis Store C(t) coefficients END loop on A END loop on t Scheme 3: Scheme of the G block of the TIDEP program.
where the two nested loops run over time and A. There are, obviously, some outer loops running on J and other initial conditions, like the vibrational (v) and the rotational (j) quantum numbers, which are fully decoupled. 3
Suitable parallelization schemes
The two programs allow the exploitation of the parallelism at levels external to the G block in an SPMD fashion. In the case of TIDEP this can be applied,
94
for example, to v and j quantum numbers. For both TIDEP and ABM it can also be applied at a lower level to J. This choice is, however, not exempt of problems since an increase of J makes the associated matrices very large and the computing time very long, leading to memory and load imbalance problems. This is the main reason for pushing the decomposition to a finer level inside the G block where a natural parallelization is not applicable. To this end different actions can be undertaken depending on the program considered and, sometimes, on the physics of the problem. 3.1
ABM
In ABM, the coarsest grain of G for which parallelization can be enforced is the loop over the reaction coordinate grid-points. Single grid-point eigenfunction calculations and overlap integral (with eigenfunctions of the previous grid-point) evaluations are, in fact, computational tasks small enough to be easily left with a single processor. At the same time, related memory requirements are limited enough to be handled by the node local memory. This makes it convenient, in certain cases, to group the calculations related to several grid-points and assign them to a single processor in a static way using a data parallel approach (for example, this is what has been found convenient when dealing with reduced dimensionality techniques 8 ). For procedures based on full dimensional approaches cpu and memory requirements are so large that each grid-point calculation has to be assigned to an individual processor and the distribution has to be performed by adopting a dynamic task farm model. However, to make this parallelization scheme viable, the evaluation of the overlap integrals considered has to be decoupled by repeating the calculation of the surface functions performed on the preceeding node 7 . In the actual implementation of ABM, the master process sends to each worker a grid-point for which the eigenfunctions are to be calculated. Once the calculation is performed, the eigenvectors are stored on disk for use by the processor dealing with the subsequent reaction coordinate grid-point. The next processor retrieves from disk the preceeding grid-point eigenvectors and evaluates the eigenfunctions at the quadrature points of the current sector. To prevent attempts to read not yet stored information, nodes are synchronized before reading from disk. This parallelization scheme has been implemented using MPI 9 and the reactive probabilities of the Li + HF reaction 10 have been calculated. Typical processor time measurements for runs performed on the Cray T3E of EPCC (Edinburgh, UK) using 32 (solid line) and 64 (dashed line) processors, are shown in Figure 1. The excellent performance of the model is confirmed by the good balance of the load assigned to the various processors whose values deviate only by a few
95 50 n 32 processors ••• 64 processors 40
30
20
10-
-5
0
_L 5
_EL 10
15
Percentua] deviation from average processor time
Figure 1: Frequency of percentual deviation from the average processor time.
percents from the average time. Calculated speedups S, shown in Figure 2, also indicate that the program scales quite well, since for up to 128 processors the speedup is never smaller than 70 % the ideal value. 3.2
TIDEP
Direct calculations cannot be used to decouple the t loop of TIDEP inside the G block. The solution is, in fact, recursively accumulated at each iteration. Nonetheless, a partial decoupling can be introduced by assuming that A is conserved within each sector during the one propagation. This confines the coupling between different A blocks of the solution matrix to that exerted by the Coriolis term of the Hamiltonian on adjacent blocks. Test calculations were performed on the ORIGIN 3800 at CINECA (Bologna, I) u using a task farm model for the 0( 1 D)+HC1 atom diatom reaction 12 on a gridsize of dimension 127 x 119 x 8. In this model node zero acts as a master. It performs preliminary calculations and distributes fixed J and A propagations to the workers. Average processor computing time for J and A pair calculations are given in Figure 3 as a function of J (each J calculation includes all the fixed A components of one parity). The excellent performance of the model is confirmed by the fact that the average processor computing time increases
96
50
0
50 Number of processors
100
Figure 2: Limit (dashed line) and measured speedup (solid line) for the ABM program
only of about 20 % in going from J = 0 to J = 1 3 . Speedups calculated on the ORIGIN 3800 are shown in Figure 4. They clearly indicate that the scalability of this program is very satisfactory for platforms with up to 16 processors. 4
Towards a SklE based approach
The detailed analysis of a computational application (as the one outlined above for ABM and TIDEP) requires a great deal of know-how not only about the physics background of the code itself but also about the computing platform adopted. An increasingly popular approach is to make use of parallelization environments like SklE (Skeleton-based Integrated Environment) 13 and to exploit the features of the related coordination language (SklE-CL). The use of SklE-CL makes the simplification of the parallel restructuring of the considered codes and the exploitation of alternative parallelization strategies extremely simple as shown below. 4.1
ABM
The parallelism on the initial conditions of ABM (outer loops) can be dealt by SklE as a data parallelism using a map structure (in which the same com-
97
5400
o
5200
o o
/ /
-S5000
p 4800
o
/ Jt
O
4600 s o 4400i 0
^
° '
' 5
'
1 10
' 15
J
Figure 3: Average processor computing time.
putation is applied to all the elements of a data array). In this case, each computation is totally independent of the others, and the main body of the program preserves its sequential structure. On the other hand, when the parallelization is pushed to the finer grain of the reaction coordinate grid-points, inside G, a farm construct needed to be considered in order to assign a virtual processor to each grid-point calculation in a dynamic way. Accordingly, a certain number of grid-point calculations are sent in a data stream and then each of them is assigned to a worker of the farm. As soon as the worker completes its task, it receives the next grid-point calculation as channeled by the stream (here, it would not be needed to recalculate the surface functions of the preceeding grid-point if a shared memory facility could be used). 12
TIDEP
Also TIDEP can use a map construct to parallelize the outer loops of v and j (at fixed value of J ) . A certain amount of work-load imbalance can arise when j and v become large and require a denser grid (and, therefore, a larger dimension of the matrices). When J increases, however, because of the large number of A projections, fixed J calculations do not fit into the node memory. To push the parallelization to a finer level it is appropriate to nest a farm construct (to
98
-cos{xy),f{x,y) where, u$
= «^f)(*.l/).
- u_^)(x,y)
+cos(2ny),
(0
(2)
104
0.2
0.4
0.6
0.8
1
Figure 3. Processed image - ENO reconstruction
ifr < --yU^x^y)
ifr > -,u^{x, o
if I
r
=
y) = 2r - 1 +
-rsin(-r2),
b
-rsin(3irr),
\< 2>u(x>y) =1 sin(2irr) \,
r = x +
tan(4>)y,
(3)
(4)
(5)
(6)
In Figure 2, the graph of the function in the domain [-1,1] is presented. The discrete data are obtained on mesh 51x51 points. The image reconstruction was done using the few following steps. From a starting level node, three steps of interpolation were carried out to obtain three new forward values of the function in new nodes. The distance between nodes shrinks to half of its
105
Figure 4. Processed image - linear interpolation ENO
original value in every step and the amount of the nodes is eight times the original. In Figure 3 the reconstruction for ENO interpolation for (408x408) nodes is presented. Figure 4 presents reconstruction for linear interpolation. It is easy to see the difference in the behaviuor of the solution for ENO and simple linear interpolation in the vicinity of the discontinuities. A satisfying sharpness of the discontinuities was not obtain for the latter. On the discontinuities ENO reconstruction accuracy is of the first order (same as linear interpolation), but in other region the accuracy only depends on the choice of the order of the approximation. The case presented in Figure 3 is second order of accuracy. The bad result for Figure 4 is also caused by the fact that error in the vicinity of the discontinuities has accumulated over steps of interpolations. For ENO reconstruction every step introduced only low accuracy points on the sharp discontinuity, while preserving accuracy of other regions.
106
4
Aspects of parallelisation
The parallelisation of the method was carried out using domain decomposition. A two-dimensional image was divided for non-overlapping subdomains (parallel stripes). The choice of such parallelisation method has provided no restriction on scalability. Non-overlapping subdomains were used to decrease memory usage needed to store data from the neighbourhood of the subdomain borders. This means that the values of the points are only computed once in a specific subdomain. The only values needed to compute interpolated points on the boundary are sent from neighbouring domains. After every level of reconstruction, data is collected by the main processor and if a new computation is needed, the data is resent. Due to the way the algorithm is constructed, the parallel efficiency is as presented in Figure 5. The time was estimated for the image - 166464 points low accuracy (second order method, called basic case in Figure 5) and high accuracy interpolation (forth order method, larger case needing a computational time,which is about 200 times longer). The gain in computational time is very drastic between computers with only one and those with several processors. With an increased number of processors, the advantage is not that large for a basic low accuracy case because the cost of the computation starts to be comparable with the time necessary to send and collect data. In the applications usually processed, images are not any larger than in the considered case. For that reason, it seems that, the considered parallel ENO interpolation algorithm can be used with quite good efficiency for computers with less than 10 processors. Of course in some cases the data files are much larger than the case considered above (for instance, when there is a need for an interpolation of the results of engineering computations several millions of points can be reached). In this case, then an increase in the number of processors will significantly increase the efficiency of the computations. For the pictures considered in this paper, time of calculation on 16 processors was very short, allowing to obtain enhanced picture almost immediately. Hence, there was no need to study effect of parallelisation on the higher number of processors. The algorithm of the method is very general, so there is no obstacle to calculate more complex cases with the larger number of processors. For the realisation of parallelisation, MPI was used and all calculations were done at the R10000, Onyx 3800.
107 11»
1
1
i
1
1
1
1
* larger case O basic case 0.9 0.8 0.7
-
o
-
0
-
§0.6 S
H;0.5
«
-
E
-
|0.4 0
0.3
-
# 0.2
"
o
0
* 0.1 0
* '
1
8
10
12
* 1
18
number of processors
Figure 5. Parallel efficiency of the image reconstruction algorithm. Basic case represents lower accuracy interpolation. Large case is higher order interpolation
5
Future work
There are basically two directions in which recent work can be continued. The first one is the adaptation of the existing algorithm to unstructured 2D cases. It may not have a straightforward application significance for image processing, but for large scale engineering applications it is a very desirable goal. The unstructured meshes are better at adapting to complex geometries, which are common for engineering problems, but it is also known that existing methods of interpolation and differentiation do not reach the required accuracy, which is easily obtained by structural grid calculation. Because of this the unstructured ENO interpolation will be interesting for future work .The second path could be 3D approximation. It is again important for large engineering computations and particularly interesting for computational fluid dynamics. In both cases the parallelisation issue is very important to make considered methods efficient in usage.
108
6
Conclusions
In this paper, the accurate and efficient method for interpolation was presented(ENO interpolation), which can be used for image reconstruction of a discrete set of data. Due to the parallelisation, the efficiency of the method increased and made it a good alternative to other widely used interpolation methods. The parallel algorithm seems to work especially well for large set of data,which it makes it even more promising for future applications. References 1. A. Harten. Eno schemes with subcell resolution. J. of Comp. Physics, 83:148-184, 1989. 2. C-W. Shu and S.Osher. Efficient implementation of essentially nonoscillatory shock-capturing schemes. J. of Comp. Physics, 77:439-471, 1988. 3. S. Osher and C.-W. Shu. High-order essentially non-oscillatory schemes for hamilton-jacobi equations. SI AM J. of Numer. Anal., 77:907-922, 1991. 4. B.B. Kimia K. Siddiqi and C.-W. Shu. Geometric shock-capturing eno schemes for subpixel interpolation, computation and curve evolution. Tech.Report LEMS, 142, Febuary 1995. 5. P. Haeberli and D. Voorhies. Image processing by linear interpolation and extrapolation. IRIS Universe Magazine, August 1994. 6. A. Maraquina and R. Donat. Capturing shock reflections: A nonlinear local characteristic approach. UCLA CAM Report, 31, April 1993. 7. R. Abgrall. On essentially non-oscillatory schemes on unstructured meshes: analysis and implementation. J. of Comp. Physics, 114:45-58, 1994. 8. R.Abgrall and A.Harten. Multiresolution representation in unstructured meshes. SIAM J. of Numer. Anal., 35:2128-2146, 1998.
TRAINING ON-LINE RADIAL BASIS FUNCTION NETWORKS ON A MIMD PARALLEL COMPUTER A. D'ACIERNO 8 ISA -CNR, via Roma 52 A/C, 83100 - Avellino, Italy This paper describes a parallel mapping scheme for the gradient-descent learning algorithm. The problem we are dealing with is Time-Series forecasting by means of GRBF Networks so that (i) the network has one neuron in the output layer but, at the same time, (ii) the memory of each processor can (typically) hold the whole training set. A mapping scheme recently proposed seems to be the optimal one (the only one?) for speeding the learning up. The used approach is synchronous, so that both SIMD and MIMD parallel computers can be used for actual implementations. Results on a MIMD parallel computer are shown and commented.
1
Introduction
The Time-Series forecasting problem (given a sequence x(l),x(2),....,x(N) find the continuation x(N+l),x(N+2),...) can be stated in terms of estimating an unknown function/such that: x(t) «Jc(f) = f(x(t - 1 ) , x(t -2),....,x(t-M)) (1) where M is the (unknown) memory of the (unknown) system. As it is well known, the problem of estimating an unknown function given some input-output pairs (regression problem) can be parametric and non-parametric. In parametric regression the form of the functional relationship between the dependent and independent variables is known (or guessed) but it contains parameters whose values have to be determined. In the non-parametric regression, instead, there is no a priori knowledge about the shape of the function to be estimated. Generalized Radial Basis Function (GRBF) neural networks are non-parametric models (that can be linear as well non-linear) able to approximate any reasonable continuous function mapping arbitrarily well and with the best approximation property [3]. As each neural network, the behaviour of a GRBF network depends on free parameters evaluated through a learning procedure; such a procedure tend to be very time-consuming and it is obvious to capitalize on the intrinsic parallelism of neural systems to speed the computation up. Learning algorithms typically involve only local computation but, on the other
hand, the output of each unit depends on the output of many other units so that most of the running time can easily be spent in communication rather than in actual
E-Mail: dacierno.a @isa.av. enr. it
109
110
computation. The motivation for this work was exactly to try to solve the mapping problem, with reference to a well known learning algorithm. 2
RBF: A Brief Introduction
The problem of estimating an unknown function / : 3iM —> SR given a set of data {(jr.,y(.)€ 9t" x5t|_, clearly is ill-posed, since it has an infinite number of solutions; to choose a particular solution, we need some a priori knowledge about the unknown function, typically assuming that the function is smooth (i.e. two similar outputs corresponds to two similar inputs). Since we look for a function that is, at the same time, close to the data and smooth, it is natural to choose as the solution of our regression problem the function /that minimizes the following functional:
n[f]=hf(xi)-yi)2+4Pf\\
(2)
where X (> 0) is the so called regularization factor, P is a constraint operator (usually a differential operator) and II - II is a norm (usually the L2 norm) on the function space off. The operator P clearly embodies the a priori knowledge about the solution and so depends on the problem to be solved. Minimization of functional H leads to the Euler-Lagrange equation that can be written as:
PPfi*) =T2>i:"/(*,))*(*-*,)
(3)
where P is the adjoint of the (differential) operator P and the right side comes from the functional derivative (with respect to/) of//. The solution of the partial differential equation 3 is the integral transformation of its right side with a kernel given by the Green's function of the differential operator PP, that is the function G satisfying: PPG(x;y) = S(x-y) whose solution, because of the delta functions, can be written as: f(x)=\fd(yi-f(xl))G(x;xi)
(4) (5)
By evaluating equation 5 at the W data points we easily obtain a set of equations for the coefficients c = ^' ~*^X'''A '
/A.
.
111
When the operator P is translationally invariant G will depend on the difference of its arguments ( G ( x ; x i ) = G ( x - x,)) and if P is also rotationally invariant G will be a radial function, i.e. G(x;xt) = G(|x - x,|) (e.g. a Gaussian function). The solution given by standard regularization theory is expensive in computational terms (OfN*)) and, what is of course worse, the probability of ill-conditioning the system is higher for larger and larger matrices. The generalized regularization theory approach is to expand the solution on a smaller basis, so deriving:
/Oc) = £c,.G(r,f,.)
(6)
where n«N and the f,'s are called "centres": equation 6 is topologically described in figure 1, where each Green's function is realized by means of a neuron; these networks are called Generalized Figure 1. A GRBF Network. Radial Basis Function (GRBF) networks. When the G,'s are Gaussian functions, it is of course: ( lv Output
Input
Lvytr
0(*) = £ w,. *f[exp' y-i
(*,-'„> >
(7)
Kjl
To evaluate the free parameters there are several possibilities. The simplest one selects centres by means, for example, of a clustering algorithm; then selects the o's according some heuristics and last, to evaluate the w's, solves a least square problem. Using such an approach a GRBF network behaves almost linearly. A quite different approach (that makes GRBF networks highly non-linear models) assumes that all the parameters can change during the training phase; in our experiments, for example, the centres are initialised using a competitive clustering algorithm and the learning strategy tries to minimize the error E defined as:
by using the well-known gradient-descent algorithm. This is an iterative algorithm that at each step (by considering the i-th training pattern and if Xy represents component 7 of example i) applies the following formulae: Aw
/ = ~1„ TT^" = IS, Ii e x P 2 dw,
*„f
\J 2a,
(9)
112
1 def
f^-^Vr
((to-',/)
2dalr
{
^ 2a„. )
alr
)M
n n
This procedure can of course work either on-line or in batch mode; in the on-line case parameters changes are used as they are evaluated while, in the batch mode, they are accumulated to evaluate global terms that are summed after all (or some) examples have been processed. 3. The Proposed Mapping Scheme The design of a neural predictor requires the choice of the parameter M to create the training set from the available historical sequence (this is one of the complications that make Time-Series forecasting a more difficult problem that straight classification). Analyse mis fact from the neural designer point of view; even if tricks and methods to guess the right M exist, it is a matter of fact that (theoretically) an over-sized M could be used, since not useful values should (or, maybe better, could) be neglected by the neural structure simply making a\ —» +,=E,
(12)
t=l
where E is the unit matrix. In dependence on the choice of matrices B; we have different realization of algorithm 1. Since for our problem x > 0, gi > 0 and b ' > 0 for i = 1, 2, ...n, we can also construct the following parallel multiplicative algorithm Algorithm 2 ( P M S U M ) 1. x ( 0 ) € R" and x(°) > 0. 2. Calculate the (k + l)th iteration by
4+1) = 4}f[yj'''
( 13 )
where
»*" = (o^) , " ,, 'c' = 1'2 »
.
;
Tims integration
\ •:
Nonlinear Solver*
i
Linear Algebra Services
\ '•
Figure 1. Block diagram illustrating the key modules for the analog portion of Xyce™.
Scaled Parallel Efficiency
•Netlist - Random Repartitioned
30
40
Number of Processors Figure 2. Graph of scaled parallel efficiency on SGI Origin 2000.
A BREAKTHROUGH IN PARALLEL SOLUTIONS OF MSC.SOFTWARE LOUIS KOMZSK (1) , STEFAN MAYER (2), PETRA POSCHMANN (3), PAUL VANDERWALT (4), REZA SADEGHI (4), CLAUDIO BRUZZO (5), VALERIO GIORGIS (6) MSC.SOFTWARE (1)
2975 Redhill Avenue, Costa Mesa.CA 92626. USA
(4)
260 Sheridan Avenue. 309 RaloAllo.CA 94306. USA
(2)
Am Moosfeld 13 81829 Munich, GERMANY
(5)
Viale ft Bisagno 2/10. 16100 Geneva, ITALY
(3)
815 Colorado Blvd. Los Angeles. Ca 900411777, USA
(6)
Via Giannone 10. 10121 Torino, ITALY
More than a decade ago MSC offered the first parallel production system of MSC.NASTRAN. During this decade MSC has intensified its effort on parallel processing and is now ready to deliver MSC.NASTRAN V70.7 and MSC.MARC V 9, both of which contain very important new parallel features. This paper describes these exciting features and provides preliminary performance results. We believe that these systems mark the best in parallel performance in commercial finite element analysis ever and present a breakthrough in parallel computing in our market.
1
Introduction
More than a decade ago MSC offered the first parallel production system of MSC.NASTRAN, based on the shared memory paradigm. At that time and during the following years the parallelization efforts concentrated on parallelizing the computations in several expensive modules, for example the matrix decomposition. However, during recent years it has become clear that a more extensive parallelization is necessary to satisfy the users' ever growing demand for higher performance. Moreover, it has turned out that not only computations, but also the I/O traffic must be parallelized in order to obtain highly efficient parallel analysis solutions. MSC started to work on new parallelization approaches earlier this decade, this time based on the distributed memory paradigm to be able to address parallel I/O issues as well. First successes were obtained in the European Europort project, which resulted in the distributed parallel production version MSC.NASTRAN V69.2 available on the IBM SP architecture from 1996. Encouraged by the results of this project, the efforts on distributed parallel MSC.NASTRAN have been intensified during me past four years.
173
174
Concurrently the MARC product has also been parallelized. The first parallel version of MARC based on the domain decomposition, Version K6-3, was released in March, 1997 on SGI, HP, IBM, DEC and SUN then in 1999 NT and finally in 2001 Linux. MSC is now ready to deliver MSC.NASTRAN 2001, and MSC.MARC 2001 that contain even more exciting new parallel features. It will also be possible to execute these versions on a homogeneous cluster of workstations. The results obtained with these systems mark the best in parallel finite element analysis performance ever and present a breakthrough in distributed parallel computing in our market. Distributed memory computers are essentially tighdy coupled networks of workstations. The connection between the processors (nodes) of a distributed memory computer is usually a very fast proprietary network or switch. The communication between these nodes is via standard interface libraries, for example MPI (Message Passing Interface), which is used by MSC in the distributed memory parallel implementations. Each node of a distributed memory computer has its own memory, hence the name. Even more important is the fact that the nodes have their own local I/O (disk) device enabling efficient parallel I/O operations.
2
Distributed solution techniques
The distributed solution techniques implemented in Version 2001 of MSC.NASTRAN and MSC.MARC are based on the principle of domain decomposition. We found two possible and convenient ways to decompose finite element problems. They are the frequency and the geometric domain decomposition. In the following me basic principles of both of these techniques are discussed. It is important to note, that the distributed solution techniques provide the same results and output formats as the serial solutions, apart from some additional performance summaries related to the distributed processes. We also made the additional user interface needs very minimal as shown below. 2.1
Frequency domain decomposition based solutions in MSC.NASTRAN
The frequency domain decomposition technique applies only to certain solution sequences where the notion of frequency is utilized. Such are the normal modes and frequency response analyses. More specifically, the frequency domain decomposition based normal modes analysis is built on the frequency segment approach of the Lanczos method. The frequency domain decomposition based
175
frequency response analysis utilizes the independence of the discrete frequencies given on the user's list. 2.1.1
Distributed normal modes analysis in MSC .NASTRAN
The segmented version of Lanczos was first introduced in die early 1990s to alleviate the problems encountered on very wide frequency range runs mostly on Cray computers. In auto-industry tests it was found that the orthogonality is lost when very long Lanczos runs are executed while trying to span a wide frequency range of interest. We introduced me segment version to force intermittent shifts at semi-regular intervals resulting in more frequent restarts of the Lanczos process. While it has significantly improved the solution quality, i.e. we avoided aborts due to loss of orthogonality, the number of shifts was usually more than in the nonsegmented run. It is a natural extension to this logic to execute the segments in parallel, which is the basis of the distributed parallel implementation in Version 2001. This process is executed in a master/slave paradigm. One (the master) of the processors will execute the coordination and collection of the separately calculated eigenvectors into an appropriately ordered set. This guarantees the continuation and proper exit of die master process. The slave processes will contain only the eigenvalues and eigenvectors they found upon exiting the READ (Real Eigenvaue Analysis DMAP) module. 2.1.2
Distributed frequency response analysis in MSC.NASTRAN
The distributed frequency response analysis is based on assigning exclusive subsets of the user given frequency list to each processor. Each processor calculates the responses of the structure at the frequencies given in its subset. The calculation may be executed in a master/slave mode, which was the only method available in Version 69.2 or now in a more efficient symmetric ("all master") operation mode automatically selected by the code. In the master/slave mode the master processor is distributing the structural matrices and the frequency list to the slave processors in the beginning of the frequency response module and upon completion of the response calculation collects the results. This operation mode is used in the distributed modal frequency response analysis solution. In the symmetric mode all processors behave identically. They calculate their own structural matrices and respective frequency subsets and the responses. They each also complete the solution sequence, resulting in higher scalability due to the locally kept output results. This symmetric operation mode is used in the distributed direct frequency response calculations.
176
2.2
Geometric domain decomposition based solutions in MSC.NASTRAN
The principle of geometric domain decomposition is applicable on a much higher level than the frequency domain decomposition. In the frequency case the distributed solution is focused on a certain module (READ and FRRD1) and solution sequence (Sol 103 and 108, as well as 111). The geometric domain decomposition principle transcends many DMAP modules and also solution sequences. In this first production version of this technology we focused on the linear static analysis (Sol 101), however, many aspects of the development work carry over to other solution sequences to be delivered in distributed form in the future. This technology may also be used in connection with the frequency domain decomposition in a hierarchic fashion. Finally, this technology in part relies on the very strong superelement technology of MSC.NASTRAN, which gives its foundation. The cornerstone of the geometric domain decomposition is an automatic domain decomposition tool. This tool works from geometry (connectivity) information and uses heuristic algorithms to create subdomains. The main criteria in creating subdomains are: minimizing the boundary between the domains, achieving load balance and minimizing the cost of the solution of the interior of the domains. In Version 70.7 we use the EXTREME tool for domain decomposition, however, other tools such as METIS may also be used in the near future. After automatically creating the subdomains, MSC.NASTRAN's superelement process will be executed. The shortcoming of the boundary solution of the conventional superelement process, the explicit creation of the residual matrix, however, is avoided. This requires very advanced and efficient distributed algorithms designed and implemented by MSC. These proprietary algorithms encapsulate most of the interprocessor communication via MPI utilities. The data recovery for each domain is done independendy on each processor and does not require any communication. This results in linear or better speed-ups and disk-space savings during that part of the run. 2.3
Geometric domain decomposition based distributed solutions in MSCMARC
The application of the domain decomposition technique requires further considerations in MSC.MARC, due to the nonlinear focus. The various sources of nonlinearity, such as material, geometric and kinematic nonlinearity as well as contact need to be taken into consideration.
177
Additional requirements distinct from linear analysis are handling of multiple steps and die step-by-step changing load balance in contact analysis. In general, die stability plays a key role in the distributed implementation of MSC.MARC. Above considerations and requirements induced a different subdivision strategy dian in MSC.NASTRAN. Using domain decomposition in MSC.MARC, die geometric model is subdivided in me pre/post processor MENTAT (generate option). This enables us to take into consideration user given natural or CAD system specified substructures and handle other physical constraints. It also enables an interactive control of the subdivision process. On the post processing side, mis approach facilitates a data recovery process by domains or in the full complexity of die model, as the user wishes. Finally, diis approach results in a domain decomposition methodology trans- parent to MSC.MARC application or solutions. The metiiodology may be applied in general nonlinear analysis applications including contact. 3
Application Examples
3.1
MSC.NASTRAN Version 70.7 results
To demonstrate the performance improvement obtained by die distributed Lanczos technique, we choose a carbody model of approximately 200,000 degrees of freedom consisting of mainly shell (QUAD4) elements (Figure 1).
Figure 1. Carbody model
Figure 2. Exhaust manifold model
Figure 3. Crankshaft model
The normal modes analysis was executed up to 400 Hz, extracting 1876 eigenvectors, using lOOMw of memory. The disk usage (the main processor's scratch database high water mark) and the elapsed time of the total solution are shown in Table 1.
178
Number of CPUs 1 2 4 8
Disk Gbytes 10.3 7.7 7.0 6.7
Elapsed min:sec 449:27 329:23 122:39 87:57
Speedup — 1.4 3.7 5.1
Table 1. Distributed normal modes performance results (fig. 1) The scalability and the speedups are quite remarkable. We obtained a 5 fold elapsed time speedup on 8 processors. It is even more important to point out the significantly lower disk requirements on the individual processors in the case of the parallel runs due to the smaller scratch space needs. The distributed parallel frequency response capability is demonstrated by the analysis of an exhaust manifold model (Figure 2). The model consists of app. 50,000 degrees of freedom and was built of mixed (shell and solid) elements. Direct frequency response analysis (Sol 108) with 100 frequency steps resulted in the performance measurements of Table 2. Number of CPUs 1 2 4 8
Elapsed Seconds 9521 5001 2672 1546
Speedup 1.9 3.6 6.2
Table 2. Distributed frequency response performance results (fig. 2) This analysis represents an even better scalability. This is due to the fact that me complete frequency response solution was executed in parallel and none of the processors collected the response results of others, the results were kept local and the database separate (mergeresults=no). The effect of the geometric domain decomposition based distributed execution of the linear statics analysis is shown with the following example. The model is an automobile crankshaft (Figure 3) with about 150,000 degrees of freedom and built exclusively from solid elements. Table 3 compares the elapsed time (min:sec) and disk space requirements of Solution 101 with one and 8 processors.
179
Disk Elapsed
1 Processor 1.4GB 38:42
8 Processor 0.26GB 8:35
Speedup s/Saving 5.4 4.6
Table 3. Distributed linear statics performance results (fig. 3) The speedup of 4.6 on 8 CPUs is very good and the 5.4 fold saving in maximum local disk space is significant. It is important to note that all 1 CPU runs shown in this section were executed with the fully tuned serial production implementation of MSC.NASTRAN and not from single processor runs of the parallel algorithms. 3.2
MSC.MARC2000 results
The performance improvement obtained in MSC.MARC V 9 is first shown in a metal forming application. This example is the analysis of the thread - rolling process of the manufacturing of a carseat shown on Figure 4. Such an application is widespread in the automobile industry since many components of automobiles are manufactured in that fashion.
Figure 4. Carseat model
Figure 5. Engine
The finite element model consists of 9600 elements, 9881 grids and 44 time steps were executed. The results on an SGI ORIGIN 2000 (250 MHz) shared memory parallel computer are shown in Table 4. Number of CPUs 1 2 4 8
Elapsed Seconds 7,702 4,899 2,025 1,267
Speedup 1.6 3.8 6.1
Table 4. Distributed metal forming application results (fig. 4)
180
This excellent speedup is almost identical to the speedup obtained in the MSC.NASTRAN Version 70.7 parallel frequency response analysis example shown in Table 2. This fact demonstrates the applicability of the domain decomposition principle across products and solution techniques. An additional example of a nonlinear analysis of an engine block is shown in the following Figure 5. The analysis was to determine the transient thermal response to a high temperature heat source applied within the cylinders. The model has about 500,000 elements and approximately 1.8 million degrees of freedom, 10 time steps were executed. The results on a SUN E3500 (333 MHz) computer were as follows: Number of CPUs 1 8
Elapsed Minutes 78 12
Speedup — 6.5
Table 5. Distributed nonlinear analysis results (fig. 5) The efficiency result shown in tiiis application is truly remarkable and demonstrates the strategic importance of the domain decomposition methodology. 5. Conclusions We hope to have demonstrated the significant performance advantages of the new distributed solution techniques. These techniques truly unleash the power of parallel processing and present groundbreaking solutions in our segment of the CAE market. We believe that these capabilities will be very useful for a wide range of our user community. We are continuing the distributed parallel development and are committed to further improvements. We are planning to expand the geometric domain decomposition based technology to other solution sequences of MSC.NASTRAN, for example normal modes and transient analysis in the future. We are also continuing to improve the domain decomposition solutions in MSC.MARC, especially in the area of multi-disciplinary (structure, heat, electromagnetic, fluid) applications.
181 Implementation of a parallel Car-Parrinello code on H i g h Performance Linux-based Clusters S.Letardi 1 , M . C e h W ^ F . C l e r i 2 ' 3 , V.Rosato 1 ' 3 ENEA, Casaccia Research Centre, HF'CN Project, Roma, Italy ENEA, Casaccia Research Centre, Divisione Materiali Avanzati, Roma, Italy 3 INFM, Unita di Ricerca Roma I 1
2
4
A. De Vita 4 ' 6 , M. Stengel 4 Insitute Romand de Recherche Numefique en Physique des Materiaux (IRRMA), INR~Ecublens, 1015 Lausanne, Switzerland 5 INFM and Dipartimento di Ingegneria dei Materiali, Universita di Trieste, via A.Valerio 2, 3^127, Italy A Car-Parrinello Molecular Dynamics code has been implemented on two different Linux-based parallel platforms, based on Alpha processors. The clusters are characterized by state-of-the-art interconnection networks, enabling high throughput and low latency. The performance and the scalability of the code are reported in detail, together with the analysis of the performance of the most intensive computational tasks. Results indicate a good scalability of the code on the platforms and the adequacy of the interconnection networks to sustain a large workload.
1
Introduction
Molecular Dynamics based on Density Functional Theory x (DFT), the socalled Car-Parrinello approach 2 (CP hereafter) has gained a great deal of success in several application domains, from quantum chemistry to materials science, from biology to reaction's chemistry, as its accurate modeling of the bonding properties allows to make qualitative and quantitative predictions of the physical phenomena. The CP approach is particularly intensive from the computational point of view. Its application to complex systems has coincided with the advent and the diffusion of parallel computing. The availability of high performance parallel computers has, in fact, given the possibility of significantly extending the applicability of the CP approach to large systems, to long time scales up to accessing the scales of complex chemical reactions. From the side of the computational platforms, the recent years have been characterized by a change of paradigm which has imposed parallel platforms, assembled with commodity-off-the-shelf (COTS) components, on the proprietary parallel architecture. The new architectural solution, based on COTS processors, interconnected via commercial networks (IN) and managed by Unixlike operating systems, such as Linux, (we will refer to them, hereafter, as Beowulf Clusters, BC), are becoming the most widespread parallel platforms. The
182
first generation of BC's, based on low-cost COTS components, has revealed, however, to be inadequate to cope with highly parallel tasks. Communication networks and the general-purpose architecture of low-cost processors generally fail to sustain large workloads when a relevant number of processes are used. A second generation of BC's has been thus introduced by assembling high-end COTS processors and commercial IN's networks characterized by low latency and high bandwidth. Although the cost of this solution is higher than that of the first generation BC's, it results to be competitive with that of proprietary solution. The cost-effectiveness has triggered the diffusion of these platforms in the last years. In this work we have tested the implementation of a parallel computational code (a CP-based molecular dynamics) on two second-generation cluster-based platforms: the first, located at the ENEA Casaccia Research Center of Rome (Italy), is called Feronia; the second, located at the Centre for Advanced Parallel Applications (CAPA) of the Ecole Polytechnique Federate at Lausanne (Switzerland), is called Swiss TO. They keep the open-source operating system, from the BC experience, but they introduce more sophisticated IN's to increase platform scalability when a large number of processing elements is used. 2
Description of the platforms: the Feronia and the Swiss TO clusters
The processors (PEs) of the Feronia platform are the Compaq-Alpha EV67 (667 MHz) with 4MBytes of L2-cache. They are assembled in dual-processors nodes (API UP2000) in the SMP configuration. Each node is characterized by 1 Gbyte of DRAM and by the presence of the Tsunami chipset for increasing the intra-node communication bandwidth. The peak performance of each dualprocessor node is about 2.6 Gflops. The Feronia configuration has, to date, 16 nodes, interconnected by the QsNet network. QsNet is an IN with a fattree topology and provides a peak bandwidth of 340 Mbytes/second in each direction; the process-to-process latency for remote write operations is 2 us. The design of the data network guarantees that all nodes obtain this level of performance whatever the size of system, ensuring the maximum possible scalability for parallel applications. The network interface is based on QSW's third generation "Elan" ASIC dedicated I/O processor to offload messaging tasks from the main CPU: a 66 Mhz 64-bit PCI interface, a QSW data link (a 400 Mhz byte wide, full duplex link), cache and local memory interface. QsNet offers a remote access latency of 5 fisec and a bandwidth of 210 Mbyte/s on MPI applications.
183
The Swiss TO architecture is based on Compaq Alpha EV56 (500 MHz) processor. There are 8 processors connected via an EasyNet bus. The IN sustained performances are 35 MByte/s, with 5 fisec and 12 fisec latencies for FCI and MPI communications, respectively. The theoretical peak power of the platform is of 8 Gflop/s. The Swiss plan for high performance computing will upgrade the Swiss TO architecture to 504 processors interconnected by TNet network with a performance of about one Tflops. 3
Description of the code
The first-principles Molecular Dynamics based on the Car-Parrinello method allows to simulate the behavior of atomic aggregates where the forces acting on the atoms are evaluated from the knowledge of the electronic ground state. This is calculated in the frame of the DFT theory in the Kohn-Sham (KS) formulation 3 and within the Born-Oppenheimer approximation. In this formulations the energy of a system of electrons in presence of nuclei can be written in terms of the electron density ne(r) and of orthonormal orbitals *&*(?"). Car and Parrinello 2 proposed that the electron wave functions could be treated as classical variables and introduced a classical Langrangian L = Y,»j i
dr|*i|2 + ^ M / i ? f - £ [ t f i , . R / ]
+
^ A i j ( < 9i\9j
> - % ) (1)
ij
I
where the VPj's are the KS orbitals, Mj the ionic masses, /< the fictitious electron mass, Ay are the Lagrangian multipliers needed to impose the orthonormality constraints and E[ipi,Rj] is the energy of an interacting ionic and electronic system. From this Lagrangian the equations of motion are derived. In the CP technique, plane-waves (PW) are chosen as orbital basis set. The number of PW elements to be used to build up atomic orbitals depends on the level of accuracy required by the calculation. It is usually of the order 10 4 -10 5 . The CP code evaluates of different terms of eq.(l) by using the representation of the *j(r) in terms of PW:
{Si Because some terms of the energy E[ipi,Ri] are diagonal in real space and others are diagonal in the reciprocal space, computational resources can be saved computing each quantity in the space where it is diagonal. The use of PW's, in fact, allows to switch from the position to the momentum (k-) space representation by means of Fast Fourier Transform (FFT). This implies that
184
the FFT's represent the most intensive part of the calculation; FFT scales as 0(M log M), where M is the number of PW's. A further relevant task of the CP code is the orbital's orthogonalization: this task scales like N2M, where AT is the number of KS states. Moreover, the DFT algorithm in the usual Local Density Approximation is cubic-scaling with the volume of the physical system. Therefore much care must be devoted to a wise implementation of the computational task. 4
Parallelization s t r a t e g y
The LAUTREC code (The LAUsanne Total REal to Complex energy package) has been used as test code; it implements a first-principles Molecular Dynamics in the CP approach and, since June 1994, it represents the first implementation of a CP-based code on a parallel platform. The parallelization scheme follows the pioneering work of Clarke and co-workers 4 adapting it to the CP method's features. The code is written in Fortran 90, and uses the MPI communications interface. Either on Feronia and on Swiss TO the MPICH primitives have been suitably coded to efficiently run on the available proprietary IN's. The code is communication-intensive and achieves optimal performance and good scalability if optimised distributed linear algebra routines and hardware communications are used. The code essentially performs: 3D FFTs and dense linear algebra operations, like matrix-matrix multiplications and eigenvalue solving. For these reasons, BLACS/SCALAPACK subprograms like e.g. NUMROC, DESCINIT, PSGEMM, PSSYEVX are used. The parallelization strategy is data driven. An equal subset of the planewaves basis set, used to expand the electronic orbitals, is assigned to each processing element (PE). The distribution of the scalar-product workload among the different PEs is a key issue in the parallelization strategy, as the task of performing scalar products between PW, described in eq. (1), is one of the most computationally-cost intense of the code. The PW distribution is made in a way to localize on the PEs as much as possible the scalar product, and to perform some final internodes sum of the partial results. Data (electronic wave-functions) are distributed on a 3D mesh, and the algorithm requires continuous transformations between direct and Fourier space. Therefore, a computationally relevant task is constituted by a 3-D complex FFT to be performed on a grid whose dimensions depend on the system size and on the cut-off energy. For this task, the parallelization strategy takes also into account the symmetry properties of the reciprocal space. Real wavefunctions are thus used when adopting T-point sampling of the reciprocal space.
185
In particular, this halves the basis set effectively used in the calculation by using the symmetry relation Ci(—g) = c*(g) on the expansion coefficients in eq. (2). To avoid conflicts between this (G-space non-local) symmetry and the main parallelization strategy, the distributed data unit effectively consists of couples of conjugate reciprocal space FFT mesh columns, so that the electronic orbitals can be paired in complex vectors constructed from private PE data before the complex FFTs are carried out. Special features of the porting (so called 'T-point symmetry") and of the static load balancing (so called " self-inverse mesh tiling") require the use of an algorithm-specific parallel FFT package. 5
Benchmark details
We performed several test-cases in order to check the performance and the scalability of the LAUTREC code on the two different BC platforms by using the dedicated hardware. The test-cases are related to the simulation of model-systems whose size is typical of those used in scientifically meaningful simulations. On the Feronia cluster we have simulated a carbon system. Two different model-systems have been used: a large one, hereafter referred to as F l , formed by 120 carbon atoms, with a cell dimensions of 13.00 * 12.50 * 6.65 A3 and an FFT grid of 100 * 96 * 56 points; and a small system, F2, composed of 60 carbon atoms, with a cell dimensions of 13.00 * 12.50 * 3.32 A 3 and an FFT grid of 100 * 96 * 32 points. The choice of the cut-off of 40 Ry results in the use of about 15600 plane waves for F l and of about 7800 plane waves for F2. Benchmarks on the Swiss TO cluster have been performed with the following three systems: a system S I , composed of 108 aluminium atoms, with a cell dimension of 12.143 A 3 , a cut-off energy of 10 Ry, an use of about 3200 plane waves and a FFT grid of 48 3 points; system S2, composed of 32 water molecules, with a cell dimension of 9.883 A 3 , a cut-off energy of 30 Ry, the use of about 8900 plane waves, and an FFT grid of 723 points; system S3, composed of 32 water molecules, with a cell dimension of 9.883 A 3 , a cut-off energy of 90 Ry, an use of 46700 plane waves, and an FFT grid of 1203 points. It must be underlined that the number of FFT grid points (nrl *nr2*nr3) must be chosen in such a way to have n r l * nr2, nr2 * nr3, nrZ * nrl multiple of 2 * PEs. 6
R e s u l t s and discussion
We firstly analyse the results obtained on the Feronia cluster on the F l and F2 systems.
186
Figure 1: Speed-up of the LAUTREC code on the Feronia cluster for the F\ and F2 cases, (a): simulations with P / N = 2 ; (b) simulations with P / N = l
For the largest system, F l , we have, at most, a memory occupancy of 450 MBytes on a single node that allows to run the test-cases also on a single node (1 GByte of DRAM). The resulting speed-up data have been reported in Fig.l. It must be noticed that the LAUTREC code compels the use of a number of processes equal to 2P, (with p = l , 2...) for this reason, we have defined the Speed Up as S = TCpu(Np = 1)/TCpu{Np = 2 P ), where Np is the number of processes and TQPU is the wall clock time of the run. The TQPU has been averaged on several runs. We have run the code either by allotting one process or two processes per node. In the first case (hereafter indicated as P / N = l ) only one PE is used on each node, in the second case (hereafter indicated as P / N = 2 ) , two PEs per node are used. Figure 1 shows an overall good scalability of the code on the Feronia platform. It allows a good sustained scalability also for the most time-consuming routines, namely FFT and the orthogonalization one (GRAHAM). The former is also intensive from the point of view of the IN workload. Furthermore we can observe, from Figure 1(b), that there is an improvement of speed-up when passing from 16 PEs to 32 PEs. It could be due to a better distribution of the FFT among the PEs. A further appreciation of the performance of the QsNet can be also extracted from the comparison of data of Table 1 and 2. It comes clear that the most time-consuming routines are mathematical routines, using to solve
187
2
4
6
8
number of PEs
Figure 2: Speed-up of the 5 1 , 52 and 5 3 systems on the Swiss system 5 3 ; diamonds= system 52, triangles= system 5 1 .
TO platform.
Circles=
the parallel FFT and the wave functions orthogonalization. The data domain, in both cases, is completely distributed among the processors. By comparing Table 1 with Table 2 one can observe that the CPU time for the case P / N = l is slightly lower than that for the case P / N = 2 . However this difference becomes progressively smaller by increasing the number of PEs. We guess that the improved performance of the case P / N = l could be due to the lower amount of data in each node that engaged the DRAM and the communication port toward the QSnet. We can observe, from Table 2, that we have a very good speed-up for the GRAHAM routines. When passing from 16 PEs to 32 PEs, however, the communications become too intensive, although this fact does not affect the overall speed-up. In Fig. 2 we show the results obtained on three cases (SI, S2 and S3) on the Swiss TO platform. We can note, observing the graph in Fig.2, that for the smaller systems (diamonds and circles) we have more pressure on communications. They clearly pinpoint where the IN technology starts loosing edge. On the other side, the behaviour of the SI system (triangles) is even slightly superlinear, presumably due to a more efficient use of L2-cache when the system size is spread over a larger number of processors. Second-generation BCs, based on fast interconnecting network (such as QsNet and Easy Bus), are suitable parallel platforms to deal with DFT-based molecular dynamics codes which are intensive applications for both number-
188
P/N 2/2 4/4 8/8 16/16
Total (sec) 924 514 241 194
FFT (sec) 490 251 131 98
GRAHAM (sec) 215 129 44 22
Table 1: Execution times of the different tasks for 20 time-steps of the Fl system on the Feronia platform, for the case P/N = 1. Total is the total execution time, FFT the time spent in the Fourier transform routine, and GRAHAM is the time spent in the orthogonalization procedure.
P/N 2/1 4/2 8/4 16/8 32/16
Total (sec) 1288 627 313 273 143
FFT (sec) 612 294 162 126 51
GRAHAM (sec) 329 155 66 36 61
Table 2: Same as Table 1, for the case of P/N=2
crunching and communication activities. The special care that has been taken in controlling load balancing and the communications costs allowed good speed up till 16 nodes. These results open the way to the possibility of having good scalability figures on larger capability platforms (of the same BC type) which are going to be assembled, both at ENEA and at CAPA-EPFL. References 1. 2. 3. 4.
P.Hohenberger, W.Kohn, Phys. Rev. 136 (1964) B864. R.Car, M.Parrinello, Phys. Rev. Lett. 55 (1985) 2471. W.Kohn and L.J.Sham Phys. Rev. 140 (1965) A1133. L.J.Clarke, I.Stich and M.C.Payne, Comput. Phys. Commun. 72, 14 (1992).
PRAN: SPECIAL PURPOSE PARALLEL ARCHITECTURE FOR PROTEIN ANALYSIS
A.MARONGIU CASPUR E-mail: marongiu@die. ing. uniromal. it P.PALAZZARI, V.ROSATO ENEA Casaccia Research Center, HPCN Project, Rome E-mail: {palazzari, rosato}@casaccia. enea. it Computational genomics is gaming an increasing interest in the scientific community. Genomic and proteomic analysis are mainly based on character processing, so standard processors are not well suited to support the computations required, being the largest part of their silicon devoted to support memory management and floating/fixed point numerical computations. In this work we present the PRAN (PRotein ANah/ser) architecture purposely designed to recognize the occurrences of a given short sequence of characters (m-peptide) within a large proteomic sequence. PRAN has been designed using the automatic parallel hardware generator developed in our research centre in the framework of the HADES project (HArdware DEsign for Scientific applications). Due to its high degree of parallelism exploited within the architecture, PRAN is able to reach high computational efficiency. PRAN has been implemented on a prototyping board equipped with a Xilinx Virtex XV1000 FPGA. Test comparisons with standard commodity off the shelf processors evidenced that PRAN is nearfy one order of magnitude faster.
1
Introduction
The increasing relevance of computational methods in genome sequencing and analysis is compelling the design of new computing architectures. General purpose computational platforms, in fact, are purposely designed to support floating- (or fixed)-point numerical operations, while most of the computational complexity involved in genomic (or proteomic) analysis is based on "character processing". This implies that general purpose commodity-off-the-shelf (COTS) architectures are often deployed, in those contexts, with a very low efficiency. The partial inadequacy of COTS platforms to sustain high computational efficiency on this class of computations has fostered a renewed interest in the design of specialised hardware devices, purposely realised to cope with the algorithmic complexity inherent to character processing ([l]-more extended in [2]). In this work we present the design of a dedicated hardware device, based on the FPGA technology, purposely designed to be used in a field which can be called "linguistic proteomics", i.e. the search of the frequency of occurrence of basic m-peptides in a given protein (or in a whole proteomic set). The developed chip implementing this computational task has been called PRAN (PRotein ANalyzer).
189
190
Given the m-string (representing the m-peptide) and the M-string (representing the protein or the proteome), the exhaustive search for the number of occurrence of all me possible m-strings (whose number is 20m) into a M-string requires mx(M-m)x20m character comparisons. To give a feeling of the complexity of this computation, nearly 9xl0 16 character comparisons are required to search for the frequency of epta-peptides (m=7) inside a proteome with 10 amino acids; wim a sustained power of 109 character comparisons per second (a reasonable number for the current leading edge processor technology), this computation would require 34 months. Our approach is based on an automatic hardware generator which is able to produce a synthesizable VHDL from high level specifications. 2
Finding m-peptides within a proteome
A proteome P is a sequence of characters defined over the alphabet A = {I,F,V,L,W,M,A,G,C,Y,P,T,S,H,E,D,Q,N,K,R} Each character in A represents an amino acid. The typical length of P is M=106-s-107 characters. Peptides are sequences of characters belonging to A; m-peptides are sequences of m characters. Each character aj within P has an occurrence frequency ffa) _ „ number of occurrences of a,
flfr) =
—
~
(1)
M Given a m-peptide mpe Am, its Theoretical Occurrence Frequency TOF(mp) is the product between the extraction probability of mp, supposed contained in the proteome, and its existence probability, i.e.
TOF(mp)=-r
Il f ( a i)
(2)
M-m a,emp The Measured Occurrence Frequency of mp, MOF(mp), is obtained through an experimental measure. All the m-peptides are generated and are searched within the whole proteome; MOF(mp) is the number of matches of mp divided by the whole number of trials, i.e. w^™ x number of occurrences of mp in P MOF(mp)= — = (3) 20m(M-m) It is interesting to determine the m-peptides having MOF(mp) significantly larger than TOF(mp): they are good candidates to be studied by the biologists searching the 'basic lexicon' of proteins. As there is no 'a priori' criterion to determine which are the m-peptides with large value of the ratio , all the 20m m-peptides must be, in principle, analysed TOF(mp)
191
and their MOF(mp) must be computed counting, for each mp, the number of its occurrences in P. m character comparisons are required to check if the m-peptide mp is present in the j * position of P: thus, the determination of MOF(mp) involves 20m(M-m)m character comparisons (all the m-peptides are searched in the whole proteome). Clearly more sophisticated algorithms could be adopted to determine MOF(mp) for all the m-peptides but, due to the very small granularity of the basic operations involved, the overhead to manage conditional statements is significant, thus vanishing the advantage of these algorithms. We refer to the following m-peptides searching algorithm input proteome P of length M, m (order of the peptides), threshold value T output • , MOF(mpK „, all the peptides witfi *—^ > T TOF(mp) begin mp_result = {} ; for each a^eA compute f(ai); while not all the m-peptides have been generated generate a new m-peptide mp; scan P to determine MOF(mp); mc
MOF(mpK„ 2. 1 then insert mp into mp result;
if
TOF(mp) end
The computational intensive part of previous algorithm is contained in the ' s c a n P t o d e t e r m i n e MOF(mp)' statement; it has been decided, in order to enhance the processor capability to deal with this computational problem, to develop a special purpose parallel architecture, called PRAN (PRotein ANalyser), to efficiently implement such a statement. 3
Design Approach
The design approach is based on the automatic Parallel Hardware Generator (PHG) tool developed in ENEA (Italian Agency for New Technology, Energy and Environment) by two of the authors in the framework of the HADES project (HArdware DEsign for Scientific applications). The PHG theoretical framework is described in [4] while the detailed theory is reported in [3]. PHG produces a synthesizable VHDL [5] from high level specifications given by means of affine recurrence equations [6-8]. In order to achieve the final circuit description, PHG performs the following steps:
192 •
parsing of recurrence equations describing the algorithm to be implemented and generation of the intermediate format; details on the language used to describe recurrence equations can be found in [9]; • automatic extraction of parallelism by allocating and scheduling the computations through a space-time mapping [4]. The mapping is represented by an integer unimodular matrix derived through an optimization process [10]. This step produces the architecture of the system expressed as a set of interconnected functional units (data path) managed by a control Finite State Machine (data path controller) which enforces the scheduling; • generation of the synthesizable VHDL representing the architecture determined in the previous step. The VHDL code is then synthesized through the standard Electronic Design Automation tools. We used Synopsys FPGA compiler II to produce the optimized netlist and Xilinx Foundation Express 3.1 to place and route it into the target FPGA. Using the SIMPLE (Sare IMPLEmentation) language [9], the design behaviour has been specified through the following SIMPLE program: /*Input definition*/ Input str[i] 0 < i < M-l Input substr[i] 0 < i < m-l /*Result definition*/ Result res, restree; /•Initialisation equation*/ res[i,j]=l -m+1 < i < M-l j = -l /*match equation for the first (m-l) characters of the m-peptide*/ Eq.l: res[i,j]=res[i,j-1] and (str[i+j] == substr[j]) 0 < i+j < M-l, 0 < j < m-2 /*match equation for the m-th character of the m-peptide. This equation is defined for each akeA*/ Eq.2: restree [i, j ] = res[i,j-l] and (str [i+j ] ==akeA) 0 < i+j < M-l, j=m-l /•Output definition*/ Write(restree[i,j]) 0 < i < M-m, j=m-l
Input definition specifies the input variables along with their validity domain. Result definition specifies the intermediate/final result of the algorithm. The output definition specifies which final results must be produced by the algoritiim. Equation 1 computes the partial result r e s [ i , j ] : r e s [ i , j ] is set to 1 if, starting from position i, the first j characters of the input string match the first j characters in the m-peptide, otherwise r e s [ i , j ] is set to 0. Equation 2 computes the output result r e s t r e e [ i , j ] . Equation 2 has the same meaning of equation 1. The only difference is that in equation 2 the match is checked against each character ak 6 A. The final result r e s t r e e [ i , m - l ] assumes so either the value 'not matched' if at least one of previous m-l characters does not match, or the value ak, i.e. the last character of the m-peptide. Given the initial m-l characters of a m-peptide,
193
{mim2v,mm_i}, r e s t r e e [ i , m - l ] = a^eA means that the m-peptide {mim2,...,mm_i,aic} is contained, starting from the position i, in the input sequence. 4
PRotein ANalyser Architecture
As COTS processors are powerful enough to analyze m-peptides with m ranging up to 6, we designed PRAN to study the case m=7. Moreover the M value has been set equal to 4096 and 5 bits are used to encode the characters. The architecture obtained applying the PHG to the recurrence equations defining the problem is the pipelined structure sketched in figure 1. mp(0)
mp(\)
mp(2)
mp{3)
mp(4)
mp(5)
r - = - D - = - D - = - Q - = - Q - = - D - = -O-i Strln(f)
P0
StrOul(i)
"11111" Figure 1: Pipelined structure implementing the string matching algorithm
The pipeline structure receives the first 6 characters of the epta-peptide from the input mp(0)...mp(5) and the input sequence from the input port Strln. The output character is produced at the output port StrOut. The pipeline structure contains 6 compare blocks, represented by a box with the '=' label. The logical scheme of a compare block is shown in figure 2. As we see, compare blocks are combinatorial circuits setting to 1 their output when the two characters presented at the A and B input ports are equal and the C input port receives the 1 value. Output StrOut, which corresponds to equation 2, is implemented through a multiplexing operation which produces the current string character, if a 7-peptide matched, otherwise returns the special character 'not matched' ("11111"). A
Out
Figure 2: Logical scheme of the 'compare block'
194
The PRAN architecture has been designed to be hosted by a prototyping board (see figure 3) equipped with PCI interface, 4 independent SRAM banks (512Kx32 each), 1 Xilinx Virtex XV1000 FPGA, 2 I/O ports (8 bit width). SRAM 4->
To host '
m u
z
SRAM 4-+
J
Input & Control Port
Figure 3: Block diagram of the prototype board hosting the PRAN architecture
In order to increase the parallelism, and according to the board I/O constraints, during the analysis the whole proteome P is divided into N not-overlapping strings which feed N replicas of the previous pipelined structure. Due to the I/O constraints, in our implementation of PRAN we chose N=12, i.e. we search for 20 7-peptides simultaneously on 12 strings of length 4096. The architecture of PRAN is sketched in figure 4, being the blocks labelled 'pipe compare' depicted in figure 2. As we see in figure 4, PRAN receives the 12 input strings through 60 input lines, being each character encoded through 5 bit. In our test-bed board, the input strings are stored in 2 SRAM memory banks, each one organized as 32x512Kbit. Two other SRAM banks are used to collect the output of PRAN; the output consists of 12 characters, each one coding if one of the 20 7-peptides matched and, if a match is detected, which was the 7-peptide that matched. PRAN drives the lines, and the addresses, to control the 4 SRAM memory banks. Through a dedicated input port the first 6 characters of the 7-peptide to be checked are loaded into 6 internal registers (R0,...,R5). These registers are connected to the inputs of all the 12 'pipe compare' blocks. Constrained at the speed grade of the FPGA we used (XV1000 6), the synthesized design is clocked at a frequency fck= 40 MHz. The scheme of the algorithm to analyse P is the following begin mp_result = {}; for each a^eA compute f(ai); while not all the (m-1)-peptides have been generated generate a new (m-1)-peptide m_lp; for k=l to M/(4096*12) DMA input data to board; use PRAN to analyse the 12 sequences containing 4096 characters; DMA result data from board; compute MOF(mp);
.f MOF(mp) ^ T then TOF(mp)
insert mp into mp_result;
195 Input & Ctrl Port
-y-
RO
Rl
LI
R2
R3
I J
R4
R5
SRAM addresses &. control signals
Addr(4xl9)
Ctrl
StrInO 5 /
y
OutO
5/
Outl
5
,.
-y-*\ Slrln! 5
¥-
l
-P-
pipe compare 11
•y-
Figure 4: PRAN architecture
5
Results
In order to test the performance of the implemented PRAN prototype, we searched for the 7-peptides on the proteome constituted by all the protein sequences contained in the yeast Saccharomyces cerevisiae. We implemented the searching algorithm on three different systems, based respectively on a Sun UltraSparc 60 (450 MHz), on an Alpha EV6.7 (667 MHz) and on a Pentium II (333 MHz) connected to the development board equipped with the PRAN prototype. The implementation of the 7-peptides search on the UltraSparc and Alpha systems fully exploits the parallelism of the data bus, allowing the simultaneous comparison of a number of sub-strings equal to the number of characters that can be encoded on the data word. To fix the ideas, the search is performed simultaneously on 6 sub-strings on systems equipped with a 32 bit wide data bus, because 6 characters (corresponding to 30 bit) can be encoded on a 32 bit word. Table 1 summarizes the results, reporting for each system a) the time required to check the number of occurrences of one 7-peptide on a proteome with 10 characters, b) the time to search all the 7-peptides on the same string c) the speed-up of the PRAN architecture vs the systems.
196 Sun UltraSparc 60 Time to check one 7-peptide over 106 characters (sec) Time to check all the 7-peptides over 10' characters (days) Speed-UP=t„deni/tp|UN
Alpha EV6.7
Pentium II + PRAN
0.0322 478
0.0119 176
0.0021 32
15
5.6
1
Table 1: test results The exploitation of the parallelism inherent to the problem has allowed to sustain, with the PRAN processor, significantly higher performances with respect to those attainable with conventional high-end general purpose processors. This device is going to be used in the search of genomic structures called "tandem repeat" (repeated sub-sequences containing a few bases). For this problem, being 3 the number of bits needed to encode the basic character, more functional units can be added, thus increasing the overall length of the treated tandem repeat structure. References 1. Proceedings of the SIMAI Symposium on "Formal Methods for HW/SW design for Grand Challenge Scientific Applications" - Ischia (Italy), June 2000 2. Special Issues of the Computer Physics Communication Journal on "Formal Methods for HW/SW design for Grand Challenge Scientific Applications" Guest Editors P.Palazzari, V.Rosato - to appear in 2001 3. A.Marongiu, "Hardware and Software High Level Synthesis of Affine Iterative Algoridims", Ph.D Thesis in Electronic Engineering, "La Sapienza" University of Rome, February 2000. 4. A.Marongiu, P.Palazzari, "Automatic Mapping of System of Affine Recurrence Equations (SARE) onto Distributed Memory Parallel Systems", IEEE Trans, on Soft. Eng., 26, (2000), 262. 5. IEEE standard VHDL language reference manual. IEEE std. 1076-1993 6. C.Mongenet, P.Clauss, G.R.Perrin, "Geometrical Tools to Map System of Affine Recurrence Equations on Regular Arrays", Acta Informatica, 31, (1994), 137 7. K.H.Zimmermann, "Linear Mapping of n-dimensional Uniform Recurrences onto ^-dimensional Systolic Arrays", Journal of VLSI Signal Processing, 12, (1996), 187. 8. A.Darte, "Regular Partitioning for Synthesizing Fixed-Size Systolic Arrays". Integration, The VLSI Journal, 12, (1991), 293. 9. A. Marongiu, P. Palazzari, L. Cinque and F. Mastronardo, "High Level Software Synthesis of Affine Iterative Algorithms onto Parallel Architectures". Proc. of the HPCN Europe 2000. May 2000 Amsterdam, The Netherlands. 10. A. Marongiu, P. Palazzari, "Optimization of Automatically Generated Parallel Programs", Proc of the 3rd IMACS International Multiconference on Circuits, Systems, Communications and Computers (CSCC'99) - July 1999, Athens
DESIGN OF A PARALLEL A N D DISTRIBUTED SEARCH ENGINE
WEB
S. ORLANDO 0 , R. PEREGO*, F. SILVESTRr # °Dipartimento di Information, Universita Ca' Foscari, Venezia, Italy "Istituto CNUCE-CNR, Pisa, Italy ' Dipartimento di Informatica, Universita di Pisa, Italy This paper describes the architecture of MOSE (My Own Search Engine), a scalable parallel and distributed engine for searching the web. MOSE was specifically designed to efficiently exploit affordable parallel architectures, such as clusters of workstations. Its modular and scalable architecture can be easily adjusted to fulfill the bandwidth requirements of the application at hand. Both task-parallel and data-parallel approaches are exploited within MOSE in order to increase the throughput and efficiently use communication, storing and computational resources. We used a collection of html documents as a benchmark and conducted preliminary experiments on a cluster of three SMP Linux PCs.
1
Introduction
Due t o the explosion in the number of documents available online today, Web
Search Engines (WSEs) have become the main means for initiating navigation and interaction with the Internet. Largest WSEs index today hundreds of millions of multi-lingual web pages containing millions of distinct terms. Although bigger is not necessarily better, people looking the web for unusual (and usual) information prefer to use the search engines with the largest web coverage. This forced main commercial WSEs to compete for increasing the indexes. Since the cost of indexing and searching grows with the size of the data, efficient algorithms and scalable architectures have to be exploited in order to manage enormous amount of information with high throughputs. Parallel processing thus become an enabling technology for efficiently searching and retrieving information from the web. In this paper we present MOSE, a parallel and distributed WSE able to achieve high throughput by efficiently exploiting a low cost cluster of Linux SMPs. Its expansible architecture allows the system to be scaled with the size of the data collection and the throughput requirements. Most of our efforts were directed toward increasing query processing throughput. We can think of a WSE as a system with two inputs and one output. One input is the stream of queries submitted by users. The other input is the read-only database, which contains the index of the document collection. The WSE process each query of the stream by retrieving from the index the references to the I most
197
198
relevant documents. Such set of I references is then put on the output stream. The main parallelization strategies for a WSE are thus: Task parallel. Since the various queries can be processed independently, we can consider query processing an embarrassingly parallel problem. We can thus exploit a processor farm structure with a mechanism to balance the load by scheduling the queries among a set of identical workers, each implementing a sequential WSE. Data parallel. The input database is partitioned. Each query is processed in parallel by several data parallel tasks, each accessing a distinct partition of the database. Query processing is in this case slightly heavier than in the previous case. Each data parallel task has in fact to retrieve from its own partition the locally most relevant / references. The final output is obtained by combining these partial outputs, and by choosing the I references which globally result to be the most relevant. Task have Data work
+ Data parallel. A combination of the above two strategies. We a processor farm, whose workers are in turn parallelized using a parallel approach. The farming structure is used to balance the among the parallel workers.
The modular architecture of MOSE allowed us to experiment all the three strategies above. The third parallelization strategy, which combines Task and Data parallelism, achieved the best performances due to a better exploitation of memory hierarchies. The paper is organized as follow. Section 2 introduces WSE and Information Retrieval (IR) principles, and surveys related work. Section 3 describes MOSE components, discusses parallelism exploitation, and shows how MOSE modular and scalable architecture can be adjusted to fulfill bandwidth requirements. The encouraging experimental results obtained on a cluster of three Linux SMPs are shown in Section 4, while Section 5 draws some conclusions. 2
W S E and IR Principles
A typical WSE (see Figure 1) is composed of the spidering system, a set of Internet agents which in parallel visit the web and gather all the documents of interest, and by the IR core constituted by: (1) the Indexer, that builds the Index from the collection of gathered documents, and, (2) the Query Analyzer, that accepts user queries, searches the index for documents matching the query, and return the references to these documents in an understandable
199
Figure 1. Typical organization of a WSE.
form. Query results are returned to users sorted by rank, a kind of relevance judgment that is an abstract concept largely linked to users taste. Ranking is performed on the basis of an IR model that allows to represent documents and queries, and to measure their similarity. In general, as the size of the indexed collection grows, a very high precision (i.e. number of relevant documents retrieved over the total number of documents retrieved) has to be preferred even at the expense of the recall parameter (i.e. number of relevant documents retrieved over the total number of relevant documents in the collection). In other words, since users usually only look at the first few tens of results, the relevance of these top results is more important than the total number of relevant documents retrieved. In order to grant high precision and computational efficiency, WSEs usually adopt a simple Weighted Boolean IR model enriched with highly effective ranking algorithms which consider the hyper-textual structure of web documents 1 ' 2 . Moreover, due to its compactness, most WSEs adopt an Inverted List (IL) organization for the index. An IL stores the relations among a term and the documents that contain it. The two main components of an IL index are: (1) the Lexicon, a lexicographically ordered list of all the interesting terms contained in the collection, and, (2) the Postings lists, lists associated to each term t of the Lexicon containing the references to all the documents that contain t. Many large-scale WSEs such as Google, Inktomi and Fast, exploit clusters of low-cost workstation for running their engines, but, unfortunately, very few papers regard WSE architecture design 1,3 , since most developments were done within competitive companies which do not publish technical details. On the other hand, many researchers investigated parallel and/or distributed IR systems 4 ' 5 ' 6 ' 7 ' 8,9 focused on collections of homogeneous documents. Lin and Zhou7 implemented a distributed IR system on a cluster of workstations, while Lu 8 , simulated an interesting distributed IR system on a Terabyte collection, and investigated various distribution and replication strategies and their impact on retrieval efficiency and effectiveness.
200 SabcollectkHis
Indexing
Figure 2. Indexing phase.
3
M O S E Structure
The IR core of MOSE is composed of the Indexer and the Query Analyzer (QA) modules. In this paper we only briefly surveys indexing issues, and focus our attention on the QA whose functionalities are carried out by two pools of parallel processes: Query Brokers (QBs) and Local Searchers (LSs). MOSE parallel and distributed implementation exploits a data-parallel technique known as document partitioning . The spidering phase returns p subcollections of documents with similar sizes. The subcollections are then indexed independently and concurrently by p parallel Indexers (see Figure 2). The result of the indexing phase is a set of p different indexes containing references to disjoint sets of documents. The p indexes are then taken in charge by a data-parallel QA whose task is to resolve user queries on the whole collection. To this end the QA uses k QBs and p LSs. The k QBs run on a front-end workstation, and fetch user queries from a shared message queue. Every fetched query is then broadcast to the associated p LSs (workers), possibly running on different workstations. The p LSs satisfy the query on the distinct subindexes, and return to the QB that submitted the query the first I references to most relevant documents contained within each subcollection. The QB waits for all the / • p results and chooses among them the / documents with the highest ranks. Finally, such results are returned to the requesting user. Figure 3 shows the logic structure of the MOSE architecture. A QB, along with the p associated LSs, implements a data parallel worker which concurrently serve the user queries. In order to manage concurrently more queries and to better exploit LSs' bandwidth, k QBs are introduced within a QA. System performances can be furthermore increased by replicating the QA in n copies. All the parallelization strategies depicted in Section 1 can be thus realized by choosing appropriate values for n, k, and p. A pure task parallel approach
201
Figure 3. Structure of MOSE Query Analyzer.
corresponds to p = 1, while n > 1 and/or k > 1. By choosing p > 1, n = 1 and fc = 1 we obtain a pure data-parallel implementation. A hybrid task + data parallel strategy is finally obtained for p > 1, while n > 1 and/or k > 1. Indexer. The Indexer has the purpose of building the index from the gathered web documents. The indexing algorithm used is a parallel version of the Sort Based algorithm which is very efficient on large collections due to the good compromise between memory and I/O usage 2 . Moreover, the index built is Full Text and Word Based. The Lexicon is compressed exploiting the common prefixes of lexicographically ordered terms (Shared Prefix Coding), while the Postings lists are compressed by using the Local Bernoulli technique 2 . MOSE parallel Indexer exploits the master/worker paradigm and standard Unix SysV communication mechanisms (i.e. message queues). Since each subcollection of web documents is indexed independently (and concurrently on different workstations), the current Indexer implementation exploits parallelism only within the same SMP architecture. The master process scans the subcollection, and sends the reference to each document (i.e. the file offset) along with a unique document identifier to one of the worker processes on a self-scheduling basis. The workers independently read each assigned document from the disk and indexes it. When all documents have been processed, the workers write their local indexes to the disk, and signal their completion to the master. At this point the master merges the local subindexes in order to create a single index for the whole subcollection. A distributed implementation of the Indexer could be easily derived, but should require all the processing nodes to efficiently access the disk-resident subcollection, and that at least a single node can access all the subindexes during the merging phase. Query Broker. Each QB loops performing the following actions: Receipt and broadcasting of queries. Independently from the mechanism ex-
202
ploited to accept user queries (e.g., CGI, fast CGI, PHP, ASP), user queries are inserted in a SysV message queue shared among all the QBs. Load balancing is accomplished by means of a self scheduling policy: free QBs access the shared queue and get the first available query. Once a query is fetched, the QB broadcasts it to its p LSs by means of an MPI asynchronous communication. Receipt and merge of results. The QB then nondeterministically receives the results from all the LSs (i.e., p lists ordered by rank, of I pairs document identifier, and associated rank value) . The final list of the / results with the highest ranks is than obtained with a simple 0(1) merging algorithm. Answers returning. The list of / results is finally returned to the CGI script originating the query that transforms document identifiers into URLs with a short abstract associated, and builds the dynamic html page returned to the requesting user. Local Searcher. LSs implement the IR engine of MOSE. Once a query is received, the LS parses it, and searches the Lexicon for each terms of the query. Performance of term searching is very important for the whole system and are fully optimized. An efficient binary search algorithm is used at this purpose, and a Shared Prefix Coding technique is used to code the variable length terms of the lexicographically ordered Lexicon without wasting space 2 . Minimizing the size of the Lexicon is very important: a small Lexicon can be maintained in core with obvious repercussions on searching times. LS exploit the Unix mmap function to map the Lexicon into memory. The same function also allows an LS to share the Lexicon with all the other LS that run on the same workstation and process the same subcollection. Once a term of the query is found in the Lexicon, the associated posting list is retrieved from the disk, decompressed, and written onto a stack. The LS then processes bottomup query boolean operators whenever their operands are available onto the top of the stack. When all boolean operators have been processed, the top of the stack stores the final list of results. The / results with the highest ranks are then selected in linear time by exploiting a max-heap data structure 2 . Finally, the I results are communicated to the QB that submitted the query. 4
Experimental Results
We conducted our experiments on a cluster of three SMP Linux PCs interconnected by a switched Fast Ethernet network. Each PC is equipped with two 233MHz Pentiumll processors, 128 MBytes of RAM, and an ULTRA SCSI II disk. We indexed 750.000 multi-lingual html documents contained in the CDs of the web track of the TREC Conference and we built both a monolithic
203 Task Parallel: one va two WSs (2 QBs per L.S, 5000 emeries)
Task Parallel vs. Hybrid (2 QBs per LS, 5000 queries) T P - ^ TP + DP — * - -
\ s 1 E
"\. ,,._ 10000 number of QAs (n)
(a)
number of LSs (p * n)
(b)
Figure 4. Results of the experiments conducted.
index (p = 1) and a partitioned one (p = 2). The monolithic index contains 6.700.000 distinct terms and has a size of 0.96 GBytes (1.7 GBytes without compression), while each one of the two partitions of the partitioned index occupy about 0.55 GBytes. The queries used for testing come from an actual query log file provided by the Italian WEB Search Company IDEARE S.p.A. We experimented Task-Parallel (TP), and hybrid (TP + DP) configurations of MOSE. We mapped all the QBs on a single workstation, while the LSs were placed on one or both the other machines. Independently of the configuration used (one or two index partitions), two QBs were introduced (k = 2). Figure 4.(a) reports the average elapsed times, i.e. the inverse of the throughput, required to process each one of 5000 queries for the TP case (p = 1) as a function of n, i.e. the number of QAs exploited. The two curves plotted refer to the cases where the LSs were mapped on one or two SMP machines. We can see that when two QAs are used they can be almost indifferently placed on one or two SMP machines, thus showing the efficacy of the sharing mechanisms used. On the other hand, as we increase the number of QAs, the difference between exploiting one or two machines increases as well. We can also observe that it is useful to employ more QAs than the available processors. Figure 4.(b) compares the TP solution with the hybrid one (TP + DP). Testing conditions were the same as the experiment above. In the case of the hybrid configuration, all the LSs associated with the same partition of the index were placed on the same workstation in order to allow the LSs to share the lexicon data structure. The better performance of the hybrid approach is evident. Superlinear speedups were obtained in all the TP + DP tests. They
204 derive from a good exploitation of memory hierarchies, in particular of t h e buffer cache which virtualize the accesses t o the disk-resident posting lists. 5
Conclusions
We have presented t h e parallel and distributed architecture of MOSE, and discussed how it was designed in order to efficiently exploit low-cost clusters of workstations. We reported the results of preliminary experiments conducted on three S M P workstations. T h e results highlighted the greater performances resulting from exploiting a hybrid Task + Data parallelizafcion strategy over a pure Task-parallel one. There are a lot of i m p o r t a n t issues we plan t o investigate in the near future. The most i m p o r t a n t is performing an accurate testing of M O S E on larger clusters and document collections in order t o analyze in greater detail the scalability of the different parallelization strategies. Fastest interconnection network such as Myrinet have also t o be tested. Moreover, we are interested t o study query locality and the effectiveness of caching their results within QBs, and "supervised" document partitioning strategies aimed at reducing the number of index partitions needed t o satisfy each query. References 1. S. Brin and L. Page. The anatomy of a large-scale hypertextual web search engine. In WWW7 / Computer Networks, vol. 1-7, pages 107-117, April 1998. 2. I. H. Witten, A. Moffat, and T. C. Bell. Managing Gigabytes - Compressing and Indexing Documents and Images. Morgan Kaufmann Publishers Inc., 1999. 3. Fast Search &: Transfer ASA. The fast search server - white paper. Technical report, Fast Search & Transfer ASA, December 1998. 4. LA. Macleod, T.P. Martin, B. Nordin, and J.R. Phillips. Strategies for building distributed information retrieval systems. Information Processing & Management, 6(23):511-528, 1987. 5. T.P. Martin, I.A. Macleod, J.I. Russell, K. Lesse, and B. Foster. A case study of caching strategies for a distributed full text retrieval system. Information Processing & Management, 2(26), 1990. 6. F.J. Burkowski. Retrieval performace of a distributed text database utilizing a parallel process document server. In Proc. of the Intern. Symp. On Databases in Parallel and Distributed Systems, Doublin, Ireland, July 1990. 7. Z. Lin and S. Zhou. Parallelizing I/O intensive applications for a workstation cluster: a case study. Computer Architecture News, 5(21), December 1993. 8. Zhihong Lu. Scalable Distributed Architectures for Information Retrieval. PhD thesis, University of Massachussets Amherst, 1999. 9. B. Cahoon, K.S. McKinley, and Z. Lu. Evaluating the performance of distributed architectures for information retrieval using a variety of workload. IEEE Transactions on Information Systems, 1999.
ACTIVE CONTOUR BASED IMAGE SEGMENTATION: A PARALLEL COMPUTING APPROACH
V. POSITANO, M.F. SANTARELLI, A. BENASSI, C.N.R. Institute of Clinical Physiology, Via Moruzzi, 1 - 56124, Pisa, Italy
[email protected]. it C. PEETRA,
NETikos Web Factory, Via Matteucci 34b, Pisa, Italy L. LANDINI Department of Information Engineering: EIT, Via Diotisalvi, 2 - 56126 Pisa, Italy Segmentation is a fundamental task in image processing, but its hard computational requirements represent a not easily defeated drawback, mainly for tasks in which real time computing is a primary requirement, such as critical medical applications. In the present paper a parallel algorithm for image segmentation will be proposed, which allows fast convergence for active contour algorithm. A parallel computing approach to this problem has led to develop a multithreaded application based upon a parallel algorithm that allows reducing processing time. The algorithm has been implemented on a shared memory multiprocessor machine, using ANSI C++ language and Posix Threads libraries, in order to exploit code portability. Two basic schemes have been proposed for parallelism development and performance issues have been analyzed for both implementations.
1
Introduction
Today available medical imaging systems are able to collect a massive amount of data in short time. As an example, the number of images acquired by a magnetic resonance scanner in a typical cardiovascular examination can vary from 200 to 1000. Due to the large amount of data provided by acquisition devices, the traditional way to perform data analysis (i.e. sequential examinations of images and mental three-dimensional reconstruction) becomes ineffective. So that, software tools for automatic segmentation and quantitative analysis are needed to fully exploit the features of the today available medical devices [1,2]. One of the most important drawbacks in image segmentation is the very high computational complexity of the related algorithms, that require a large amount of computing time, especially for analysis of large medical images or continuous motion tracking applications; the use of a parallel computing approach can answer to this problem [3], as some of the authors showed in a previous works [4,5]. The
205
206
need of fast parallel algorithms for image segmentation emphasizes the meaning of new parallel computing solutions from both software and hardware perspective. In this paper we propose a parallel implementation of an active contour algorithm, also known as "snake" algorithm, for the automatic image segmentation. The snake algorithm was demonstrated effective in medical image segmentation [1,2,6,7], because it preserves the continuity of the segmented anatomical tissues and it can be used to dynamically track the motion of moving structures, as the human hearth. A lot of improvements of the original active contour algorithm [8] were proposed in the literature. With the aim of taking advantage of the processing power offered by a shared memory multiprocessor, we have implemented a multithreaded version of the "Greedy Algorithm" proposed by Williams and Shah [9] rearranging its basic idea to match the logics of a parallel computing approach. We chose to use a shared memory multiprocessor machine due to its programming facilities about load balancing issues and its flexibility about "allpurpose" applications; on the other hand, the need for "ad hoc" algorithms to exploit the inherent parallelism of an application is a central requirement, in order to exploit all the available processing power. The Posix threads [10] freely available libraries allow implementing such application increasing both code portability and the use of all available system resources in a "built-in" optimized way. Moreover, SMP machines are often used as console of medical devices (e.g. NMR, PET), so that the development of algorithms able to exploit SMP platforms it is of great interest. 2
Algorithm description
Active contour algorithms literature starts from the original Kass, Witkin and Terzopoulos [8] solution, which uses the variational calculus technique. The Williams and Shah [9] approach was based on dynamic programming; its goal was to enhance the existing method, preventing some problems including numerical potential instability. The last algorithm we referred to is the most computationally efficient one, so it has been chosen as a starting point to develop our parallel solution. An active contour is defined as a closed curve s(x,y) that moves through the spatial domain of an image to minimise the following energy functional: i
E
= J { £ t a t [*(x,y
)]+£„,
E = 1 - cL (x,vj +fis [x,y\ Hit
[s(x,
y )]}
dxdy
(2.1) (2.2)
/*
where Eint represents the internal energy of the contour, due to bending or discontinuities; it contains a first-order term, which increases where there is a gap in the curve, and a second-order continuity term which increases where the curve is
207 bending rapidly. The "Greedy" algorithm prevents the shrinking of the curve, encourages the even spacing of the points and minimizes the formula complexity that has been used to give a reasonable estimate of curvature. The elastic parameters a and |3 are to be chosen to realize the utmost homogeneous behavior for contour evolution; guidelines for the most effective choice are given in the Williams and Shah's paper [9]. Eal is the image energy, which depends on various features: the ones presented by Kass et al. are lines, edges and terminations. Given a gray-level image I(x,y) viewed as a function of continuous position variables (x, y), typical external energies designed to lead an active contour toward step edges are:
^H
V /
M
2
or Eex,=^(Ga(x,y)*i(x,y)f
where Ga(x,y) is a two-dimensional Gaussian function with standard deviation crand V is the gradient operator. In the "Greedy" algorithm, the snake is treated as a circular array of coordinates, assuming to take the same point as first and last one of the contour. The algorithm is an iterative one; during each iteration, for each snake point Si the energy function is computed for the image pixel corresponding to the current snake point location and for its m neighbors (figure 1, left). The location having the smallest energy value (say m8) is chosen as the new position of Si (figure 1, right). Si-1 has already been moved to its new position during the current iteration. Its location is used with mat of each of the proposed locations for Si to compute me first-order continuity term.
Figure 1: The greedy algorithm
The location of Si+1 has not yet been moved. Its location, along with that of Si-1, is used to compute the second-order constraint for each point in the neighborhood of Si. For this reason SO is processed twice, one as the first point in the list, one as the
208
last point. This helps to make its behavior more like that of the other points. The iterative process continues until the number of points moved at last iteration isn't small enough to discourage further computations. A Pseudo-code for "Greedy" algorithm is given in [9]. 3
A parallel solution
Our basic idea is to divide up the entire snake in some portions that are to be elaborated by different "processes". As described in the previous section, in the "Greedy" algorithm the behavior of a snake point depends only upon the couple of neighboring points. This allows to create portions of snake sharing only the edge points. In figure 2, a 12-points snake (SO, ..., S l l ) is divided in three portions (c0,..,c4 c0,..., c4 c0,...,c4). For each iteration, every snake portion is updated by a different process (A, B, C processes). The edge points are elaborated by two different processes because they belong to two snake portions. As example, in figure 2 the snake point SO is used by both A and C processes (aO and c4 points). At the iteration end, the first point of each snake portion is updated with the value of the last one in the previous portion, in our example the aO point is update by the c4 point. In this way, each edge point is treated as the single (first and last) edge point in the sequential solution, as described in section 2. This allows obtaining a more homogeneous snake behavior. The overlapping points introduce a computational complexity overhead; in fact the original complexity of the algorithm was 0(n*m), where n is the number of snake points and m is the neighborhood area (see figure 1); the parallel algorithm, instead, must elaborate n + k points for the snake, considering that if A: is the number of snake portions that have been thought to evolve each on its own, k is also the number of overlapping points. The introduction of overlapping points responds to the need for dividing the snake and distributing the work between processes with the minimum requirement for communication and synchronization between processes, without loosing the continuity of the contour; in fact, a couple of overlapping points must be exchanged between each pair of neighboring processes at each iteration of the algorithm to let the snake evolving as a whole. The segmentation of the snake and the separated evolution of each snake's part introduce an unavoidable "error", that is a difference between the snake deformed as a whole (by the sequential algorithm) and the one deformed in an autonomous way, portion by portion (by the parallel algorithm). In order to reduce this kind of error, we vary at each new iteration the points where the breaking up of the snake takes place. This avoids the persistence of the error in the same area, "spreading" instead the error all along the entire contour. Some validation tests have proven that the error tends to disappear when the number of iterations is large enough.
209
Figure 2. Snake partitioning.
4
Multithreaded implementation of the parallel algorithm
We chose the multithreaded programming paradigm for the algorithm's implementation in order to exploit the "lightness" of threads in comparison with real processes. The absence of a real context switching during the commutation of the threads that are executing on various processors, and the address space sharing between threads, is a good feature for low-level synchronization machineries effectiveness in a shared memory environment. Furthermore such a choice allows to transparently gaining good scalability features, due to me underlying automated "hardware" load balancing mechanism. We have proposed two alternative schemes for parallelism exploitation: the first one (Al) includes parallel procedures (threads) being called repeatedly with each new iteration of the algorithm. This allows to reduce the need for communication and synchronization between threads, charging the parameter passing mechanism with those tasks; nevertheless the algorithm suffers from the continuous switching between parallel sections of code and short but unavoidable sequential parts that have the task to prepare the parameters for the call of the next iteration's parallel procedures. The second scheme (A2), instead, has parallel procedures that can be execute without being interrupted for all the needed iterations. In A2 scheme, we need to provide a communication and synchronization mechanism between threads, which allows exchanging the edge, points between threads at each new iteration. This scheme allows exploiting the parallelism much better then the first one, but it someway requires a communication overhead. The overhead is less invalidating the
210
algorithm performances if it can be redeemed with longer single iterations (larger snake portions). 5
Results and conclusions
The algorithms have been tested on a Sun Ultra Sparc 4450 (4 UltraSparc II 450 Mhz processors, 4Mb L2 cache, 2Gb Ram.) using Solaris 8 as Operative System. The two algorithms previous described (i.e. Al and A2) have been tested by changing three parameters: the total number of snake points, the number of iterations and the number of parts in which the snake is divided, i.e. the number of concurrent threads. The total number of snake points is related to the distance between two adjacent points and consequently to the size of the smallest image features segmented by the algorithm. The number of iterations is related to the maximum range in which the image features can be detected. As example, in table 1 the processing times relevant to 300 algorithm iterations for Al, A2 and the sequential (S) algorithms are showed. A 1024x1024 image was used. This test was performed splitting the snake in 4 parts. Points Al time A2 time S time Speed-up A1 Speed-up A2
300 5.15 5.72 13.92 2.70 2.43
500 8.24 9.27 22.82 2.77 2.46
700 10.55 11.72 32.17 3.05 2.74
900 12.85 12.83 41.20 3.20 3.21
1200 16.50 16.98 54.86 3.32 3.22
1600 21.22 21.48 73.19 3,35 3.40
2000 36.13 27.66 91.43 3.23 3.30
2600 40.39 34.53 118.8 3.29 3.44
Table 1: processing time (sec.) and speed-up for the two algorithms (4 threads, 300 iterations, m=9)
Table 1 shows that the performances of A2 get better when the snake has many points, because each iteration is longer and the synchronisation time, that is a constant quote, can be better redeemed. In a second test, the algorithm complexity was fixed setting at a constant value the product between the number of iterations and the number of snake points. The performance of the two algorithms was tested changing the thread number. As showed in figure 3, the algorithm performance increases with the number of snake point, because the weight of the computation time related to the overlapping points decreases when the snake portion size increases.
211 H 4 threat •Stfiisa* 3$y~ Q16threads _ a32threads 3-^—
i 266256 512/12B 1024*4 2048/32 4096/16
128/512 256/256 51S1SB 102*64 204932 4096'16
pofcWIterdiGns
poinHAtrndions
Figure 3: Speed-up for Al and A2 algorithms (m=9).
Using a high number of threads with both algorithms can introduce a communication overhead, much more visible if the snake is composed by a small number of points. The same amount of data exchange can be spread upon larger portions, enhancing the performances, if each portion is large enough. On the other hand, a large number of threads minimizes the loss of performance due to the thread synchronization at the end of each iteration. That is, at the end of each iteration, all threads have to wait for the slowest one. Increasing the number of threads, the difference among execution times will be reduced. points/Iterations 128/512 20 15 E 10 » E 5
w—•
"
512/128
1024/64
•—•
2048/32
4096/16
..-»•••...y5* 'N^* . - J T - V ^ " * •s*-^?^
2 1-5
1
-* ----"
|
- -10 •a $ -15 M-20 -25 -30
256/256
.•••V"
•'-
yr
- - • - - 4 threads • — o tn reads - -s - 16 threads - -x- - 32 threads
»"
Figure 4: Speed improvement for A2 algorithm respect to Al (m=9).
In figure 4, the speed improvement in percentage obtained using the A2 algorithm instead of the Al one is showed. The algorithm Al works better for a little number of points, the algorithm A2 allows better performance for a big one. For a number of threads equal to 8, the A2 algorithm get better for any number of points/iterations. From figure 3, it is clear that a thread number equal to 8 is a good
212 compromise in order to obtain good performances for every value of point/iterations index. In conclusion, real time image processing and motion tracking applications can get advantage from the use of parallel algorithm. An algorithm for image segmentation using the active contour technique was proposed, that have proven to be suitable for shared memory multiprocessors, responding to the need of speeding up segmentation. Moreover, the use of multi-purpose machine and portable libraries allows to spread the use of parallel computing technique on me medical community using the resources available in a typical medical environment. References
1. Santarelli M. F, Positano V., Landini L., Benassi A. A New Algorithm for 3D Automatic Detection and Tracking of Cardiac Wall Motion. Computers in Cardiology , IEEE, Los Alamitos, 1999; pp. 133-6. 2. Ayache, I. Cohen, and I. Herlin. Medical image tracking. In A. Blake and A. Yuille, eds., Active Vision, Chapt 17, MIT-Press, 1992. 3. Stacy M., Hanson D., Camp J. and Robb R. A., High Performance Computing in Biomedical Imaging Research. Parallel Computing 24:9(1998) 1287-1321. 4. Positano V., Santarelli M. F., Landini L., Benassi A. Using PVM on computer network to perform fast pre-processing of large medical data set, Par. Comp.Fund. and Appl. Proceeding of ParCo99, ICP.pp. 185-192 5. Positano V., Santarelli M.F., Landini L., Benassi A., Fast and Quantitative Analysis of 4D cardiac images using a SMP architecture. PARA'98, Lect. Notes in Comp. Science, No. 1541, Springer-Verlag, 1998, pp 447-451. 6. Cohen L.D., and Cohen I. Finite element methods for active contour models and balloons for 2D and 3D images. IEEE Trans On Pattern Analysis and Machine Intelligence, 15:11,Nov. 1993.pp. 1131-1147. 7. Avedisijan A. et. alt., CAMRA: Parallel application for segmentation of left ventricle (LV) in short cardiac axis MR images. Med. Imag. Understanding and Analysis, July 1999. 8. Kass M., Witkin A., Terzopoulos D. Active contours models. Int. J. Comp. Vision., 1987, pp. 321-331. 9. Williams D. J., Shah M. Fast Algorithm for Active Contours. Image Understanding, vol. 55, January, pp. 14-26, 1992. 10. Sun Microsystem, "Multithreaded Programming Guide", 2000.
PARALLELIZATION OF A N U N S T R U C T U R E D FINITE V O L U M E SOLVER FOR T H E M A X W E L L EQUATIONS J. RANTAKOKKO AND P. EDELVIK Uppsala University, Information Technology, Department of Scientific Box 120, SE-75104 Uppsala, Sweden E-mail:
[email protected] Computing,
An unstructured finite volume solver in 3D has been parallelized with OpenMP. The code is used by the Swedish industry for electromagnetic computations. Results from a grid around a small aircraft show good scaling on a SUN Enterprise 6000 server with 16 processors. A parallelization strategy for a distributed memory model is also presented and some preliminary results are given.
1
Introduction
The Maxwell equations are a mathematical formulation of propagating electromagnetic waves. The equations describe phenomena such as scattering of radar signals from airplanes or radiation from mobile phones. Modeling these phenomena for high frequencies is computationally very demanding, especially on complex geometries in 3D. The Maxwell equations can be solved very efficiently with a finite difference scheme on a staggered grid using the classical FD-TD method proposed by Yee5. Also, the structured grid makes it easy to parallelize the computations. Very large and realistic problems can be solved with this method. The drawback with structured grids is that it is difficult to model complex geometries accurately. Unstructured grid methods such as finite element and finite volume solvers can better resolve the details in the geometry but are in general less efficient than a structured grid solver. A remedy is then to combine the unstructured and structured grid methods in a hybrid method, i.e. use an unstructured grid near the object and then connect this to a structured grid in the outer region. In this paper we will describe the parallelization of an unstructured finite volume solver. The solver will be used as one part in a hybrid solver as described above but can also be used as a stand alone solver. Due to the complex and unstructured data dependencies we have chosen to use OpenMP for the parallelization but a strategy for a distributed memory model using MPI will also be discussed.
213
214
2
Finite volume solver
The finite volume solver is based on the integral formulations of Faraday's and Ampere's laws. The geometry is discretized with an unstructured staggered grid using tetrahedrons in the primary grid, generated by a Delaunay grid algorithm. A dual grid, the Dirichlet tessellation, is then constructed with nodes in the centers of each tetrahedron. The electric field variables reside normal to the dual faces and in each primary node, while the magnetic field variables are located normal to the triangular faces and in each dual node. Additional variables are used on the edges of the two grids. Furthermore, the time derivatives are approximated with an explicit time-stepping scheme, the third-order Adams-Bashforth method. The calculations are organized such that as much as possible is computed initially. The update of the field variables is reduced to matrix-vector and vector-vector operations. The matrices are stored in compress sparse row format. For further details on the solver see Edelvik2.
3 3.1
Shared memory model Parallel implementation
There are eight matrices with different structure that are used to update the field variables in each time step. The more or less random structure of the matrices makes it very difficult to do an efficient distributed memory parallelization. On the other hand, using a shared memory model is straightforward. Thus, we have chosen to use OpenMP for our parallelization of the solver. The only critical issue is the load balance. The number of nonzero elements in the rows of the matrices vary between three up to about twenty. A static decomposition over the rows would then cause a load imbalance in the matrix-vector multiplications. This was solved by using the dynamic scheduling directives of OpenMP. Now, only the chunk size parameter must be tuned for the application in order to optimize the parallel performance. Minor modifications of the code were sufficient to port it to OpenMP. We had to rewrite some of the Fortran 90 array syntax to be able to parallelize the loops and to avoid temporary array allocations. Temporary arrays can degrade the parallel performance since the memory allocation is handled sequentially. This did speed up the serial performance as well.
215
3.2
Performance results
We have run the solver on three different parallel computers that support OpenMP; SUN Enterprise 6000 (UltraSparc 2), SGI Onyx2, and SGI Origin 2000. The characteristics of the computers are given in Table 1 and the compiler options that we have used are found in Table 2. The application we have run is the generic aircraft RUND, see Figure 1. The grid consists of 131000 tetrahedrons and 32500 primary nodes. A snapshot of the computed solution is shown in Figure 2.
Figure 1. Part of the unstructured grid around the aircraft RUND.
Figure 2. A snapshot of the computed solution showing surface currents.
The serial code runs fastest on SGI Origin but the SUN server gives the best parallel scaling. As we can see in Figure 3, the code runs at the same speed on all three machines using 8 threads. When using more threads SUN will give the best performance. (It was not possible to use more than eight threads on the two SGI machines at the time of the experiments.)
216 15 - « - SGI ONYX - B - SGI ORIGIN 2000 - * - SUN E6000 Ull
10
/
Q.
/
a
/
/ / / f / / //
LLI UJ Q. W
5"
/
/
/ . /
/
tit
//
V
A
10 15 20 NUMBER OF THREADS
25
30
(a) Fixed size speedup
SGI ONYX SGI ORIGIN 2000 SUN E6000 Ull
10 15 20 NUMBER OF THREADS
30
(b) Runtime per iteration
Figure 3. Speedup and runtime per timestep from the OpenMP parallelization. different parallel computers are compared. The application is the generic aircraft modeled with an unstructured grid consisting of 131000 tetrahedrons.
Three RUND
217 Table 1. Hardware configurations of the three computers.
Model SGI0nyx2 SGI 02K SUN E6K
CPU/MHz/# R10000/250/12 R10000/250/32 USH/250/32
Main Memory 4GB 32GB 8GB
L2 Cache 4MB 4MB 4MB
LI Cache 32KB 32KB 32KB
Table 2. Compilers and compiler options used.
Model SGI Onyx2 SGI 02K SUNE6K
Compiler MlPSpro 7 MlPSpro 7 FORTE 6.1
Options -64 -mp -mips4 -freeform - 0 3 -OPT:Olimit=0 -64 -mp -mips4 -freeform - 0 3 -OPT:Olimit=0 -fast -openmp
SGI Origin gives a noticeable drop in the parallel efficiency going from two to four threads. This is probably due to its memory hierarchy. The main memory is distributed over the nodes and at each node there are two CPUs. Going to four threads leads to remote node memory accesses which is slower than the on-node memory accesses. We get the same effect on the Sun system going from 14 to 16 threads. The Sun system consists of two servers with 16 processors each. The servers have their own memory but are connected with the Wildfire interconnect. For more than 29 threads the performance will drop as several threads are then scheduled to the same processors. 4 4.1
Distributed memory model Data distribution strategy
We have an unstructured grid consisting of tetrahedrons. Additionally, there is a dual grid with nodes in the centers of the tetrahedrons. There are six solution variables residing at different locations of the grids, as explained in Section 2. This is illustrated in Figure 4 below. The variables depend only on their nearest neighbors, either in the primary grid or in the dual grid. An obvious data distribution strategy would then be to use domain decomposition and partition the grids with a graph partitioning algorithm. This would minimize the data dependencies between the processors, i.e. minimize the communication. The partitioning of the nodes in the tetrahedral grid can be done with standard methods. A layer of ghost points overlapping the neighbor partitions can be added. The edges are then
218
Figure 4. A cell in the primary grid and a dual face.
assigned to the same processors as the nodes. In the ghost point layer the edges are replicated in the two partitions. The partitioning of the dual grid can follow the partitioning of the primary grid in the same way as the edges but then some of the nodes may be replicated in several partitions, at most in four (the four surrounding nodes of a tetrahedron may all be in different partitions). 4-2
Implementation
issues
With a local numbering of the nodes within each partition and a layer of ghost points overlapping neighbor partitions it will be possible to re-use the serial solver for each partition. Moreover, to distinguish between ordinary nodes and ghost points, the ghost points can be gathered at the end of the arrays. The ghost points are updated by the neighbor processors and are then communicated. Thus, the solver needs to be extended with functionality for partitioning the grids and the corresponding field variables, translation from global to local numbering, reassembling or reorganization of the matrices for the local numbering of the field variables, distribution from global to local arrays, communication of ghost points, and gathering from local to global arrays. Also, data redistribution routines may be needed in a parallel hybrid model. The partitioning of the grids can be done with the graph partitioning package Metis, Karypis 4 . To minimize the communication overheads in updating the ghost points we utilize a communication table. The communication table is computed once
219
and for all initially and is then used for packing and unpacking data in and out of the communication buffers. It keeps track of which elements should be communicated and with which processor. The actual communication is implemented with the asynchronous operations MPLIsend and MPLIrecv. The requests are handled with first come service, i.e. with MPLWaitany. Letting the system decide in which order to process the messages minimizes the total wait time for the communication requests.
4-3
Performance results
All the above described functionality has been implemented for parallelization of one specific operation in the solver, the electric field filtering operator at the tetrahedral nodes. The code was organized such that all computations were duplicated on all processors, except for the filtering of electric node values which was parallelized. In practice, we ran the serial code with all data on all processors simultaneously until the filtering operation. At this point, the data was distributed to private local arrays and the filtering operation was done in parallel. Then, the data was gathered globally again continuing in the serial fashion with the other operations. Now, only a small fraction of the solver was parallelized. The other operations are similar matrix-vector updates as the filtering operation. Hence, by timing this part of the code we will get an idea of how the full parallel code will perform. Again, we have run the solver with the generic aircraft RUND. The results are summarized in Table 3. We could not run on more than four processors as all data was replicated on all processors and the parallelization required additional memory. We simply ran out of memory. In the full parallel code the data will not need to be replicated on all processors, it will be distributed. Hence, we will then be able to run larger and larger problems as we increase the number of processors.
Table 3. Performance results from the distributed memory model using the RUND geometry. The timings are from IBM SP2 for one filtering operation of the electric node values.
Proc Time
1 0.383
2 0.199
4 0.114
The speedup is modest, 3.35 on four processors. This is somewhat better than the OpenMP version on the two SGI machines.
220
5
Further work
The performance of the OpenMP version can be improved by reordering the nodes and the edges with, for example, the Reverse Cuthill McKee algorithm. This will give better data locality and reduce the off-node data accesses. Also, the serial performance can be significantly improved, Anderson 1 , due to less cache misses. Furthermore, the MPI version needs to be completed to be able to run the code on distributed memory machines that do not support OpenMP. Also, for very large problems that require many processors we may need MPI to keep up the scalability. It seems that OpenMP may have some limitations in scalability for large processor configurations due to memory hierarchies. In the full hybrid solver, a parallelization strategy could be to block the grids geometrically letting each unstructured part be a complete block. The unstructured parts should anyway be kept small due to efficiency reasons. We can then use a dual paradigm programming model combining MPI and OpenMP. We distribute the blocks to shared memory nodes. Within the nodes we use OpenMP and between the nodes MPI for the communication. Then, it would not be necessary to parallelize the unstructured parts with MPI and we would still get good scaling. 6
Conclusions
We have parallelized an unstructured finite volume solver for the Maxwell equations using OpenMP. We have run the code on three different parallel computers, SUN Enterprise 6000, SGI Onyx2, SGI Origin 2000, that support the shared memory programming model. The results from SUN show very good scaling of the code up to 14 threads. The problem size is small, only 131000 tetrahedrons. Large applications can contain several millions of tetrahedrons, Mavriplis3. For full size applications the code has a good potential to scale very well. From the run-times we can clearly see the impact of the memory hierarchy to OpenMP. Going from local memory accesses to remote node memory accesses gives a significant drop in efficiency and a decrease of the slope in speedup. This indicates a limitation of the scalability of the OpenMP version for large processor configurations with distributed memory. To keep up the efficiency MPI will be needed. We have outlined a strategy to parallelize the code with a distributed memory programming model, i.e. with MPI. Preliminary results show a performance similar to the OpenMP code for a small number of processors. In
221
conclusion, it was far simpler to parallelize the code using OpenMP than using MPI. Finally, we have also discussed the idea of a dual paradigm model combining OpenMP and MPI for a hybrid solver. Acknowledgments The work presented in this paper was performed within the framework of the Parallel and Scientific Computing Institute (PSCI) in cooperation with the Swedish industrial partners Ericsson and SAAB. Computer time was provided by the National Supercomputer centers, PDC at Royal Institute of Technology in Stockholm and by NSC at Linkoping University. References 1. W.K. Andersson, W.D. Gropp, D.K. Kaushik, D.E. Keyes, B.F. Smith, Achieving High Sustained Performance in an Unstructured Mesh CFD Application, ICASE Report No. 2000-2, NASA Langley Research Center, Hampton, Virgina, USA. 2. F. Edelvik, Finite Volume Solvers for the Maxwell Equations in Time Domain, Licentiate thesis 2000-005, Department of Information Technology, Uppsala University, Box 120, S-751 04 Uppsala, Sweden. 3. D. Mavriplis, Parallel Performance Investigations of an Unstructured Mesh Navier-Stokes Solver, ICASE Report No. 2000-13, NASA Langley Research Center, Hampton, Virgina, USA. 4. G. Karypis, V. Kumar, Metis: Unstructured Graph Partitioning and Sparse Matrix Ordering System, Technical Report, University of Minnesota, Department of Computer Science, Minneapolis, 1995. 5. K. S. Yee, Numerical solution of initial boundary value problems involving Maxwell's equations in isotropic media, In IEEE Trans. Antennas Propag., 14(3):302-307, March 1966.
A N H Y B R I D O P E N M P - M P I PARALLELIZATION OF THE P R I N C E T O N O C E A N MODEL G. S A N N I N O , V. A R T A L E ENEA,
C. R. Casaccia,Via
Anguillarese
301, 00060 Rome,
Italy
P. L A N U C A R A CASPUR,
P.le A. Moro 5, 00185 Rome, E-Mail:
[email protected] Italy
This paper deals with the parallelization of one of the most popular threedimensional oceanographic model: the Princeton Oceean Model (POM). The parallelization is achieved using standard tools like MPI and OpenMP, to ensure portability of the code. A robust and efficient domain decomposition method is used to solve, in principle, large scale examples on clusters of shared memory machines. A comparison between the pure MPI and the hybrid M P I + O p e n M P is shown in terms of elapsed time for a standard seamount problem varying the number of grid points and cpus.
1
Introduction
Ocean models represent one of the fundamental tools to investigate the physics of the ocean. In the last twenty years they have been successfully applied to a wide range of oceanic and climate problems. The numerical methods used by ocean models consist in discretizing the Navier-Stokes equations on a three dimensional grid and computing the time evolution of each variable for each grid point. Nowadays, to answer the huge amount of computational demand raised to conduct simulation on high resolution computational grid, a parallelization strategy is needed. In this paper we present a hybrid OpenMP-MPI parallel version of the Princeton Ocean Model (POM), one of the most used ocean model, and evaluate the parallel performance obtained. In particular in section 2 we shortly describe the ocean model features and algorithm, in section 3 we illustrate the technique used to parallelize the serial code and in section 4 we report the performance obtained. Conclusions are summarized in section 5. 2
Princeton Ocean Model (POM) description
The Princeton Ocean Model is a three-dimensional primitive equation model, i.e., solves in finite difference form the Navier-Stokes equations along with a
222
223
nonlinear equation of state which couples the two active tracers (temperature and salinity) to the fluid velocity 7 . It has been extensively applied to a wide range of oceanic problems including estuarine and shelf circulation studies 1, data assimilation in the Gulf Stream 8 and general circulation studies in the Mediterranean Sea 10 . The model algorithm uses an explicit differencing scheme for the two horizontal velocity components, the temperature and the salinity; the other variables, the pressure, the density and the vertical component of velocity are calculated by implicit differencing. The model also solves a set of vertically integrated equations of continuity and momentum, usually called external mode to provide free surface variations. Because of the very fast speed of surface waves in the ocean, the time step used to solve this mode is very short. For computer time economy the 3D-equations (internal mode), are solved using a larger time step, at the end of which the two modes are combined; this procedure is known as time splitting technique. The model specifies the values of all variables at the nodes of a curvilinear orthogonal grid, staggered as an Arakawa-C scheme, conserving linear and quadratic quantities like mass and energy. The model also uses a sigma-coordinate system for which details on the transformed equations and numerical algorithm can be found in 2 ' 6 . 3
Parallelization
POM is a FORTRAN 77 code that was initially designed for serial processors and later converted to vector processors. In the last decade, vector supercomputers are begun obsolete and new hardware based on commodity chip have take place. Nevertheless, the serial code is still widely used and can be downloaded from web 13 . In the last years some attempt towards a parallel implementation of the Princeton Ocean Model was carried out (see for example 9>3>5). The key points for the above parallelization are the use of message passing tools (PVM or MPI) coupled with domain decomposition technique or data parallel language (HPF) and Fortran 90 style. We found some disadvantages in both approaches: PVM is no longer used by the High Performance community while HPF suffers a lack of performance on a large number of hardware platforms; moreover, a vendor implementation of HPF is not installed on all machines (for example on IBM systems). Last but not least, not all the above parallel implementations are available for the POM user and, where available, configuring and optimizing a black box parallel code for a particular architecture or
224
physical test is a very complex task. Thus, we decided to develop our parallel version of the code assuming that the optimal choice has to be based on the message passing library MPI and domain decomposition technique. Here follows a brief description of the MPI code (see 4 for a complete description). The MPI code is structured assuming a two (or one) dimensional geometric decomposition of the ocean computational grid into a set of smaller sub-domains. In particular, the grid is horizontally partitioned across the latitude and/or longitude dimensions into rectangular blocks, leaving the vertical dimension unchanged (see Fig 1)).
Figure 1. 2D domain decomposition
Each sub-domain is then allocated to a single processor, which ran a complete copy of the code, i.e. it is responsible for stepping forward the dynamic ocean variables on the sub-domain under consideration. This technique has been chosen because is both easy to develop from uni-processor code and easy to maintain. In order to guarantee the same results of the serial code at the last bit precision of the machine, and to reduce the frequency of communication, all the sub-domains are overlapped at the inner boundaries on a slice of 2 grid point's thickness. These slices represent grid points (halo points) that contain copies of the boundary values stored on neighboring sub-domains (Kernel points) in the grid topology. At some stages, during the calculation, the v£&ues stored on
225
these slices must be updated receiving values from neighboring kernel points. Two types of inter-machines communications are needed to keep the computation consistent with the sequential code: (1) point-to-point communication to update halo values among neighboring machines and (2) global communication at each external time step to check the stability (CFL) condition, and to gather the output variables. One of the main result of this kind of parallelization is that the memory storage is reduced. This is because each processor only needs to store a portion of the global memory, and is therefore possible to solve, on a cluster, problems larger of the memory size of a single machine. The result is more evident treating large size oceanographic experiments. The balance of the computational load, and the minimization of the interprocessor communications are simultaneously and automatically achieved in our MPI code. A routine, placed at the beginning of POM code, computes an optimal decomposition both in I D slices or 2D rectangular sub-grids, depending on the number of grid points of the global horizontal model domain and on the number of requested machines respectively. It is interesting to note that Fortran 90 features are widely used in the MPI code (dynamic memory management and array syntax instructions) giving us a more readable and efficient program. The results are very promising; nevertheless, the memory size and the demand of CPU for real problems (for example, for a Model of the Mediterranean Sea, with a horizontal resolution of 5 Km, are necessary about 844 x 332 grid points) are so big that naturally lead us to develop a code for a specific target architectures: the cluster of SMP (Symmetric Multiprocessor) machines. The interest for this kind of architecture is recent, but it seems to be the trend for building a supercomputer at a relatively low cost. Compaq clusters based on Linux and OSF operative systems are installed at CASPUR and ENEA respectively. We remark that, on such a cluster, MPI should be used for internode communication, while shared memory should be the best choice for intranode communication. The code presented in this work has been optimized for fully exploiting the parallelism on such architecture. Within each SMP machine OpenMP is used to divide the entire work among different processors using threads. OpenMP is a fully accepted standard for SMP parallelization and is efficiently implemented from all vendors ( n ) . It realizes the so-called multi-threaded paradigm in which each thread access shared and private data items. Parallel regions are used to parallelize do-loops in the code. An example: C ADD VISCOUS FLUXES DO 860 J=2,JMM1 DO 860 1=2,IM
226 860
FLUXVA(I,J)=FLUXVA(I,J) 1 -D(I,J)*2.E0*AAM2D(I,J)*(VAB(I,J+1)-VAB(I,J))/DY(I,J) it becomes: C ADD VISCOUS FLUXES !$OMP PARALLEL DO !$OMP&PRIVATE (I,J) DO 860 J=2,JMM1 DO 860 1=2,IM 860 FLUXVA(I,J)=FLUXVA(I,J) 1 -D(I,J)*2.E0*AAM2D(I,J)*(VAB(I,J+1)-VAB(I,J))/DY(I,J) !$OMP END PARALLEL DO
Distribution of the entire work is automatically done by compiler, but any privatization of data or synchronization points in the code, needed to avoid race conditions, are done by programmer. We have used a very optimized implementation of OpenMP available on a great variety of platforms, Guide from KAI ( 12 ) for portability reasons. Also, a parallel debugger (ASSURE) and a Performance Viewer (Guideview) are available and we used strongly in the parallelization steps. PH-POM uses the same input and output files as the original POM. File input/output is performed only by the master processor. 4
Performance results: the seamount case
To demonstrate the potential performance of PH-POM we have compared the pure MPI version and the hybrid MPI+OpenMP one with the serial code. In particular, an idealized seamount case was defined in two different configurations: in the first configuration 42 sigma levels and a horizontal grid made by 1000 x 300 grid points {Big case), while 32 sigma levels and the same horizontal resolution for the second configuration (Medium case). The bottom topography is completely flat at —4500 m except in the center of the domain where a steep seamount is defined as H (x, y) = H0 [l.O - 0.9e(x2+y^/L2j
(1)
where H0 is 4500 m and L is 25 -103 m. In both cases, the horizontal computational model grid is staggered onto a rectangular grid. In particular it is stretched so that the resolution is highest at the center where is defined the seamount. The resulting bathimetry is illustrated in Fig 2. The external and internal time steps are 6 sec and 180 sec respectively. Performance analysis were done on a cluster of 4 IBM SP3 interconnected via High Performance
227
Figure 2. Seamount geometry. The grid is stretched so that the resolution is highest at the center.
Switch, a fast communication link. Each SP3 node is a 16 processors machine equiped with Power3 processor at 375 Mhz clock rate. We have tested two different kind of communication among different machines: SLOW communication, setting the Internet Protocol (ip) mode both for SP3 switch and MPI calls within the same node (MPLSHARED-MEMORY no); FAST communication, using the switch in user space (us mode) and MPI calls via shared memory (MPLSHARED-MEMORY yes). Table 1 shows that for a cluster in which the interconnection between different machines is relatively slow, the hybrid configuration is faster than the pure MPI one, both for the Medium and the Big test case. On the contrary, the MPI configuration has in general better performance with respect to the hybrid one in the FAST communication case; nevertheless, for a choosen set of MPI process and OpenMP threads (32 x
228
2) both the Big and the Medium case shows a decreasing of the elapsed time with respect to the pure MPI.
MPI Medium HYBRID Medium HYBRID Medium HYBRID Medium HYBRID Medium MPI Big HYBRID Big HYBRID Big HYBRID Big HYBRID Big
N u m . Proc. 64 64 64 64 64 64 64 64 64 64
Task Decomp.
4x16 8x8 16x4 32x2
SLOW 2327 1739 1385 1521 2080
FAST 1022 1375 1115 1012 978
2807 2245 1845 1962 2433
1374 1818 1490 1336 1291
4x16 8x8 16x4 32x2
Table 1. Elapsed time in seconds. 1 DAYS of simulation The final comparison with the serial POM shows very good speed-up results with respect to the hybrid code setted in the best configuration; more precisely, a speed-up of almost 47 for the Big configuration and 51 for the Medium one. 5
Conclusion
The parallelization of the Priceton Ocean Model (POM) has been succesfully realized. The code is in principle able to solve very large problems with great efficiency; portability has been succesfully realized using MPI and OpenMP standard. Moreover, using Fortran 90 features, a clear and modular code has been developed and the program is suitable to run efficiently on generic clusters (using MPI) but also on clusters of shared memory machines using the hybrid approach. The preliminary results are encouraging. The idea is to provide more and more complicated and challenging physical tests using bigger and bigger clusters. Acknowledgments We wish to thanks Giorgio Amati for useful suggestion and discussion.
229 References 1. Blumberg A. F. and G. L. Mellor, 1983: Diagnostic and prognostic numerical circulation studies of the South Atlantic Bight. J. Geophys. Res., 88, 4579-4592. 2. Blumberg A. F. and G. L. Mellor, 1987: A description of a threedimensional coastal ocean circulation model. Three-Dimensional Coastal Ocean Models, Coastal Estuarine Science, N. S. Heaps, Ed., Amer. Geophys. Union, 1-16. 3. Boukas, L. A., N. T. Mimikou, N. M. Missirlis, G. L. Mellor, A. Lascaratos, and G. Korres, The parallelization of the Princeton Ocean Model, in: Lecture Notes in Computer Sci., Amestoy et al. (Eds.), Springer, 1685, 1395-1402, 1999. 4. Sannino G., Artale V., Lanucara P., Parallelization of the Princeton Ocean Model: a Domain Decomposition approach, submitted to Parallel Computing 5. POM Benchmark Results page: http://www.aos.princeton.edu/WWWPUBLIC/htdocs.pom/POMcpu .txt 6. Mellor G. L., 1991: User's guide for a three-dimensional, primitive equation, numerical model. AOS Program Rep., Princeton University, Princeton, NJ, 34 pp. 7. Mellor G. L., 1991: An equation of state for numerical models of oceans and estuaries. J. Atmos. Oceanic Technol., 8, 609-611. 8. Mellor G. L. and T. Ezer, 1991: A Gulf Stream model and an Altimetry Assimilation Scheme.J. Geophys. Res., 96, 8779-8795. 9. Oberpriller, W. D., A. Sawdey, M. T. O'Keefe and S. Gao, 1999: Parallelizing the Princeton Ocean Model using TOPAZ, Parallel Computer Sys. Lab.,Dept. Elec. Comp. Eng., University of Minnesota, Tech. Report, 21pp. 10. Zavatarelli M. and G. L. Mellor, 1994: A numerical Study of the Mediterranean Sea Circulation. J. Phys. Oceanogr.,25, 1384-1414. 11. OpenMP home page http://www.openmp.org/ 12. KAI Software home page http://www.kai.com/ 13. POM home page http://www.aos.princeton.edu/WWWPUBLIC/htdocs.pom/
A SIMD SOLUTION TO BIOSEQUENCE DATABASE SCANNING
BERTH. SCHMIDT, HEIKO SCHRODER AND THAMBIPILLAI SRKANTHAN School of Computer Engineering, Nanyang Technological University, Singapore E-mail: asbschmidt(a),ntu.edu.ss, asheiko&ntu.edu.se, astsrikan&ntu.edu.ss Molecular biologists frequently compare an unknown protein sequence with a set of other known sequences (a database scan) to detect functional similarities. Even though efficient dynamic programming algorithms exist for the problem, the required scanning time is still very high, and because of the exponential database growth finding fast solutions is of highest importance to research in this area. In this paper we present an approach to high-speed biosequence database scanning on the Fuzion ISO, a new parallel computer with a linear SIMD array of 1536 processing elements on a single chip. This results in an implementation with significant runtime savings.
1
Introduction
Scanning protein sequence databases is a common and often repeated task in molecular biology. The need for speeding up this treatment comes from the exponential growth of the biosequence banks: every year their size scaled by a factor 1.5 to 2. The scan operation consists in finding similarities between a particular query sequence and all the sequences of a bank. This operation allows biologists to point out sequences sharing common subsequences. From a biological point of view, it leads to identify similar functionality. Comparison algorithms whose complexities are quadratic with respect to the length of the sequences detect similarities between the query sequence and a subject sequence. One frequently used approach to speed up this time consuming operation is to introduce heuristics in the search algorithms [1]. The main drawback of this solution is that the more time efficient the heuristics, the worse is the quality of the results [9]. Another approach to get high quality results in a short time is to use parallel processing. There are two basic methods of mapping the scanning of protein sequence databases to a parallel processor: one is based on the systolization of the sequence comparison algorithm, the other is based on the distribution of the computation of pairwise comparisons. Systolic arrays have been proven as a good candidate structure for the first approach [5,11], while more expensive supercomputers and networks of workstations are suitable architectures for the second [4,7]. Special-purpose systolic arrays provide me best price/performance ratio by means of running a particular algorithm [6]. Their disadvantage is the lack of flexibility with respect to the implementation of different algorithms. Programmable SIMD architectures strive for algorithmic flexibility and the speed of special-
230
231
purpose systems. In this paper we present a biosequence database scanning implementation on the Fuzion 150, a single chip SIMD array containing 1536 processing elements [10]. We will show that this approach leads to significant runtime savings. This paper is organized as follows. In Section 2, we introduce the basic sequence comparison algorithm for database scanning and highlight previous work in parallel sequence comparison. Section 3 provides a description of the Fuzion 150 architecture. The mapping of database scanning onto the parallel architecture is explained in Section 4. The performance is evaluated and in Section 5. Section 6 concludes the paper with an outlook to further research topics. 2
Parallel Sequence Comparison
Surprising relationships have been discovered between protein sequences that have little overall similarity but in which similar subsequences can be found. In that sense, the identification of similar subsequences is probably the most useful and practical method for comparing two sequences. The Smith-Waterman (SW) algorithm [12] finds the most similar subsequences of two sequences (the local alignment) by dynamic programming. The algorithm compares two sequences by computing a distance that represents the minimal cost of transforming one segment into another. Two elementary operations are used: substitution and insertion/deletion (also called a gap operation). Through series of such elementary operations, any segments can be transformed into any other segment. The smallest number of operations required to change one segment into another can be taken into as the measure of the distance between the segments. Consider two strings 51 and 52 of length /l and 12. To identify common subsequences, the SW algorithm computes the similarity H(ij) of two sequences ending at position i andy of the two sequences 51 and 52. The computation of H(ij) is given by the following recurrences: Wij) = max{0, E(ij), F(iJ), H(i-lj-l)+Sbt(SlhS2j)}, \)=e(p-n
+ N)
(1)
where
(18)
I Pr Figure 4. An adaptive grid and its coefficient matrix
as is sketched in Fig. 4. This results in a coefficient matrix which combines sparsity with a diagonal structure, if we take into account the interaction with shadow
259 points, which are marked by dots in Fig. 4, by modifying the right hand side. It is this structure that we exploit in the following. 4
Adaptivity and Parallelism
In order to speed up the simulation process even further we apply parallel processing. To this end we subdivide the computational grid into slices each of which is taken care of by one processor. The boundary layers of each slice are padded with a plane of shadow points which serve as a buffer for holding the boundary values of the neighbouring slice. As the computation is refining the computational grid adoptively, the number of grid points per processor becomes imbalanced. Fig. 5 exemplifies the situation in the case of three processors: In the beginning each processor is responsible for a 4 x 5-grid; when the critical points marked by a cross have been refined, processor 1 holds 49, processor 2 holds 12, and processor 3 holds 24 grid points. In order to keep the number of grid points per processor nearly constant, which means approximately 20, we introduce a fourth processor and redistribute the slices as sketched in Fig. 6. Although the redistribution of the computational grid requires communication among processors, we must refrain from administrating the grid in a
Figure 5. The imbalance caused by adaptive grid refinement
260
(1)
v =
22
P) 21
(3) V =
22
4) - 2i n j< ' = 0
Figure 6. The redistribution of grid points during adaptive refinement
centralised fashion, because this will inevitably prevent the parallel simulation from being scalable. The problem becomes even more complicated, if we take into account that the number of active processors may change during the computation. By a global sum operation the processors communicate the total number of active grid points. To exploit the diagonal structure of the coefficient matrix in Sect. 3, only complete planes are assigned to each processor by a rounding procedure. This initial distribution is improved by a few iterations of a diffusion method. Fig. 7 summarizes the result of our decentralised redistribution of grid points. Whereas the imbalance of grid points on 8 processors leads to a 40 percent deviation from the mean computation time, the redistribution achieves a 2 percent deviation! Summing up, our simulation, which was written from scratch, achieves an efficiency of 79 percent using 61 processors of a Cray T3D; the execution times decreases from 2.5 hours on a single processor to 3 minutes. This means that simulating complex systems interactively is feasible!
261
2 3 4 5 6 7 S Processor id
1 2 3 4 5 6 7 8 Processor id
Figure 7. Adaptivity and load balancing
5
Conclusions
From this application we may draw the following conclusions: • Parallel computing opens the door to an interactive simulation of complex systems in science and engineering. • Adaptive computing paves the way to handling complex systems and has to go hand in hand with parallel processing. • Adaptive parallel computing requires that the complete solution process be local and decentralised in order to achieve scalability. References 1. C. Cercignani: The Boltzmann Equation and its Applications. New York, NY: Springer (1988). 2. J. D. Jackson : Classical Electrodynamics. New York, NY: Wiley, 3rd edition (1999). 3. Q. Lin, N. Goldsman, and G. Tai: A globally convergent method for solving the energy balance equation in device simulation. Solid-State Electronics 36 (1993)411-419. 4. J. Slotboom: Computer-aided two-dimensional analysis of bipolar transistors. IEEE Trans. Electron Devices 20 (1973) 669-679.
This page is intentionally left blank
ALGORITHMS
This page is intentionally left blank
ANALYSIS OF COMMUNICATION OVERHEAD IN PARALLEL CLUSTERING OF LARGE DATA SETS WITH P-AUTOCLASS STEFANO BASTA ISI-CNR,
Via P. Bucci, cubo 41-C, Rende (CS), Italy E-mail: basta@si. deis. unical. it DOMENICO TALIA
DEIS,
Universita della Calabria, Via P. Bucci, cubo 41-C, Rende (CS), Italy E-mail: talia@deis. unical. it
This paper focuses on the advantages derived by the use of P-AutoClass, an MIMD parallel version of the AutoClass system, for parallel data clustering. AutoClass is an unsupervised data mining system that finds optimal classes in large data sets. The P-AutoCIass implementation divides the clustering task among the processors of a multicomputer so that each processor works on its own partition and exchanges intermediate results with the other processors. In particular, we point out that a main reason of P-AutoCIass scalability is that communication costs do not depend on size of data to be analyzed. In fact, our analysis puts in evidence that the overhead due to data exchange among processors depends on the number of attributes and classes but not on data set size. We present theoretical considerations that support this P-AutoClass feature and illustrate some experiments showing this behavior.
1
Introduction
Clustering is an unsupervised learning technique that separates data items into a number of groups or clusters such that items in the same cluster are more similar to each other and items in different clusters tend to be dissimilar, according to some measure of similarity or proximity. Differently from supervised learning, named classification, where training examples are associated with a class label that expresses the membership of every example to a class, clustering assumes no information about the distribution of the objects. Therefore the clustering task must both discover the classes present in a data set and to assign items to such classes in the best way. Most of the early clustering analysis algorithms come from the area of statistics and have been originally designed for relatively small data sets. In the recent years, clustering techniques have been efficiently extended to deal with large databases so that they play a role in the field of Knowledge Discovery [7]. Clustering algorithms are very computing demanding and thus require the use of high-performance machines to get results in a reasonable amount of time. Executions of clustering algorithms taking one week or about 20 days of computation time on sequential machines are not rare [8]. Scalable parallel computers can provide the appropriate setting to efficiently execute clustering algorithms for extracting knowledge from
265
266
large-scale databases. Recently, there has been an increasing interest in parallel implementations of data clustering algorithms. Generally, very large data sets are analyzed by data mining algorithms, thus performance scalability is possible when computation overhead does not strictly depend from data size. Parallel approaches to clustering can be found in [6, 11, 12]. In this paper we consider P-AutoClass [5], a distributed-memory parallel implementation of a clustering algorithm based on Bayesian classification. We discuss this parallel clustering algorithm and, in particular, evaluate the effect of communication overhead on the global computation time. We show some effective advantages that justify the effort of parallelization of AutoClass. The rest of the paper is organized as follows. Section 2 introduces clustering mediods and provides an overview of Bayesian classification and sequential AutoClass. Section 3 outlines the main features of P-AutoClass. Section 4 discusses a very important property of P-AutoClass: me communication overhead does not depends on the number of tuples of a data set but on the number of attributes and classes. This makes the parallel algorithm scalable when the size of analyzed data sets increases. On the basis of this behavior, it seems very appealing the utilization of P-AutoClass for speeding up clustering of large data sets. Section 5 contains some experiments validating these theoretical results. Finally, Section 6 draws some conclusions. 2
Clustering and AutoClass
Clustering techniques play a major role in data mining of large data sets and have received an increasing interest in various fields such as sociology, biology, statistics, artificial intelligence, and information retrieval. Clustering algorithms can roughly be classified into the three following types [4]: hierarchical, partitional and density or mode-seeking. Hierarchical methods generate a nested sequence of clusters by a hierarchical decomposition of a set of N objects represented by a dendogram. Partitional methods produce a partition of a set of objects into K clusters by optimizing a given criterion function. One of the most known criterion is the squared error criterion. The K-Means clustering [1] is a well-known and effective method for many practical applications that employs the squared error criterion. Density search techniques [10] consider objects as points in a metric space and suggest that clusters should be those parts of the space characterized by a high density of data. High density regions are called modes. A statistical density search-based approach considers dense regions of probability density of data objects as different groups. This approach uses statistical concepts with the aim to represent the probability density function through a mixture model. The Bayesian approach to unsupervised classification, used in AutoClass [3], provides a probabilistic method to inductive inference. The Bayesian classification,
267
rather than just partitioning objects into distinct classes, searches for the best class descriptions that predict the data objects in a model space. In this approach class membership is expressed probabilistically, that is an instance is not assigned to a unique class, instead it has a probability of belonging to each of the possible classes. The classes provide probabilities for all attribute values of each instance. Class membership probabilities are then determined by combining all these probabilities. Class membership probabilities of each instance must sum to 1, thus there are not precise boundaries for classes: every instance must be a member of some class, even though we do not know which one. Bayesian theory is based on the assumption that the objects under consideration are controlled by probability distributions and predictions on new objects can be made by reasoning about these probabilities and the observed objects. Let D={X,,...Xm} denotes the observed data objects, where instances or items X,. are represented as ordered vectors of attribute values X = {Xtl, ...Xik}. Unsupervised classification aims at determining the best class description (hypothesis) h from some space H that predicts data D. The term best can be interpreted as the most probable hypothesis given the observed data D and some prior knowledge on the hypotheses of H in the absence of D, that is the prior probabilities of the various hypotheses in H when no data have been observed. Bayes theorem provides a way to compute the probabilities of the best hypothesis, given the prior probabilities, the probabilities of observing the data given the various hypotheses and the observed data. 3
P-AutoClass
Parallel execution of data mining algorithms require dividing up the computation, so that each processing element can perform a part of the data mining task in parallel with the other processing elements of a parallel computer. The results are thus achieved as fast as possible. In doing this parallelization it is important to split the computational task in a balanced way among the processing elements, but it is equally important to minimize the data communication among them during the data mining process and separate accesses to the data set. The execution of AutoClass on large data sets requires times that in many cases are very high. For example, in the clustering of a satellite image, AutoClass took more than 130 hours [9] and for the analysis of protein sequences, the discovery process required from 300 to 400 hours [8]. These considerations and experiences suggested the necessity to implement a parallel version of AutoClass, called P-AutoClass, to mine very large data sets in a reasonable time. This has been done by exploiting the inherent parallelism presents in the AutoClass algorithm implementing it in parallel, using the MPI tool kit, on MIMD multicomputers [5]. Among the strategies for parallelization, has been selected the Single Program Multiple Data (SPMD) approach because it does not
268
require to replicate the entire data set on each processor, as other strategies do. That strategy also does not raise load balancing problems because each processor executes the same code on a data set of equal size. Finally, most of the operations are performed locally on each processor. Other parallel prototypes of AutoClass have been designed, but those have mainly been realized on SIMD parallel machines by using parallelizing compilers that automatically generated data-parallel code starting from the sequential code [3, 15] or have been implemented running in parallel several sequential AutoClass programs and combining results [16]. This last case does not guarantee sequential AutoClass semantics and results, whereas SIMD implementations cannot exploit all the inherent parallelism of the algorithm. 3.1
Design of the parallel algorithm
The main part of the algorithm is devoted to classification, generation and evaluation of generated classification. This loop is composed of a set of substeps, among which the new try of classification step is the most computationally intensive. In AutoClass class membership is expressed by weights. This step computes the weights of each item for each class and then it computes the parameters of classification. These operations are executed by the base_cycle function which calls the three functions updatewts, update parameters and update approximations. The time spent in the basecycle function is about the 99,5% of the total time of the algorithm. Therefore, this function has been identified as mat one where parallelism must be exploited to speed up the AutoClass performance. In particular, if we analyze the time spent in each of the three functions called by base_cycle, it appears [5] that the updatewts and updatejparameters functions are the most time consuming functions whereas the time spent in the update approximation is negligible. Therefore, P-AutoClass is based on the parallelization of these two functions using the SPMD approach. To maintain the same semantics of the sequential algorithm of AutoClass, the parallel version is based on partitioning data and local computation on each of P processors of a distributed memory MIMD computer and on exchanging among the processors all the local variables that contribute to form the global values of a classification. The function update wts calculates the weights wy for each item / of the active classes to be the normalized class probabilities with respect to the current parameterizations. The parallel version of this function first calculates on each processing element the weights wtj for each item belonging to the local partition of the data set then it sums the weights w,- of each class j (wj=2^ Wjj) relatively to its own data. Then all the partial w, values are exchanged (using the MPI_Allreduce operation) among all the processors and summed in each of them to get the same value in every processor. The updateparameters function computes for each class a set of class posterior parameter values, which specify how the class is distributed along the various
269 attributes. In the parallel version of this function, the partial computation of parameters is executed in parallel on all the processors, then all the local values are collected on each processor (using also in this case the MPI_Allreduce operation) before to utilize them for computing the global values of the classification parameters. 4
Reasoning about Communication Time
Before analyzing communication time, we discuss how the computation time of the sequential and parallel versions of AutoClass can be modeled by simple functions. The computation time Ts of sequential AutoClass depends on the number of classes, the number of instances, the number of attributes, the number of tries and the number of cycles that must be executed in each try until convergence is reached: Ts(l, Xb Cj) = 0(NTCT X, A Cj). Notation Cj
X, A P
Definition number of classes number of data instances number of attributes number of processors
Notation NT R-TI.T
CT
Definition the number of tries time to reduce n data of type T number of cycles per try
Table 1. Notation used in this section. In the parallel version of AutoClass the compute time TP depends on the size of a single partition of the instances, not on the total data set. Furthermore, two terms that take into account the time to exchange (reduce) data classification (of double type) among the processors must be considered. In particular, the term R)d (Cj +1) is related to the reduce operations in the update wts function, and the term R3d CjA is related to the reduce operations in the updateparameters function. Then the compute time in P-AutoClass is expressed by the following formula: TP (P, Xb Cj) =0(NTCT
f(X,/P) ACj+ R,,d (Cj +1) + R3.d CjAJ)
= 0(Ts/P + NT CT [R,,d (Cj +1)+ R3.d CjAJ). From these formulas, we can observe that the communication time is proportional to the number of classes and attributes and not to the number of data items. This is because, differently from other parallel data mining algorithms, in P-AutoClass processors exchange values related to attributes, classes and classification parameters. Therefore, the amount of data exchanged among the processors does not depend on size of mined data sets but on the number of attributes composing tuples and clusters in which data are grouped. This property assures that the communication overhead of P-AutoClass does not increase when the number of instances in the data set increases, but it is higher when the items consists of many attributes. However, the number of attributes is very low
270
in comparison with the number of attributes. In conclusion, we can deduce that PAutoClass is a scalable algorithm for clustering massive data repositories. In the following section, we show some experiments we performed to verify theoretical hypothesis on communication. 5
Experimentation
We run our experiments on a Meiko CS 2 with up to 10 SPARC processors connected by a fat tree topology with a communication bandwidth of 50 Mbytes/s in both directions. We describe here results on two data sets: the synthetic data set DS1 described in [13] which is composed of 100000 items with 2 real attributes and the IRAS Low resolution Spectral Atlas [2]. This data set consists of 5425 mean spectra of IRAS point sources, where each spectrum is a multi-dimensional data that consists of 100 channels ( so we have 100 attributes per tuple). To perform our experiments on the first data set we used different partitions of data from 5000 tuples to the entire data set, and asked the system to find the best clustering starting from different numbers of clusters. Each classification has been repeated 10 times and times presented here represent the mean values obtained after these classifications. We measured the elapsed time and the absolute speedup (Sp=Ti/Tp) of P-AutoClass in comparison with sequential AutoClass. For the sake of brevity, we only show the speedup (figure 1 and figure 2) in clustering the DS1 and IRAS data sets.
** 1
4
4
4 ! 4 4
a -
s
J^
9
& ,
t ^
r
*
4 \
«
i
2i
S\
4
1
2
3
4
5
6
7
B
B
number of processors
Figure 1. Speedup for DS1.
ID
number of processors
Figure 2. Speedup for IRAS.
From figures 1 and 2, we can notice that for the DS1 data set the speedup confirms the advantages obtained using several processors; differently, in the case of IRAS data set, the speedup is limited (for 7 processors speedup is 3.2 and for 10 processors it is 3.5). Furthermore, to verify the effect of the size of data set on the
271
application speedup, we doubled the 5425 spectra of IRAS data set. On this artificial data set composed of 10850 tuples we got a better speedup (not shown in detail here): using 10 processors the speedup is 4.7. This suggests that as the size of the data set increases the application can get a larger benefit from the use of a large number of processors. However, on these speedup results there is a significant influence of communication overhead that in the first case (DSl) is low and in the second one (IRAS) is significantly higher. To confirm our considerations on communication time, it suffices observing the following two figures. It should be mentioned that the influence of number of classes on the performance of PAutoClass on these two data sets is comparable; in fact for DSl the number of classes discovered is 100 and for IRAS classes obtained are about 80. 400 350
*^dL^_
300
^^
y
/
. 250 - predicted
