Conceptual Modeling for Novel Application Domains: ER 2003 Workshops ECOMO, IWCMQ, AOIS, and XSDM, Chicago, IL, USA, October 13, 2003, Proceedings

Lecture Notes in Computer Science Edited by G. Goos, J. Hartmanis, and J. van Leeuwen 2814 3 Berlin Heidelberg New Y...

Author: Manfred A. Jeusfeld | Oscar Pastor

10 downloads 558 Views 6MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form

DOWNLOAD PDF

Lecture Notes in Computer Science Edited by G. Goos, J. Hartmanis, and J. van Leeuwen

2814

3

Berlin Heidelberg New York Hong Kong London Milan Paris Tokyo

Manfred A. Jeusfeld Óscar Pastor (Eds.)

Conceptual Modeling for Novel Application Domains ER 2003 Workshops ECOMO, IWCMQ,AOIS, and XSDM Chicago, IL, USA, October 13, 2003 Proceedings

13

Series Editors Gerhard Goos, Karlsruhe University, Germany Juris Hartmanis, Cornell University, NY, USA Jan van Leeuwen, Utrecht University, The Netherlands Volume Editors Manfred A. Jeusfeld Tilburg University Department of Information Systems and Management P.O. Box 90153, Tilburg, 5000 LE, The Netherlands E-mail: [email protected] Óscar Pastor Polytechnical University of Valencia Camino de Vera s/n, Valencia, 46022 Spain E-mail: [email protected] Cataloging-in-Publication Data applied for A catalog record for this book is available from the Library of Congress. Bibliographic information published by Die Deutsche Bibliothek Die Deutsche Bibliothek lists this publication in the Deutsche Nationalbibliografie; detailed bibliographic data is available in the Internet at .

CR Subject Classification (1998): H.2, H.3, H.4, H.5, K.4.4, I.2 ISSN 0302-9743 ISBN 3-540-20257-9 Springer-Verlag Berlin Heidelberg New York This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, re-use of illustrations, recitation, broadcasting, reproduction on microfilms or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer-Verlag. Violations are liable for prosecution under the German Copyright Law. Springer-Verlag Berlin Heidelberg New York a member of BertelsmannSpringer Science+Business Media GmbH http://www.springer.de © Springer-Verlag Berlin Heidelberg 2003 Printed in Germany Typesetting: Camera-ready by author, data conversion by PTP Berlin GmbH Printed on acid-free paper SPIN: 10950012 06/3142 543210

Preface

ER 2003, the 22nd International Conference on Conceptual Modeling in Chicago, Illinois, hosted four workshops on emerging and maturing aspects of conceptual modeling. While the entity-relationship approach is used to address data (base) modeling, the increasingly connected information infrastructure demands answers that can handle complexity and can develop models about systems that are maintainable. We received seven excellent proposals for workshops to be held at ER 2003, out of which we selected the following four based on peer reviews: – Conceptual Modeling Approaches for E-Business (eCOMO 2003) brought together researchers and practitioners interested in conceptual modeling techniques for e-business. – The International Workshop on Conceptual Modeling Quality (IWCMQ 2003) concentrated on approaches to quality assurance in the modeling process. – The International Bi-Conference Workshop on Agent-Oriented Information Systems (AOIS 2003) was devoted to investigating the agent paradigm for information systems development. – Finally, the International Workshop on XML Schema and Data Management (XSDM 2003) addressed the impact of XML on topics like data integration, change management, and the Semantic Web. All four workshops highlighted relatively new viewpoints on conceptual modeling. Conceptual modeling as such has been greatly influenced and shaped by the entity-relationship model of Peter Chen. However, new developments like object-orientation and the World-Wide Web require adaptions and new techniques. No longer can developers assume that they can completely understand or model the information system. The new developments create challenges in various directions; some of these were discussed in detail in the four ER 2003 workshops: E-Business and E-Commerce. The rise of the Internet has created new opportunities for defining and enacting business relations between partners. The question is how information systems can help in finding business partners, creating new services, and enacting those new services. Any lack of information about some business partners or their products and services needs to be compensated for using some kind of trust-building institution or mechanism. Moreover, services for e-business are not necessarily linked tightly togethers, as used to be the case for information systems developed for single enterprises. Can a service be modeled independently from the provider of the service who is selected at run time? Last but not least, one has to take into account different business (process) models, business contracts, and their monitoring. Hence, the field of e-business stresses the need for comprehensive modeling and analysis techniques. Model Quality. Conceptual models are products of modeling processes undertaken by a group of human experts. Industrial quality management has shifted

VI

Preface

from quality tests at the end of the production process to quality assurance over all product development steps, including the early stages of requirements analysis. The same idea is being applied to improving or at least assessing the quality of conceptual models and the related modeling processes that create them. The more that such models are abstracted from the final implementation, the more difficult it appears to be to assess and control their quality. What constitutes an error in a model? Can we distinguish useful parts of a conceptual model from not so useful parts? Certainly, a team of modelers who are aware of the quality of their products has better opportunities to improve than a team of modelers who are not assessing quality aspects at all. Still, the questions are: what aspects to measure, with which methods, and how frequently? Agent Orientation. Object-orientation is a programming and modeling paradigm that aims at encapsulation (hiding internal details) and re-use (of code and models). While this paradigm is still successful and valid, the lack of information about some components of an information system makes it less applicable to loosely coupled system, like Web services or complex factories that are under constant evolution. Agent orientation provides a promising approach to deal with the increased complexity by including a flavor of autonomy into the components of an agent-oriented system: the co-operating agents have goals and they govern over multiple possible strategies to achieve their goals. The challenge from a conceptual modeling perspective is to represent agent systems in a way that makes them subject to analysis. Suitable languages from agent communication, goal representation, etc., are still under development. XML Data and Schema. The last, but not least, topic covered by the ER 2003 workshops is XML. XML was, after the revolutionary rise of the Internet, in particular the World-Wide Web, an attempt to bring some order into the Web by tagging data elements with labels that indicate their interpretation (or schema). In a way, it is the global representation of interoperable data and perhaps processes. But does XML solve the problems of data/schema integration or does it just shift the problem to a new (yet uniform) syntax? XML databases are already on the market, including XML-based query languages. So, what parts of the traditional data modeling theory can be translated for the XML case? The ER 2003 workshops addressed these issues and created a forum for fruitful discussions. The fact that three of the four workshops have already a long history shows that such discussions are long-term, and convincing answers will only appear after some time. We thank our colleagues in the ER 2003 organization committee for their support. In particular, we thank the organizing chairs of the four workshops who came up with the ideas and imagination that made the workshop program at ER 2003 possible. Last but not least, our special thanks go to the paper authors and the reviewers who created the content of this volume and ensured its high quality. October 2003

Manfred Jeusfeld ´ Oscar Pastor

ER 2003 Workshop Organization

General ER 2003 Workshops Chairs

Manfred A. Jeusfeld Tilburg University, The Netherlands ´ Oscar Pastor Politechnical University of Valencia, Spain

eCOMO 2003 Organization Heinrich C. Mayr Willem-Jan van den Heuvel Christian Kop

University of Klagenfurt, Austria Tilburg University, The Netherlands University of Klagenfurt, Austria

IWCMQ 2003 Organization Jim Nelson Geert Poels Marcela Genero Mario Piattini

Ohio State University, USA Ghent University, Belgium Universidad de Castilla, Spain Universidad de Castilla, Spain

AOIS 2003 Organization Paolo Giorgini Brian Henderson-Sellers

University of Trento, Italy University of Technology, Sydney, Australia

XSDM 2003 Organization Sanjay Madria

University of Missouri-Rolla, USA

VIII

Organization

eCOMO 2003 Program Committee Fahim Akhter Boldur Barbat Boualem Benatallah Anthony Bloesch Antonio di Leva Vadim A. Ermolayev Marcela Genero Martin Glinz J´ ozsef Györk¨ os Bill Karakostas Roland Kaschek Stephen Liddle Zakaria Maamar Norbert Mikula ´ Oscar Pastor Barbara Pernici Matti Rossi Michael Schrefl Daniel Schwabe Il-Yeol Song Bernhard Thalheim Jos van Hillegersberg Ron Weber Carson Woo Jian Yang

Zayed University, United Arab Emirates Lucian Blaga University, Sibiu, Romania University of New South Wales, Sydney, Australia Microsoft Corporation, USA University of Torino, Italy Zaporozhye State University, Ukraine University of Castilla-La Mancha, Ciudad Real, Spain University of Zurich, Switzerland University of Maribor, Slovenia City University, London, UK Massey University, New Zealand Brigham Young University, USA Zayed University, United Arab Emirates Intel Labs, Hillsboro, USA University of Valencia, Spain Politecnico di Milano, Italy Helsinki School of Economics, Finland University of Linz, Austria PUC-Rio, Brazil Drexel University, Philadelphia, USA BTU, Cottbus, Germany Erasmus University, Rotterdam, The Netherlands University of Queensland, Australia UBC, Vancouver, Canada Tilburg University, The Netherlands

Organization

IX

IWCMQ 2003 Program Committee Deb Armstrong Sjaak Brinkkemper Giovanni Cantone Guido Dedene Brian Henderson-Sellers Paul Johannesson Barbara Kitchenham John Krogstie Heinrich Mayr Daniel Moody Jim Nelson Jeff Parsons ´ Oscar Pastor Gustavo Rossi Houari Sahraoui Reinhard Schuette Keng Siau Guttorm Sindre Monique Snoeck Bernhard Thalheim

University of Arkansas, USA Baan, The Netherlands University of Rome, Italy Katholieke Universiteit Leuven, Belgium University of Technology, Sydney, Australia Stockholm University, Sweden Keele University, UK Sintef, Norway University of Klagenfurt, Austria Norwegian University of Science and Technology, Norway Ohio State University, USA Memorial University of Newfoundland, Canada University of Valencia, Spain National University of La Plata, Argentinia Université de Montreal, Canada University of Essen, Germany University of Nebraska-Lincoln, USA Norwegian University of Science and Technology, Trondheim, Norway Katholieke Universiteit Leuven, Belgium Brandenburg University of Technology at Cottbus, Germany

X

Organization

AOIS 2003 Program Committee B. Blake P. Bresciani H.-D. Burkhard L. Cernuzzi L. Cysneiros F. Dignum B. Espinasse I.A. Ferguson T. Finin A. Gal U. Garimella A.K. Ghose G. Karakoulas K. Karlapalem L. Kendall D. Kinny S. Kirn M. Kolp N. Jennings G. Lakemeyer Y. Lespérance D.E. O’Leary F. Lin J.P. Mueller J. Odell O.F. Rana M. Schroeder N. Szirbik F. Zambonelli C. Woo Y. Ye B. Yu

Georgetown University, Washington, DC, USA ITC-IRST, Italy Humboldt Univ., Germany Universidad Cat´ olica Nuestra Se˜ nora de la Asunci´ on, Paraguay York University, Toronto, Canada Univ. of Utrecht, The Netherlands Domaine Universitaire de Saint-Jérôme, France B2B Machines, USA UMBC, USA Technion, Israel Institute of Technology, Israel Andra Pradesh Govt., MSIT, India Univ. of Wollongong, Australia CIBC and Univ. Toronto, Canada Indian Inst. of Information Technology, India Monash University, Australia University of Melbourne Techn. Univ. Ilmenau, Germany Université catholique de Louvain, Belgium Southampton University, UK RWTH Aachen, Germany York University, Canada Univ. of Southern California, USA Hong Kong Univ. of Science and Technology, Hong Kong Siemens, Germany James Odell Associates, USA Cardiff University, UK City University London, UK Technische Universiteit Eindhoven, The Netherlands University of Modena and Reggio Emilia, Italy Univ. British Columbia, Canada IBM T.J. Watson Research Center, USA North Carolina State University, USA

Organization

XI

XSDM2003 Program Committee Elisa Bertino Bharat Bhargava Sourav Bhowmick Tiziana Catarci Qiming Chen Chakravarthy Sharma Kajal Claypool Ee-Peng Lim David W. Embley Alberto H.F. Laender Le Gruenwald Mengchi Liu Qing Li Mukesh Mohania Wee-Keong Ng Stefano Paraboschi Giuseppe Psaila Elke A. Rundensteiner Kian-Lee Tan Katsumi Tanaka Christelle Vangenot Osmar R. Zaiane Xiaofang Zhou

Universit´ a di Milano, Italy Purdue University, USA Nanyang Technological University, Singapore Universit´ a degli Studi di Roma “La Sapienza,” Italy Commerce One, USA University of Texas, Arlington, USA University of Massachusetts, Lowell, USA Nanyang Technological University, Singapore Brigham Young University, USA UFMG, Brazil University of Oklahoma, USA Carleton University, Canada City University of Hong Kong, China IBM Research Lab, India Nanyang Technological University, Singapore University of Bergamo, Italy University of Bergamo, Italy Worcester Polytechnic Institute, USA National University of Singapore, Singapore Kyoto University, Japan EPFL, Switzerland University of Alberta, Canada University of Queensland, Australia

External Referees Gajanan Chinchwadkar Farshad Fotouhi Lars Olsen

Muhammed Al-Muhammed

Table of Contents

Conceptual Modeling Approaches for E-Business at ER 2003 (eCOMO 2003) Preface to eCOMO 2003 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Heinrich C. Mayr, Willem-Jan van den Heuvel Managing Evolving Business Workflows through the Capture of Descriptive Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Sébastien Gaspard, Florida Estrella, Richard McClatchey, Régis Dindeleux The Benefits of Rapid Modelling for E-business System Development . . . . Juan C. Augusto, Carla Ferreira, Andy M. Gravell, Michael A. Leuschel, Karen M.Y. Ng

3

5

17

Prediction of Consumer Preference through Bayesian Classification and Generating Profile . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Su-Jeong Ko

29

Developing Web Applications from Conceptual Models. A Web Services Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ´ Vicente Pelechano, Joan Fons, Manoli Albert, Oscar Pastor

40

A Framework for Business Rule Driven Web Service Composition . . . . . . . Bart Orriëns, Jian Yang, Mike P. Papazoglou

52

Virtual Integration of the Tile Industry (VITI) . . . . . . . . . . . . . . . . . . . . . . . ´ Ricardo Chalmeta, Reyes Grangel, Angel Ortiz, Ra´ ul Poler

65

Second International Workshop on Conceptual Modeling Quality at ER 2003 (IWCMQ 2003) Preface to IWCMQ 2003 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jim Nelson, Geert Poels, Marcela Genero, Mario Piattini

79

Multiperspective Evaluation of Reference Models – Towards a Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Peter Fettke, Peter Loos

80

On the Acceptability of Conceptual Design Models for Web Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Franca Garzotto, Vito Perrone

92

XIV

Table of Contents

Consistency by Construction: The Case of MERODE . . . . . . . . . . . . . . . . . . 105 Monique Snoeck, Cindy Michiels, Guido Dedene Defining Metrics for UML Statechart Diagrams in a Methodological Way . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118 Marcela Genero, David Miranda, Mario Piattini Visual SQL – High-Quality ER-Based Query Treatment . . . . . . . . . . . . . . . 129 Hannu Jaakkola, Bernhard Thalheim Multidimensional Schemas Quality: Assessing and Balancing Analyzability and Simplicity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140 Samira Si-Said Cherfi, Nicolas Prat Conceptual Modeling of Accounting Information Systems: A Comparative Study of REA and ER Diagrams . . . . . . . . . . . . . . . . . . . . . . 152 Geert Poels

Agent-Oriented Information Systems at ER 2003 (AOIS 2003) Preface to AOIS 2003 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167 Paolo Giorgini, Brian Henderson-Sellers Bringing Multi-agent Systems into Human Organizations: Application to a Multi-agent Information System . . . . . . . . . . . . . . . . . . . . . 168 Emmanuel Adam, René Mandiau Reconciling Physical, Communicative, and Social/Institutional Domains in Agent Oriented Information Systems – A Unified Framework . . . . . . . . . 180 Maria Bergholtz, Prasad Jayaweera, Paul Johannesson, Petia Wohed An Agent-Based Active Portal Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . 195 Aizhong Lin, Igor T. Hawryszkiewycz, Brian Henderson-Sellers Agent-Oriented Modeling and Agent-Based Simulation . . . . . . . . . . . . . . . . . 205 Gerd Wagner, Florin Tulba REF: A Practical Agent-Based Requirement Engineering Framework . . . . 217 Paolo Bresciani, Paolo Donzelli Patterns for Motivating an Agent-Based Approach . . . . . . . . . . . . . . . . . . . . 229 Michael Weiss Using Scenarios for Contextual Design in Agent-Oriented Information Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241 Kibum Kim, John M. Carroll, Mary Beth Rosson

Table of Contents

XV

Dynamic Matchmaking between Messages and Services in Multi-agent Information Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244 Muhammed Al-Muhammed, David W. Embley

International Workshop on XSDM at ER 2003 (XSDM 2003) Preface to XSDM 2003 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249 Sanjay Madria A Sufficient and Necessary Condition for the Consistency of XML DTDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 250 Shiyong Lu, Yezhou Sun, Mustafa Atay, Farshad Fotouhi Index Selection for Efficient XML Path Expression Processing . . . . . . . . . . 261 Zhimao Guo, Zhengchuan Xu, Shuigeng Zhou, Aoying Zhou, Ming Li CX-DIFF: A Change Detection Algorithm for XML Content and Change Presentation Issues for WebVigiL . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273 Jyoti Jacob, Alpa Sachde, Sharma Chakravarthy Storing and Querying XML Documents Using a Path Table in Relational Databases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285 Byung-Joo Shin, Min Jin Improving Query Performance Using Materialized XML Views: A Learning-Based Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 297 Ashish Shah, Rada Chirkova A Framework for Management of Concurrent XML Markup . . . . . . . . . . . 311 Alex Dekhtyar, Ionut E. Iacob Object Oriented XML Query by Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323 Kathy Bohrer, Xuan Liu, Sean McLaughlin, Edith Schonberg, Moninder Singh Automatic Generation of XML from Relations: The Nested Relation Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 330 Antonio Badia Toward the Automatic Derivation of XML Transformations . . . . . . . . . . . . 342 Martin Erwig VACXENE: A User-Friendly Visual Synthetic XML Generator . . . . . . . . . . 355 Khoo Boon Tian, Sourav S Bhowmick, Sanjay Madria A New Inlining Algorithm for Mapping XML DTDs to Relational Schemas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 366 Shiyong Lu, Yezhou Sun, Mustafa Atay, Farshad Fotouhi

XVI

Table of Contents

From XML DTDs to Entity-Relationship Schemas . . . . . . . . . . . . . . . . . . . . 378 Giuseppe Psaila Extracting Relations from XML Documents . . . . . . . . . . . . . . . . . . . . . . . . . . 390 Eugene Agichtein, C.T. Howard Ho, Vanja Josifovski, Joerg Gerhardt Extending XML Schema with Nonmonotonic Inheritance . . . . . . . . . . . . . . . 402 Guoren Wang, Mengchi Liu

Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 409

Preface to eCOMO 2003

Today’s increasingly competitive and expanding global marketplace requires companies to cope more effectively with rapidly changing market conditions than ever before. In order to survive in these highly volatile business eco-systems, companies are organizing themselves into integrated (virtual) enterprises, e.g., according to an integrated value chain. Conceptual business, enterprise and process models, either at the level of isolated or integrated enterprises, are heralded as an important mechanism for planning and managing these changes and transitions as well as for designing and constructing the necessary enterprise information systems. However, effective modeling methods and tools still are an issue under research and development. In addition, research issues in the area of business policy specification and change management of enterprise component based models are of paramount importance. The eCOMO workshop series is devoted to these questions. It aims at bringing together experts from practice and academia who are working from several independent, but related perspectives on the same research questions, such as from a business modeling, enterprise application integration, semantic web, business meta-data and ontologies, process management, business reengineering and business communication language perspectives. eCOMO2003 continues three highly successful predecessor eCOMO workshops which were held during ER’2000 in Salt Lake City, ER’2001 in Yokohama, and ER’2002 in Tampere. The program of eCOMO2003 is the result of a thorough review process in which each of the submitted papers was assessed by three experienced reviewers. At the end of the review process, more than the six papers printed herein were rated worth to be published and presented at the workshop. However, the program committee had to make its final decision according to the rules of ER and LNCS. The selected six contributions mainly deal with business process modeling and management aspects in the context of agile web-application development, most of them adopting the emerging Service Oriented Computing (SOC) paradigm. This paradigm is nowadays principally manifested by web-service technology, and reflects the ongoing migration from object-oriented and component-based modeling and development to a novel way of conceptualizing, designing and constructing light-weighted and web-enabled software components on an as-needed basis. Many persons deserve appreciation and recognition for their contribution to make eCOMO’2003 a success. First of all we have to thank the authors for their valuable contributions. Similarly, we thank the members of the program committee, who spent a lot of time in assessing submitted papers and participating in the iterated discussions on acceptance or rejection. Special appreciation is due to Christian Kop, who organized and co-coordinated the whole preparation

4

Preface to eCOMO 2003

process including the composition of these proceedings. Last but not least we thank the ER organizers and the ER workshop co-chairs Manfred Jeusfeld and Oscar Pastor for their support in integrating eCOMO2003 into ER’2003.

October 2003

Heinrich C. Mayr Willem-Jan van den Heuvel

Managing Evolving Business Workflows through the Capture of Descriptive Information 1,2,3

Sébastien Gaspard 1

1

1

, Florida Estrella , Richard McClatchey , 2,3 and Régis Dindeleux

CCCS, University of the West of England, Frenchay, Bristol BS16 1QY UK [email protected] 2 LLP/ESIA, Université de Savoie, Annecy, 74016 CEDEX, France [email protected] 3 Thésame Mécatronique et Management, Annecy, 74000, France rd@thésame-innovation.com

Abstract. Business systems these days need to be agile to address the needs of a changing world. In particular the discipline of Enterprise Application Integration requires business process management to be highly reconfigurable with the ability to support dynamic workflows, inter-application integration and process reconfiguration. Basing EAI systems on model-resident or on a socalled description-driven approach enables aspects of flexibility, distribution, system evolution and integration to be addressed in a domain-independent manner. Such a system called CRISTAL is described in this paper with particular emphasis on its application to EAI problem domains. A practical example of the CRISTAL technology in the domain of manufacturing systems, called Agilium, is described to demonstrate the principles of model-driven system evolution and integration. The approach is compared to other modeldriven development approaches such as the Model-Driven Architecture of the OMG and so-called Adaptive Object Models.

1 Background and Related Works As the global marketplace becomes increasingly complex and intricately connected, organizations are constantly pressured to re-organize, re-structure, diversify, consolidate and slim down to provide a winning competitive edge. With the advent of the Internet and e-commerce, the need for coexistence and interoperation with legacy systems and for reduced ’times-to-market’, the demand for the timely delivery of flexible software has increased. Couple to this the increasing complexity of systems and the requirement for systems to evolve over potentially extended timescales and the importance of clearly defined, extensible models as the basis of rapid systems design becomes a pre-requisite to successful systems implementation. One of the main drivers in the object-oriented design of information systems is the need for the reuse of design artefacts or models in handling systems evolution. To be able to cope with system volatility, systems must have the capability of reuse and to adapt as and when necessary to changes in requirements. The philosophy that has been investigated in the research reported in this paper is based on the systematic M.A. Jeusfeld and Ó. Pastor (Eds.): ER 2003 Workshops, LNCS 2814, pp. 5–16, 2003. © Springer-Verlag Berlin Heidelberg 2003

6

S. Gaspard et al.

capture of the description of systems elements covering multiple views of the system to be designed (including processes and workflows) using common techniques. Such a description-driven approach [1, 2] involves identifying and abstracting the crucial elements (such as items, processes, lifecycles, goals, agents and outcomes) in the system under design and creating high-level descriptions of these elements which are stored and managed separately from their instances. Description-driven systems (DDS) make use of so-called meta-objects to store domain-specific system descriptions that control and manage the life cycles of metaobject instances or domain objects. The separation of descriptions from their instances allows them to be specified and managed and to evolve independently and asynchronously. This separation is essential in handling the complexity issues facing many web-computing applications and allows the realization of inter-operability, reusability and system evolution as it gives a clear boundary between the application’s basic functionalities from its representations and controls. In a description-driven system as we define it, process descriptions are separated from their instances and managed independently to allow the process descriptions to be specified and to evolve asynchronously from particular instantiations of those descriptions. Separating descriptions from their instantiations allows new versions of elements (or element descriptions) to coexist with older versions. In this paper the development of Enterprise Resource Planning (ERP) in flexible business systems is considered and the need for business process modelling in Enterprise Application Integration (EAI) [3] is established. Workflow systems are considered as vehicles in which dynamic system change in EAI can be catered for as part of handling system evolution through the capture of system description. A description-driven approach is proposed to enable the management of workflow descriptions and an example is given of an application of the CRISTAL descriptiondriven system developed at CERN [4] to handle dynamic system change in workflows. This approach has today two parallel implementations that are called CRISTAL for CMS and Agilium. The two applications are based on the same kernel called CRISTAL KERNEL which provides the DDS functionalities. Both applications inherit these functionalities even if the goals and the specifics of each application is radically different.

2 The Need for Integrated Business Process Modelling in EAI In recent years, enterprises have been moving from a traditional function-led organisation, addressing the needs of a “vertical” market, to a “horizontal” organisation based on business processes. The emergence of new norms such as the ISO 9001 V2000 and the development of inter-enterprise exchanges are further drivers towards process-led reorganisation. However, currently available information systems and production software are still organised following function models. Consequently, they are not well adapted to the exchange of information between enterprises nor to coping with evolving process descriptions. In modern enterprises organised following a horizontal structure, industrial EAI solutions are very dependent on process performance and on the ability of the underlying enterprise management to execute and automate the business processes. Furthermore the requirement for the support of enterprise activities is not only for the execution of

Managing Evolving Business Workflows

Processes ROUTING Data Transformation

Transport

7

Workflow Engine

Specific DB

External System

Fig. 1. The three basic layers of an Enterprise Application Integration (EAI) system

internal processes but also for external processes, as in the support of suppliercustomer relationships especially in supply chain management. Enterprise processes have to integrate increasingly more complex business environments including domain-dependent processes for managing both interapplication operation and inter-organisation operation where efficient communications is crucial. Integration sources across enterprises are numerous and multi-technological and can include ERP, human resource management, Customer Relation Management (CRM), administration software, Intranet /Internet, proprietary software and a plethora of office tools. The first step that an enterprise must make in order to move from a standard vertical organisation to a horizontal organisation is to chart its existing business processes and the interactions between these processes. Following this it must update and manage its internal processes based on existing information systems. For that, the enterprise may be confronted by a collection of different production software systems and their abilities to interact. Most of the software offerings that support ERP deal with the description of enterprises through its organisation by function and examples of these products include systems for purchase service, stock management, production management etc. However individual systems need to synchronise with each other and each normally has their own process management models. Most commercial software do not provide tools to aid in process description and evolution. Even when workflow engines (which can provide synchronisation between systems) are integrated within ERP systems, they are for the most part not synchronised with external applications of the ERP system. EAI [3] systems concentrate on an architecture centred on software which is dedicated to the interconnection of heterogeneous applications that manage information flows. The heart of EAI software is normally based on the concept of interface references where transformation, routing and domain dependent rules are centralised. Standard EAI architecture is normally built on three layers: processes, routing and transport layers as shown in figure 1. At the lowest layer are the so-called “connectors” which allow external applications to connect to the EAI platform. It is this level that manages the transport of messages between applications. This Transport layer can be based on basic technologies such as Message Oriented MiddleWare (MOM) [5], and on file reading,

8

S. Gaspard et al.

email and technologies such as HTTP. The middle layer level of standard EAI software (the Routing layer) manages the transformation of data and its routing between internal systems. More evolved technologies such as XML/XSLT/SOAP, Electronic Data Interchange (EDI), “home-made” connectors and a transition database are used to provide the routing capabilities in this layer. The function of this layer is to apply transformation rules to source application data and to route the new information to the required target application. The third layer of an EAI system is dedicated to system modelling. At this layer, a workflow engine managing domaindependent specific processes is often employed (when available). Technically this EAI model suffers from a number of problems: • The management and modelling of processes needs specific development. Where a workflow engine is used for this purpose, workflows are often based on a monolithic architecture using a matrix definition of workflows that is fixed for the lifecycle of the system. • Specific technologies are used. In MOM solutions data transformations are not normally based on non-generic tools but on internal developments. Even if XML is used, for most of the case the data dictionary is not defined. • Guidelines for implementing Connectors do not exist Connectors have to be fully specified, developed and adapted to the connected application. Any change in the EAI software or the connected software requires redevelopment of the Connectors. • Most of the time, the EAI software has to be placed on a single server, which manages all the processes and has to support three different applications (one for each layer of the EAI model) that have more or less been created to be used together. • An expensive application server or database management system (DBMS) needs to be already installed and maintained. As expressed by Joeris in 1999 [6], the support of heterogeneous processes, flexibility, reuse, and distribution are great challenges for the design of the next generation of process modelling languages and their enactment mechanisms. These process modelling technologies are important for business and can make the management of systems more reactive and efficient. However it is not sufficient to concentrate solely on process management and enactment to solve all the problems identified in EAI in the previous section. If workflow systems are not coupled to a comprehensive data management model, optimum functionalities cannot be realised [7]. Most of recent workflow research has been focused on the enactment and modelling of processes based on Petri nets [8], CORBA [9] or UML concepts [10] however when issues of reconfiguration1 are considered, these research solutions only provides a high level of workflow control that does not completely address the enterprise problems listed above. The research outlined in this paper proposes an approach that deals with a high level of management of processes with the ability to manage complex data. It is based on distributed technologies and allows the relative autonomy of activity execution, with an enactment model that is relatively similar to that of Joeris [11]. Coupling this technology with some abstraction of process description that can provide generic 1

Reconfiguration : The ability of a system to dynamically change executing instances of processes in line with a change in its description.


Meta-Meta-Model Layer

9

MOF

Is an instance of Meta-Model Layer UML Is an instance of Model Layer

Information Domain

Is an instance of Instance Layer

User Objects

Fig. 2. The four-layer architecture of the OMG

workflow models [12] is a suitable alternative to standard EAI solutions and more closely addresses the problems listed earlier.

3 Handling Evolution via System Description Approaches for handing system evolution through reuse of design artefacts have led to the study of reusable classes, design patterns, frameworks and model-driven development. Emerging and future information systems however require more powerful data modelling techniques that are sufficiently expressive to capture a broader class of applications. Compelling evidence suggests that the data model must be OO, since that is the model that currently maximises generality. The data model needs to be an open OO model, thereby coping with different domains having different requirements on the data model [13]. We have realised that object metamodelling allows systems to have the ability to model and describe both the static properties of data and their dynamic relationships, and address issues regarding complexity explosion, the need to cope with evolving requirements, and the systematic application of software reuse. To be able to describe system and data properties, object meta modelling makes use of meta-data. Meta-data are information defining other data. Figure 2 shows the familiar four-layer model of the Object Management Group, OMG, embodying the principles of meta modelling. Each layer provides a service to the layer above it and serves as a client to the layer below it. The meta-meta-model layer defines the language for specifying meta-models. Typically more compact than the meta-model it describes, a meta-meta-model defines a model at a higher level of abstraction than a meta-model. The meta-model layer defines the language for specifying models, a meta-model being an instance of a meta-meta-model. The model layer defines the language for specifying information domains. In this case, a model is an instance of a meta-model. The bottom layer contains user objects and user data, the instance layer describing a specific information domain. The OMG standards group has a similar

10

S. Gaspard et al.

architecture based on model abstraction, with the Meta-Object Facility (MOF) model and the UML [14] model defining the language for the meta-meta-model and metamodel layers, respectively. The judicious use of meta-data can lead to heterogeneous, extensible and open systems. Meta-data make use of a meta-model to describe domains. Our recent research has shown that meta modelling creates a flexible system offering the following - reusability, complexity handling, version handling, system evolution and inter-operability. Promotion of reuse, separation of design and implementation and reification are some further reasons for using meta-models. As such, meta modelling is a powerful and useful technique in designing domains and developing dynamic systems. A reflective system utilizes an open architecture where implicit system aspects are reified to become explicit first-class meta-objects [15]. The advantage of reifying system descriptions as objects is that operations can be carried out on them, like composing and editing, storing and retrieving, organizing and reading. Since these meta-objects can represent system descriptions, their manipulation can result in change in the overall system behaviour. As such, reified system descriptions are mechanisms that can lead to dynamically evolvable and reusable systems. Metaobjects, as used in the current work, are the self-representations of the system describing how its internal elements can be accessed and manipulated. These selfrepresentations are causally connected to the internal structures they represent i.e. changes to these self-representations immediately affect the underlying system. The ability to dynamically augment, extend and redefine system specifications can result in a considerable improvement in flexibility. This leads to dynamically modifiable systems, which can adapt and cope with evolving requirements. There are a number of OO design techniques that encourage the design and development of reusable objects. In particular design patterns are useful for creating reusable OO designs [16]. Design patterns for structural, behavioural and architectural modelling have been well documented elsewhere and have provided software engineers with rules and guidelines that they can (re-)use in software development. Reflective architectures that can dynamically adapt to new user requirements by storing descriptive information which can be interpreted at runtime have lead to socalled Adaptive Object Models [17]. These are models that provide meta-information about domains that can be changed on the fly. Such an approach, proposed by Yoder, is very similar to the approach adopted in this paper A Description-Driven System (DDS) architecture [1, 2], as advocated in this paper, is an example of a reflective meta-layer (i.e. meta-level and multi-layered) architecture. It makes use of meta-objects to store domain-specific system descriptions, which control and manage the life cycles of meta-object instances, i.e. domain objects. The separation of descriptions from their instances allows them to be specified and managed and to evolve independently and asynchronously. This separation is essential in handling the complexity issues facing many web-computing applications and allows the realization of inter-operability, reusability and system evolution as it gives a clear boundary between the application’s basic functionalities from its representations and controls. As objects, reified system descriptions of DDSs can be organized into libraries or frameworks dedicated to the modelling of languages in general, and to customizing its use for specific domains in particular. As a practical example of our approach the next section describes the DDS architecture developed in


11

the context of research carried out in the CRISTAL project at CERN and the Agilium project at Thesame.

4 CRISTAL – A Description-Driven System (DDS) The Compact Muon Solenoid (CMS) is a general-purpose experiment at CERN that will be constructed from around a million parts and will be produced and assembled in the next decade by specialized centres distributed worldwide. As such, the construction process is very data-intensive, highly distributed and ultimately requires a computer-based system to manage the production and assembly of detector components. In constructing detectors like CMS, scientists require data management systems that are able to cope with complexity, with system evolution over time (primarily as a consequence of changing user requirements and extended development timescales) and with system scalability, distribution and interoperation. No commercial products provide the capabilities required by CMS. Consequently, a research project, entitled CRISTAL (Cooperating Repositories and an Information System for Tracking Assembly Lifecycles [4]) has been initiated to facilitate the management of the engineering data collected at each stage of production of CMS. CRISTAL is a distributed product data and workflow management system, which makes use of an OO database for its repository, a multi-layered architecture for its component abstraction and dynamic object modelling for the design of the objects and components of the system. CRISTAL is based on a DDS architecture using metaobjects. The DDS approach has been followed to handle the complexity of such a data-intensive system and to provide the flexibility to adapt to the changing scenarios found at CERN that are typical of any research production system. In addition CRISTAL offers domain-independence in that the model is generic in concept. Lack of space prohibits further discussion of CRISTAL; detail can be found in [1, 2 & 4]. The design of the CRISTAL prototype was dictated by the requirements for adaptability over extended timescales, for system evolution, for interoperability, for complexity handling and for reusability. In adopting a description-driven design approach to address these requirements, the separation of object instances from object description instances was needed. This abstraction resulted in the delivery of a three layer description-driven architecture. The model abstraction (of instance layer, model layer, meta-model layer) has been adapted from the OMG MOF specification [18], and the need to provide descriptive information, i.e. meta-data, has been identified to address the issues of adaptability, complexity handling and evolvability. Figure 3 illustrates the CRISTAL architecture. The CRISTAL model layer is comprised of class specifications for CRISTAL type descriptions (e.g. PartDescription) and class specifications for CRISTAL classes (e.g. Part). The instance layer is comprised of object instances of these classes (e.g. PartType#1 for PartDescription and Part#1212 for Part). The model and instance layer abstraction is based on model abstraction and on the Is an instance of relationship. The abstraction based on meta-data abstraction and the Is described by relationship leads to two levels - the meta-level and the base level. The meta-level is comprised of meta-objects and the meta-level model that defines them (e.g. PartDescription is the meta-level model of PartType#1 meta-object). The base level is comprised of base objects and the base level model that defines them (Part is the base-level model of Part#1212 object).

12

S. Gaspard et al.

UML Meta-Model Is an instance of Base-Level Model Of

Is described by

ClassPart

ClassPartDescription

Is an instance of

Meta-Level Model Of

Is an instance of

Is described by

PartObject#1212

PartType#1

Base- Level

Meta-Level

Objects/ Data

Is an instance of

Meta-Objects/ Meta-Data

Fig. 3. The CRISTAL description-driven architecture.

The approach of reifying a set of simple design patterns as the basis of the description-driven architecture for CRISTAL has provided the capability of catering for the evolution of a rapidly changing research data model. In the two years of operation of CRISTAL it has gathered over 25 Gbytes of data and been able to cope with more than 30 evolutions of the underlying data schema without code or schema recompilations.

5 Agilium – A Description Driven Workflow System 5.1 Agilium Functionality In order to address the deficiencies in current EAI systems a research system entitled Agilium, based on CRISTAL technologies, has been developed by a collaboration of three research partners, CERN (the European Organisation for Nuclear Research, in Geneva Switzerland), UWE (the University of the West of England, Bristol UK) and Thésame (an innovation network for technology companies in mechatronic, computer-integrated manufacturing and management, based in Annecy France). The model and technologies used in Agilium make EAI tools accessible to middle-sized enterprises and to software houses and integrators that target the EAI market. The CRISTAL architecture is providing a coherent way to replace the three application layers of EAI (as shown in figure 1) by a single generic layer, based on Items and using common tools, processes, routing and transport. In order to provide an effective EAI architecture, Agilium combines Items described using the DDS philosophy. This approach provides a development free way to provide EAI functionality whilst managing behaviour through workflows. Items can be connectors or conceptual domain-specific objects such as order forms, supplies, commands etc.


13

&5,67$/$SSURDFK ITEM : Process History Data

Item 1

Item 5

Item 2

Process History Data

ROUTING ROUTAGE

Data transformation

Data transformation

Communication

Communication

External System

External System

Fig. 4. The CRISTAL approach to integrated EAI employed in Agilium.

In the Agilium system, a connector is managed as one single Item described with a specific graphically represented behaviour. A connector can transform data (using scripting or XML technologies) and is coupled to a communication method that can be any appropriate enabling technology. In this way, it is easy to connect applications that have any arbitrary communication interface. It is sufficient simply to describe a communication mode (CORBA, HTTP, SOAP, Web Service, text file, email...), a data format (will be converted in XML) and a behaviour (with the workflow graphical interface). Basing on the concept of the DDS, connectors are easily maintainable and modifiable and make the Agilium system easy to integrate and adapt to evolving environments prevalent in enterprise information systems. By combining the Items describing domain specific functionality and those that can connect external applications, the EAI architecture is complete and presents all the functionalities of the external architectures, and more. Using the facilities for description and dynamic modification in CRISTAL, Agilium is able to provide a modifiable and reconfigurable workflow. The workflow description and enactment elements of CRISTAL are correlated and each instance stores any modifications that have been carried on it. With an efficient model of verification at both levels (i.e. description and enactment) it is possible to validate if the migration from one description to another one within an instance is possible and to detect any modifications and changes and therefore to apply the migration. Ongoing research is being conducted to mathematically model the workflow concepts that could be directly applied to the CRISTAL technologies so as to complete the modification ability. 5.2 Advantages and Limitations of Agilium Innovating technologies used in the kernel of CRISTAL provide Agilium significant advantages when compared to standard EAI solutions:

14

S. Gaspard et al.

• Flexibility. Architecture independence allows Agilium to adapt to new domains and/or new enterprises without any specific development. This is an essential factor in helping to reduce maintenance costs and to minimise conversation costs, thus providing high levels of flexibility. • Platform independence. Use of JAVA/CORBA/XML/LDAP technologies allows CRISTAL to work on any preinstalled operating system (Linux, Windows, UNIX, Mac OS) and on any machine (Mac, PC, Sun, IBM...). • Database independence. XML storage with LDAP referencing makes CRISTAL autonomous and independent of any underlying database and of any application server. • Simplified integration of applications. XML is becoming the standard for interfacing applications within an enterprise. It presents a solution that supports multiple heterogeneous environments. With the limitation of the development of a translation/transport layer, connectors are based on a generic model. • Fully distributed. This functionality provides web usability through the Internet or an Intranet. It also makes data accessible from multiple databases via a unique interface. • CRISTAL’s powerful workflow management facilities provide the ability to model and execute processes of any type that can also evolve by dynamic modification or appliance of any new description. But there are some limitations: • Because it is based on graphical descriptions, it is not always simple to determine the way to code (describing a workflow) CRISTAL actions. Using a high level of abstraction can render simple things difficult to represent in code. • Providing complete flexibility to users to define elements in the system can compromise the integrity of enactment and the implementation is not yet sufficiently advanced to provide a fully secured system without requiring human intervention. • As the connectivity technologies such as BPML and BPEL4W [19] become more complex, complete, normative and numerous, the Agilium tool has to provide and maintain many connectors that can not only be defined by the user. 5.3 Future Work Ongoing research is being conducted into the mathematical approach of process modelling in CRISTAL which may ultimately include a decision-making model based on agent technology. It is planned that these agents would verify changes to the model and dynamically modify the instances of the workflow processes basing its calculation and decision on a base of user predefined constraints that must be respected. Another aspect that may be explored is to use an Architecture Definition Language to model Items and their interactions. This would provide an efficient and secure way to create descriptions for new domains to model. Another area that could be explored is the connector aspects for the Agilium EAI. A mathematical approach of connector specifications could be envisaged and defined which would enable connector development to be automated in CRISTAL.


15

6 Conclusions The combination of a multi-layered meta-modelling architecture and a reflective meta-level architecture resulted in what has been referred to in this paper as a DDS architecture. A DDS architecture, is an example of a reflective meta-layer architecture. The CRISTAL DDS architecture was shown to have two abstractions. The vertical abstraction is based on the is an instance of relationship from the OMG meta-modelling standard, and has three layers - instance layer, model layer and metamodel layer. This paper has proposed an orthogonal horizontal abstraction mechanism that complements this OMG approach. The horizontal abstraction is based on the meta-level architecture approach, encompasses the is described by relationship and has two layers - meta-level and base level. This description-driven philosophy facilitated the design and implementation of the CRISTAL project with mechanisms for handling and managing reuse in its evolving system requirements and served as the basis of the Agilium description-driven workflow system. The model-driven philosophy expounded in this paper is similar to that expounded in the Model Driven Architecture (MDA [18]) of the OMG. The OMG’s goal is to provide reusable, easily integrated, easy to use, scalable and extensible components built around the MDA. While DDS architectures establish those patterns, which are required for exploiting data appearing at different modelling abstraction layers, the MDA approaches integration and interoperability problems by standardizing interoperability specification at each layer (i.e. standards like XML, CORBA, .NET, J2EE). The MDA integration approach is similar to the Reference Model for Open Distributed Processing (RM-ODP) [20] strategy of interoperating heterogeneous distributed processes using a standard interaction model. In addition, the Common Warehouse Metamodel (CWM) specification [21] has been recently adopted by the OMG. The CWM enables companies to manage their enterprise data better, and makes use of UML, XML and the MOF. The specification provides a common metamodel for warehousing and acts as a standard translation for structured and unstructured data in enterprise repositories, irrespective of proprietary database platforms. Likewise, the contributions of this work complement the ongoing research on Adaptive Object Model (AOM) espoused in [17] and [22], where a system with an AOM (also called a Dynamic Object Model) is stated to have an explicit object model that is stored in the database, and interpreted at runtime. Objects are generated dynamically from the AOM schema meta-data that represent data descriptions. The AOM approach also uses reflection in reifying implicit data aspects (e.g. database schema, data structures, maps of layouts of data objects, references to methods or code). The description-driven philosophy has demonstrably facilitated the design and implementation of the CRISTAL and Agilium projects with mechanisms for handling and managing reuse in its evolving system requirements. Acknowledgments. The authors take this opportunity to acknowledge the support of their home institutes and numerous colleagues responsible for the CRISTAL & Agilium software.

16

S. Gaspard et al.

References 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22.

Kovacs Z, “The Integration of Product Data with Workflow Management Systems”, PhD Thesis, University of West of England, Bristol, England, April 1999. Estrella F, "Objects, Patterns and Descriptions in Data Management", PhD Thesis, University of the West of England, Bristol, England, December 2000. Mann J., “Workflow and Enterprise Application Integration” , 2001, analyst in Middleware. Estrella F et al., "Handling Evolving Data Through the Use of a Description Driven Systems Architecture". Lecture Notes in Computer Science Vol 1727, pp 1–11 ISBN 3540-66653-2 Springer-Verlag, 1999 Rao, B.R. "Making the Most of Middleware." Data Communications International 24, 12 (September 1995): 89–96. Joeris G., “Toward Flexible and High-level Modeling and Enactment of Processes”, University of Bremen, 1999. Sheth A. P., W.M.P van der Aalst, “Processes Driving the Networked Economy”, University of Georgia, 1999. W.M.P van der Aalst, “Making Work Flow: On the Application of Petri nets to Business Process Management”, Eindhoven University of Technology, 2001. Tari Z., Pande V, “Dynamic Workflow Management in CORBA Distributed Object Systems”, RMIT University, IBMG GSA, 2000 Torchiano M., Bruno G., “Domain-Specific instance Models in UML”, IDI NTNU, 2002 Joeris G., “Decentralized and Flexible Workflow Based on Task Coordination Agents”, University of Bremen, 2000 W.M.P van der Aalst, “Generic Workflow Models: How to Handle Dynamic Change and Capture Management Information?”, Eindhoven University of Technology, 1999 Klas W and Schrefl M, “Metaclasses and their Application. Data Model Tailoring and Database Integration”, Lecture Notes in Computer Science 943. Springer. 1995. The Unified Modelling Language (UML) Specification, URL http://www.omg.org/technology/uml/ Kiczales G, “Metaobject Protocols: Why We Want Them and What Else Can They Do?”, Chapter in Object-Oriented Programming: The CLOS Perspective, pp 101–118, MIT Press, 1993. Gamma E, Helm R, Johnson R and Vlissides J, "Design Patterns: Elements of Reusable Object-Oriented Software", Addison-Wesley, 1995. Yoder J, Balaguer F & Johnson R., "Architecture and Design of Adaptive Object-Models". Proc of OOPSLA 2001, Intriguing Technology Talk, Tampa, Florida. October 2001. OMG Publications., "Model Driven Architectures – The Architecture of Choice for a Changing World". See http://www.omg.org/mda/index.htm BPML: Business Process Modeling Language. See http://www.bpmi.org/, BPEL4W : Business Process Execution Language for Web Services. See http://www.ibm.com/developerworks RM-ODP. A Reference Model for Open Distributed Processing http://www.dstc.edu.au/Research/Projects/ODP/ref_model.html Common Warehouse Metamodel & Meta-Object Facility. See http://www.omg.org/technology/cwm/ Foote B and Yoder J “Meta-data and Active Object-Models”. Proc. of the Int. Conference on Pattern Languages Of Programs, Monticello, Illinois, USA, August 1998.

The Benefits of Rapid Modelling for E-business System Development Juan C. Augusto, Carla Ferreira, Andy M. Gravell, Michael A. Leuschel, and Karen M.Y. Ng Department of Electronics and Computer Science University of Southampton, {jca,cf,amg,mal,myn00r}@ecs.soton.ac.uk

Abstract. There are considerable difficulties modelling new business processes. One approach is to adapt existing models, but this leads to the difficult problem of maintaining consistency between model and code. This work reports an investigation into creating quick models that are nonetheless useful in providing insight into proposed designs.

1

Introduction

There are considerable difficulties modelling new business processes. One approach is to adapt existing models, but this leads to the difficult problem of maintaining consistency between model and code. In eXtreme Programming [Bec00], for example, we are advised to “travel light” – questions are answered by examining the code rather than trusting written designs which may be out of date. This work reports an investigation into creating quick “throw away” models that are nonetheless useful in providing insight into proposed designs. These models are not merely pictures, but can be “executed” through animation/simulation, and can be comprehensively checked, at least for specific configurations, by model checking. This answers the criticism that pictures “can’t give concrete feedback” [Beck 2000]. In sections 2 and 3 we first provide a brief description of two of the modelling frameworks we considered, Promela/SPIN and CSP(LP)/CIA. Then in section 4 a case study that we used as the basis for our experiment is introduced. In section 5 we give some details about the modelling approach we followed when using Promela and CSP(LP) as modelling languages. Section 6 explains how tool assisted development can provide the basis for rapid modelling with important expected benefits and section 7 explains the extent to which we experienced those advantages while using the above mentioned tools to our case study. Later, in section 8 we consider a more specific modelling language, called StAC for business modelling. An analysis of work in progress is given in section 9 while an account of some lessons learnt and the final conclusions are provided in section 10. ´ Pastor (Eds.): ER 2003 Workshops, LNCS 2814, pp. 17–28, 2003. M.A. Jeusfeld and O. c Springer-Verlag Berlin Heidelberg 2003

18

J.C. Augusto et al.

2

Promela/SPIN

SPIN [Hol97] has been a particularly successful tool that has been widely adopted to perform automatic verification of software specifications. SPIN offers the possibility to perform simulations and verifications. Through these two modalities the verifier can detect absence of deadlocks and unexecutable code, to check correctness of system invariants, to find non-progress executions cycles and to verify correctness properties expressed in propositional linear temporal logic formulae. Promela is the specification language of SPIN. It is a C-like language enriched with a set of primitives allowing the creation and synchronization of processes, including the possibility to use both synchronous and asynchronous communication channels. We refer the reader to the extensive literature about the subject as well as the documentation of the system at Bell Labs web site for more details: http://netlib.bell-labs.com/netlib/spin/whatispin.html We assume some degree of familiarity with this framework from now on.

3

CSP(LP)/CIA

CSP(LP) [Leu01] unifies CSP [Hoa85] with concurrent (constraint) logic programming. Elementary CSP, without datatypes, functions, or other advanced operators, was extended in CSP-FDR [Ros99] to incorporate this features, which we want for modelling business systems. Some of the remaining limitations on pattern matching were overcome in CSP(LP) (see [Leu01] section 2.2. for a more detailed account). A basic introduction to CSP(LP) syntax [Leu01] follows: Operator stop skip prefix conditional prefix external choice internal choice interleaving parallel composition sequential composition hiding renaming timeout interrupt if then else let expressions agent definition

Syntax STOP SKIP a→P a?x : x > 1 → P P 2Q P Q P |||Q P [|A |] Q P;Q P \A P [R] P Q P i Q if t then P else Q let v = e in P A=P

Ascii Syntax STOP SKIP a->P a?x:x>1->P P [] Q P || Q P ||| Q P [| A |] Q P ->> Q P \\ A P [[ R ]] P [> Q P / Q if T then P else Q let V=E in P A = P;

The CSP(LP) Interpreter and Animator, CIA, [LAB+ 01] can be used to animate and detect deadlocks in a CSP(LP) specification.

The Benefits of Rapid Modelling

4

19

The Travel Agency Case Study

As an example of an e-business system involving collaboration across organizations consider a travel agent [LR00]. A travel agent gets requests from the users that log into the travel agency system by using a browser. After selecting an operation (book or unbook) and a service (car or room), the operation is submitted to the travel agent. The travel agent will decide what service provider (Car Rental or Hotel) to contact on the basis of previous requests made by the user. The request is passed on to one of the service providers, which will decide if the operation can be accomplished or not. For example, it could be that the user requests to book a service that is not available or to unbook a service that was not previously booked to her/him. The shop will contact the travel agent to make explicit if the operation was or was not successful. The travel agent will pass this information to the user. If the operation was successful, the shop and the travel agent will keep records of that on their own databases. A sketch of a typical session can be seen as an appendix in [AFG+ 03]. We have built a prototype of this system using J2EE technology to experiment with the expected functionality and to uncover the basic operations and communications demanded by such e-businesses systems. On the other hand we built different models to experiment with different modelling languages, different tools, and to compare the support they offer to a development team.

5

Modelling Approaches

Our models are in widely used notations that have defined semantics and tool support. These notations are capable to deal with essential notions for e-business applications like concurrency and synchronous/asynchronous message passing. These frameworks allows the creation of simple and abstract models that can be simulated and rigorously checked. Due to space constraints we cannot offer complete models but, we provide a brief description of them as appendixes A and B to give the reader a flavor of how they look like. The complete models fully documented can be seen in the appendixes given in [AFG+ 03]. Next we provide a sketch of the basic structures we need and the functionality we expect from each major part of the system. Communication between the user, the travel agent, and the shops in the prototype is accomplished trough sessions and the underlying web connectivity message system. In our models that was modelled via synchronous channels. We considered (1) a channel to pass requests from the user to the travel agent (2) channels to pass requests from the travel agent to each shop, and (3) channels to get feedback from the shops about if the operation was or was not successful. Another important aspect has to do with the side effects of the interaction in the system. For example, as a result of a successful operation each shop will have to register a change on its database to remember that a resource was taken or released so we need in the models some structures to mimic the databases implemented in the prototype by using JDBC technology. The travel agent has

20

J.C. Augusto et al.

its own database, where all the operations are recorded and its content has to be consistent with all the shops databases except for the intermediate state where a record was made in a shop database but still was not transferred to the travel agent database. But, because the communication is assumed to be synchronous that will eventually occur and because decisions in the system are based only on the shop’s database contents this do not cause any harm in the system. Of course, the travel agent will know that if a request has not been answered then the information cannot be considered as an up to date account of the system.

6

Checking Techniques

After running this experiment we were able to collect some interesting experiences. On a higher level we can say that by building the models we were forced to revise and double check the relationships between all the important parts of the system. This also suggests that a realistic expectation is therefore that modelling a system is about four times quicker than prototyping it. While the prototype involved several weeks from a team of three programmers each model was about one and a half week for one person effort. In all cases the people involved had the same level of expertise required to use the necessary technology during both the prototyping and the modelling stages of the development. We do not of course propose developers should construct multiple models. We did so ourselves only to compare notations and tools. Both tools assisted the modelling stage with syntax and type checking, basic model checking, e.g. infinite loops and deadlock detection, and animation facilities. After no more basic errors were found some simulations were carried out to compare the behavior of the model with the behavior of the prototype and the one expected from the system. By building this models of the system we have been able to check behavioral properties that allowed us to pinpoint some interesting aspects of the system: Example 1 (Credit card loop) Part of the user interaction with the system involves providing an authorized credit card brand. The initial prototype allowed users to introduce an unbounded number of attempts to provided their credit card brand. Both tools, SPIN and CIA, allowed us to detect that. Example 2 (Deadlocks) Communications between user, travel agent and shops was implemented via synchronous channels. Sometimes during the construction of the model the interaction of the different processes was very important to detect how interdependent the different parts of the system were to each other. This was especially well considered in SPIN were there is a graphical interface focused on channel communication. Example 3 (detecting subtler errors). An error was introduced in purpose during the construction of the prototype to experiment how we were able to detect it at modelling time. The error is related to the strategy that the Travel Agency has to handle second reservations. This strategy was left unfinished so that when the


21

travel agency is asked to book a room in a hotel for a second time by the same user, it tries to book the room in the same hotel used for the first booking. When failing to find another room available the travel agency will not try to book the room in another hotel. Instead it will consider the operation unsuccessful. We were able to detect the potential anomaly during simulation and then confirm it by model checking.

7

Relating Both Modelling Experiments

Some results emerged from this experience between Promela/SPIN and CSP(LP)/CIA as tools to guide the first stages of modelling: 1. Both demanded almost the same level of knowledge and effort. 2. CSP(LP) is more declarative and hence allows shorter models to be written. 3. Although Promela allows asynchronous channels, CSP(LP) has extra expressiveness due to the logic programming extension (see for example the database implementation provided in appendix B). The concept of queue can be implemented allowing for asynchronous messaging in CSP(LP). 4. SPIN currently offers more support for building the model. 5. Channel handling demands more work in CSP or CSP(LP) specifications which also have the positive side-effect of forcing the user to have a more detailed knowledge about that important side of the system. 6. Trace extraction is currently easier with SPIN. 7. CSP(LP) allows to complement CSP with the use of logic programming features which extends considerably the flexibility of the specification language. An evidence of the importance of this can be seen on [ALBF03] where the flexibility of the input language was a key feature in allowing model checking of a Business Specification Language. In summary, both tools proved to be very useful in terms of building a simplified version of the system with a slight advantage of SPIN, of being a system developed over more than one decade. In consequence it offers better interface and more information to the system but on the other side there is no impediment for the CSP(LP)/CIA combination to evolve in the same direction.

8

StAC, a More Specific Business Modelling Language

StAC (Structured Activity Compensation) is a language that, in addition to CSP-like operators [Hoa85], offers a set of operators to handle the notion of compensation. In StAC it is possible to associate to an action a set of compensation actions providing a way to repair an undesired situation. Compensations are expressed as pairs with the form P ÷ Q, meaning that Q is the compensation planned in case that the effect of P needs to be compensated at a later stage. As the system evolves, compensations are remembered. If all the activities are √ successfully accomplished then the operator accept, 2 , releases the compensations. If any activity fails then the operator reverse, , orders the system to

22

J.C. Augusto et al.

apply all the recorded compensations for the current scope. In some contexts the failure to accomplish an activity can be so critical that demands the abortion of a process, that is the role of the early termination operator. Both compensation and termination operators can be bounded to a scope of application. Definition 1 Let A represent an activity, b a boolean condition, P and Q two generic processes, x a variable and X a set of values. Then, we can define as follows the set of well formed formulas in StAC: Process ::= A (activity label) | 0 (skip) |b→P | rec(P) (recursion) | P;Q | P ||Q (parallel) | ||x ∈ X.Px | P []Q (choice) | []x ∈ X.Px | (early termination) | {P } | P ÷ Q (compensation pair) | [P ] √ | (reverse) |2

(condition) (sequence) (generalised parallel) (generalised choice) (termination scoping) (compensation scoping) (accept)

In the example below, processes written in boldface are intended to be basic activities. Each StAC specification is coupled with a B machine [Abr96] describing the state of the system and its basic activities. Basically a B machine is composed by a declaration of sets, variables, invariants, initialisation and operations over those structures. Each StAC activity in a specification will have associated an operation in the corresponding B machine explaining how that activity is implemented in logical terms. We address the reader who wants a more in-detail account of StAC to [CGV+ 02] and [BF00]. 8.1

Travel Agency Example

The travel agency example presented in this section extends the previous travel agency example. In this version the user requests a collection of services instead of a single service, and the travel agency will then try to provide all the requested services. In the StAC model we associate a compensation activity to each service reservation, as the recovery mechanism if any reservation fails or the client decides to cancel his/her requests. A trip is arranged by getting an itinerary, followed by verifying the client’s credit card, and depending on whether the card is accept or rejected the reservation is continued or abandoned: Trip = GetItinerary; VerifyCreditCard; (accepted → ContinueReservation [] ¬accepted → clearItinerary) Getting an itinerary involves continually iterating over offering the client the choice of selecting from a car or a hotel “until” (, defined by using recursion [Fer03]) EndSelection is invoked. GetItinerary = (SelectCar [] SelectHotel) EndSelection


23

ContinueReservations starts by making the reservations on the client’s itinerary. If some of the reservations failed, the client is contacted; otherwise, the process ends. The car and hotel reservations are made concurrently. ContinueReservation = MakeReservations; (okReservations → EndTrip [] ¬okReservations → ContactClient) MakeReservations = CarReservations HotelReservations CarReservations = c ∈ CAR . CarReservation(c) HotelReservations = h ∈ HOT EL . HotelReservation(h) The CarReservation process reserves a single car using the Reserve activity. The travel agency uses two compensation tasks: compensation task S, representing compensation for reservations that have been booked successfully, and compensation task F , representing compensation for reservations that have failed. The choice between which task to add the compensation to is determined by the outcome of the ReserveCar activity. Since we use two compensation tasks, instead of having a compensation pair we have a compensation triple, with a primary process P and two compensations Q1 and Q2 . We model this triple with a construction of the form: P ; (c → (null ÷1 Q1 )) [] (¬c → (null ÷2 Q2 )) If P makes c true, this is equivalent to P ÷1 Q1 with Q1 being added to compensation task 1. If P makes c false, this is equivalent to P ÷2 Q2 with Q2 being added to compensation task 2. With this construction it is possible to organize the compensation information into several compensations tasks, where each one of those tasks can later be reversed or accepted independently. All the cars reservations are made concurrently. The car reservation and its compensations is defined as follows: CarReservation(f ) = ReserveCar(c); ((carIsReserved(c) → (null ÷S (CancelCar(c) RemoveCar(c)))) [] (¬carIsReserved(c) → (null ÷F RemoveCar(c))) The RemoveCar activity removes car c from the client’s itinerary, while the CancelCar activity cancels the reservation of car c with the car rental. If the activity ReserveCar is successful, then to compensate it one has to cancel the reservation with the car rental and also remove that car from the client’s itinerary. Otherwise, if the car reservation fails its only necessary to remove the car from the client’s itinerary in order to compensate, its not necessary to cancel the car reservation. The hotel reservations are defined similarly and are omitted here. The ContactClient process is called if some reservations failed. In this process the client is offered the choice between continuing or quitting: ContactClient = (Continue; F ; GetItinerary; ContinueReservations) [] (Quit; (S F )) In the case that the client decides to continue, reverse is invoked on compensation task F , the failed reservations. This has the effect of removing all failed

24

J.C. Augusto et al.

reservations from the clients itinerary. Compensation task S is preserved as the successful reservations may need to be compensated at a later stage. The client continues by adding more items to the itinerary, which are then reserved. In the case that the client decides to quit, reversal is invoked on both compensation threads. This has the effect of removing all reservations from the clients itinerary and cancelling all successful reservations. Finally, a successful trip reservation is S 2 F . ended by accepting both compensation tasks: EndT rip = 2 8.2

Executable Semantics

One benefit of using StAC is that it would not be possible to capture advanced aspects of the system with Promela (see [ABF03]) and CSP. Modelling with StAC will focus in the higher levels of the system. Any of the previously considered languages can be a good complement by using them to model some of the more low-level features of the system as the communication between processes. During modelling we have used an animator for StAC processes [LAB+ 01] based on the CSP(PL) animator described in [Leu01]. At the moment it supports step-by-step animation and backtracking of StAC processes, and it can also detect deadlocks. Animation has helped in the verification of the travel agency, just by comparing the animation execution traces with the expected behavior of the specification several error where found: 1. There is a potential infinite loop if any of the services requested by the client fails. In this case the client can then start choosing a new itinerary that may lead again to some of his/hers requested services to fail. 2. The use of two independent compensation threads for the successful and failed reservations uses a complex notation that is difficult to understand. All this is overcome by the animating the model, because the user can observe the evolution of the compensation threads. 3. The initial StAC model did not have the EndTrip process, but the animation showed that without EndTrip the compensation information would still be available after the client’s logout.

9

Future Work

The XTL model checker allows the user to model check a wide range of system specifications, (see for example [LM99] and [LM02]) the only requirement being that the specification is made by using high level Prolog predicates describing how the system makes transitions between its different states. In this section we describe some basic aspects of XTL and exemplify how to use it to model check StAC specifications. XTL has been implemented using XSB Prolog ( http://xsb.sourceforge.net/ ). Expressiveness and performance indicators are very encouraging for XTL in the sense that it has been able to model check case studies where other tools like SPIN failed and solved problems at similar performance levels. Some domains where XTL was applied successfully are CSP and


25

B [LAB+ 01], Petri nets [LM02] and StAC [ALBF03]. The second phase of this research involves model checking both models by contrasting them against behavioral properties expressed in a formal language, LTL (Linear Temporal Logic) for SPIN and CTL (Computational Tree Logic) for XTL. Some properties had been checked by using SPIN and the next step will be to check equivalent or closely related properties in XTL. The comparison also highlights that part of SPIN success derives from a nice interface which can be even profitable for non-experts in model checking. Some of these services are available in the animators for CSP(LP) and for StAC while the others can be relatively easy added.

10

Conclusions

We conducted an experiment of modelling a prototype by using different languages which have tool support available. We considered Promela/SPIN and CSP(LP)/CIA which share many features in common but also more specific modelling languages like StAC. We left outside the models several details, e.g. all the web based communication was replaced by synchronous channels, the relation sessions/logins is simplified to a userID, the communication between the travel agency was simplified to a request and a response when in reality it is a two steps dialog. The models can be expanded in any of those directions as needed. A quick summary of our experience follows. It is also worth mentioning we have also applied these methodologies to other e-business related case studies: order fulfillment, e-bookstore and mortgage broker. Benefits of animation/simulation include a) demonstrating flow of information through the system b) exploring interaction between components c) extraction of traces that could be used for generating test cases. In general, however, animations produced by these tools are not of sufficient visual quality to be useful in end users or customer demonstrations. Benefits of model-checking include a) easy discovery of concurrency flaws, e.g., deadlock b) in depth understanding of protocols (process/object interactions) c) discovery of invariants (database consistency constraints) By comparison, benefits of prototyping include a) more realistic user interfaces b) evolution of a class structure that, we believe, would closely approximates that of the actual implementation c) opportunity to gain knowledge of the actual implementation technologies. Still, our experiments show that rapid modelling is possible (one or two weeks to develop a model, about four times faster than prototyping). Mature notations and tools such as Promela/Spin provide better automated support for modelling, animation, and model-checking. However, the higher level constructs in CSP(LP) allow more faithful modelling of, for example, database tables. Tool support for this notation is sufficiently mature to provide useful insight, but further improvements would be welcomed. Finally, application specific notations such as StAC allow the most rapid modelling of all. Given that long-running transactions are likely to be the basis of future e-business systems, we believe that it is worthwhile further developing such notations and tools to support them.

26

J.C. Augusto et al.

References [ABF03]

Juan C. Augusto, Michael Butler, and Carla Ferreira. Using spin and step to verify stac specifications. In Proceedings of PSI’03, 5th International A.P.Ershov Conference on Perspectives of System Informatics (to be published), Novosibirsk (Russia), 2003. [Abr96] J. Abrial. The B-Book: Assigning Programs to Meanings. Cambridge University, 1996. [AFG+ 03] J. Augusto, Carla Ferreira, Andy Gravell, Michael Leuschel, and Karen M. Y. NG. Exploring different approaches to modelling in enterprise information systems. Technical report, Electronics and Computer Science Department, University of Southampton, 2003. Technical Report, http://www.ecs.soton.ac.uk/∼jca/rm.pdf. [ALBF03] Juan C. Augusto, Michael Leuschel, Michael Butler, and Carla Ferreira. Using the extensible model checker xtl to verify stac business specifications. In Pre-proceedings of 3rd Workshop on Automated Verification of Critical Systems (AVoCS 2003), Southampton (UK), pages 253–266, 2003. [Bec00] Kent Beck. Extreme Programming Explained. Addison-Wesley, 2000. [BF00] M. Butler and C. Ferreira. A process compensation language. In IFM’2000 – Integrated Formal Methods, volume 1945 of LNCS, pages 61–76. Springer Verlag, 2000. [CGV+ 02] M. Chessell, C. Griffin, D. Vines, M. Butler, C. Ferreira, and P. Henderson. Extending the concept of transaction compensation. IBM Journal of Systems and Development, 41(4):743–758, 2002. [Fer03] C. Ferreira. Precise modelling of business processes with compensation. PhD Thesis (submitted), Electronics and Computer Science Department, University of Southampton, 2003. [Hoa85] C. A. R. Hoare. Communicating Sequential Processes. Prentice-Hall, 1985. [Hol97] Gerard Holzmann. The spin model checker. IEEE Trans. on Software Engineering, 23(5):279–295, 1997. [LAB+ 01] M. Leuschel, L. Adhianto, M. Butler, C. Ferreira, and L. Mikhailov. Animation and model checking of csp and b using prolog technology. In Proceedings of the ACM Sigplan Workshop on Verification and Computational Logic, VCL’2001, pages 97–109, 2001. [Leu01] M. Leuschel. Design and implementation of the high-level specification language csp(lp) in prolog. In Proceedings of PADL’01, pages 14–28. editor I. V. Ramakrishnan, LNCS 1990, Springer Verlag, 2001. [LM99] M. Leuschel and T. Massart. Infinite state model checking by abstract interpretation and program specialisation. In Proceedings of Logic-Based Program Synthesis and Transformation (LOPSTR’99), pages 63–82. editor Annalisa Bossi, Venice, Italy, LNCS 1817, 1999. [LM02] Michael Leuschel and Thierry Massart. Logic programming and partial deduction for the verification of reactive systems: An experimental evaluation. In Proceedings of 2nd Workshop on Automated Verification of Critical Systems (AVOCS’02), Birmingham (UK), pages 143–150, 2002. [LR00] F. Leymann and D. Roller. Production Workflow: Concepts and Techniques. Prentice Hall PTR, 2000. [Ros99] A. W. Roscoe. The Theory and Practice of Concurrency. Prentice-Hall, 1999.


A

Fragment of Promela Model

/* channels for communication between processes */ chan ch_ta = [queue_length] of {bit, byte, bit}; chan ch_car00 = [queue_length] of {bit, byte}; ... chan ch_car00_2_ta = [queue_length] of {bit, bit}; ... /* databases */ byte cars00[resources_length]; byte cars01[resources_length]; byte rooms10[resources_length]; byte rooms11[resources_length]; DBrecord taDB[ta_records]; proctype user() { i=0; do /* repeat choices */ :: (i < logins) -> if /* choose a user ID in {1..3} */ :: loginID = 1 :: loginID = 2 :: loginID = 3 fi; i++; checkCreditCard(ccbit1, ccbit2); if :: correctCreditCard -> if :: ch_ta!0, loginID, 0 /* unbook a car */ :: ch_ta!1, loginID, 0 /* book a car */ :: ch_ta!0, loginID, 1 /* unbook a room */ :: ch_ta!1, loginID, 1 /* book a room */ fi :: else -> atomic{ printf("Incorrect credit card !!"); fi :: (i >= logins) -> break od } ... proctype ta() { end: do :: ch_ta?0, userid,0 -> ch_car00!0,userid; CUnbooking(0, 0) :: ch_ta?1, userid,0 -> ch_car00!1,userid; CBooking(0, 0) :: ch_ta?0, userid,0 -> ch_car01!0,userid; CUnbooking(0, 1) :: ch_ta?1, userid,0 -> ch_car01!1,userid; CBooking(0, 1)

27

28

J.C. Augusto et al.

... (idem for Hotels) od } init{run user(); run ta(); run car00(); run car01(); run hotel10(); run hotel11() }

B

Fragment of CSP(LP) Model

agent User(integer) : {tadb, h11db}; User(_logins) = if (_logins > 5) then STOP else ((CheckCreditCard(1, _logins)) [] (CheckCreditCard(2, _logins)) [] (CheckCreditCard(3, _logins)) ); ... agent TA:{ch_ta,ch_car00,ch_car01,ch_hotel10,ch_hotel11}; TA = ch_ta?0?_userID?0 -> (((ch_car00!0!_userID -> SKIP) [| {ch_car00} |] CarRental00) [] ((ch_car01!0!_userID -> SKIP) [| {ch_car01} |] CarRental01) ); TA = ch_ta?1?_userID?0 -> (((ch_car00!1!_userID -> SKIP) [| {ch_car00} |] CarRental00) [] ((ch_car01!1!_userID -> SKIP) [| {ch_car01} |] CarRental01) ); ... (idem for Hotels) -- Travel Agent database agent TADB(multiset) : {tadb}; TADB(nil) = tadb!empty -> TADB(nil); TADB(_State) = tadb?member._x: (_x in _State) -> TADB(_State); TADB(_State) = tadb?add?_x -> TADB(cons(_x,_State)); TADB(_State) = tadb?rem?_x: _x in _State -> TADB(rem(_State,_x)); TADB(_State) = tadb?nexists?_x: not(_x in _State) -> TADB(_State); agent MAIN : {}; MAIN = (TADB(nil) [| {tadb} |] (C00DB(nil) [| {c00db} |] (C01DB(nil) [| {c01db} |] (H10DB(nil) [| {h10db} |] (H11DB(nil) [| {h11db} |] User(1) )))));

Prediction of Consumer Preference through Bayesian Classification and Generating Profile Su-Jeong Ko Department of Computer Science University of Illinois at Urbana-Champaign 1304 West Springfield Ave. Urbana, Illinois 61801 U.S.A. [email protected]

Abstract. Collaborative filtering system overlooks the fact that most consumers do not rate a preference; because of this oversight the consumer-product matrix shows great sparsity. A memory-based filtering system has storage problems and hence proves inefficient when applied on a large scale where tens of thousands of consumers and thousands of products are represented in the matrix. Clustering consumer into groups based on the web documents they have retrieved/fetched allows accurate recommendations of new web documents through solving the problem of sparsity. A variety of algorithms have previously been reported in the literature and their promising performance has been evaluated empirically. We identify the shortcomings of current algorithms for clustering consumer and propose the use of Naïve Bayes classifier to classify consumer into group. To classify consumer into group, this paper uses the association word mining method with weighted word that reflects not only the preference rating of products but also information on them. The data expressed by the mined features are not expressed as a string of data, but as an association word vector. Then, collaborative consumer’s profile is generated based on the extracted features. Naïve Bayes classifier classifies consumer into group based on association words in collaborative consumer’s profile. As a result, the dimension of the consumer-product matrix is decreased. We evaluate our method on database of consumer ratings for special computer study and show that it significantly outperforms previously proposed methods.

1 Introduction As the Web and its related technologies developed, various kinds of information are broadcast through the Web. But the retrieval tools on the Web provide one with useless information. Hence it is time for consumers to consider how to get their target information efficiently. If the problem is that consumer is swamped by too much information, the solution seems to lie in developing better tools to filter the information so that only interesting, relevant information gets through to the consumer[6]. Many present filtering systems are based on building a consumer profile[12]. These systems attempt to extract patterns from the observed behavior of the consumer to predict which products would be selected or rejected. However, these systems all suffer from a “cold-start” problem. New consumers start off with nothing in their profile and must train a profile scratch. Collaborative filtering system M.A. Jeusfeld and Ó. Pastor (Eds.): ER 2003 Workshops, LNCS 2814, pp. 29–39, 2003. © Springer-Verlag Berlin Heidelberg 2003

30

S.-J. Ko

overlooks the fact that most consumers do not rate a preference; because of this oversight the consumer-product matrix shows great sparsity[7]. A memory-based filtering system has storage problems and hence proves inefficient when applied on a large scale where tens of thousands of consumers and thousands of products are represented in the matrix[16]. Clustering consumer into groups based on the web documents they have retrieved/fetched allows accurate recommendations of new web documents through solving the problem of sparsity[3,4]. A variety of algorithms have previously been reported in the literature and their promising performance has been evaluated empirically[13,17]. EM is an obvious method for grouping consumer, but does not work because it cannot be efficiently constructed to recognize the constraint that web documents two consumers like must be in the same class each time. Kmeans clustering is fast but ad hoc. Gibbs sampling works well and has the virtue of being easily extended to much more complex models, but is computationally expensive. We identify the shortcomings of current algorithms for clustering consumer and propose the use of Naïve Bayes classifier to classify consumer into group. To classify consumer into group, this paper uses the association word mining method with weighted word that reflects not only the preference rating of products but also information on them. The data expressed by the mined features are not expressed as a string of data, but as an association word vector. Then, collaborative consumer’s profile is generated based on the extracted features. Naïve Bayes classifier classifies consumer into group based on association words in collaborative consumer’s profile. As a result, the dimension of the consumer-product matrix is decreased. The proposed method is tested on a database of consumer evaluated web documents, and the test result demonstrates that the proposed method is more effective than previous methods as a matter of recommendation.

2 Expression of Document Features In this paper, we use a more effective feature extraction method applying association word mining[8] to express the features of the document as either a bag-of–words[14] or a bag-of-associated-words. The association word mining method, by using Apriori algorithm[1,2], represents a feature for document not as single words but as association-word-vectors. Since the feature extraction method using association word mining does not use the profile, it needs not update the profile, and it automatically generates noun phrases by using confidence and support at Apriori algorithm without calculating the probability for index. Besides, since this method is representing document as an association word set, it prevents consumers from being confused by word sense disambiguation, and thus, it has an advantage of representing a document in detail. However, because this feature extraction method is based on a word set of association words, it makes an error of judging different documents identically. This problem decreases the accuracy of document classification. In the case of Inserting a new document into database, this method has a problem that the database should be updated each time. This paper proposes a method of giving the weight to a word that belongs to association word by using TF•IDF[11]. TF•IDF is defined to be the weight of the words in the document. We select the word that has the largest TF•IDF in

Prediction of Consumer Preference

31

association word. Both the association word and the typical word are selected as features, and it solves the problem, which is caused by using only association words. The Apriori algorithm is used to mine associated data from the words extracted from morphological analysis. The association word mining algorithm, Apriori, is used to find the associative rules of products out of the set of transactions. The mined data, or the set of associated words from each document, are represented as an associationword-vector model. As a result, documents are represented in Table 1 in the form of an association-word-vector model. Table 1. An example of features extracted from Web document Web document

Features

document1

game&participation&popularity operation&selection&match game&rank&name user&access&event

document2

data&program&music figure&data&program game&explanation&provision game&utilization&technology

The words that belong to association word at Table 1 are weighted using by TF•IDF. First, feature selection using TF•IDF makes morphological analysis of the document to extract features of the document, and then extracts only nouns from its outcome. TF•IDF of all extracted nouns can be obtained through Equation (1)[15]. n (1) W nk = f nk • [ log 2 DF + 1 ] Where fnk is the relative frequency of word nk against all words within the document, and n is the number of study documents, and DF is the number of training documents where word nk appeared. It extracts only higher frequency words as features by aligning them from higher TF•IDF words to lower ones. If feature of test document(D) is {n1,n2,…,nk,..,nm}, it is compared with words that belong to association word in Table 1. As a result, the words that belong to association word are weighted by TF•IDF. The word with the highest weight is selected as the typical word of the association word. If the typical word of (data&program&music) in Table 2 is ‘data’, we represent it as (data&program&music). Equation (2) defines the features of document dj that is composed of p association words.

dj={ AWj1 , AWj2,…, AWjk, …, AWjp}

(2)

In Equation (2), each of {AWj1, AWj2, AWjk, AWjp} means association word that is extracted from document dj. For the best results in extracting the association words, the data must have a confidence of over 85 and a support of less than 20[8].

32

S.-J. Ko

3 Collaborative Consumer Profile The collaborative filtering system based on web documents recommends a document to consumers according to {customer-product} matrix. The consumer in collaborative filtering systems doesn’t rate preference on all documents. Therefore, the missing value is occurred in {consumer-product} matrix. The missing value causes the sparsity of {consumer-product} matrix. In this section, the collaborative consumer profile generation is mentioned to reduce the sparsity of {consumer-product} matrix caused by the missing value. 3.1 The Composition of {consumer-product} Matrix If we define m products which are composed of p feature vectors and a group of n consumers, consumer group is expressed as U={cui}(i=1,2,…,n), document group is expressed as I={dj}(j=1,2,…,m). We define the consumers in collaborative filtering database to be ‘collaborative consumer’. And R={rij}(i=1,2,…,n j=1,2,…,m) is a matrix of {consumer-product}. The element in matrix rij means consumer cui’s preference to document dj. Table. 2 is the matrix of {consumer-product} matrix in collaborative filtering system. Table 2. {consumer-product} matrix in collaborative filtering system d1

d2

d3

d4

…

dj

…

dm

cu1

r11

r12

r13

r14

…

r1j

…

r1m

cu2

r21

r22

r23

r24

…

r2j

…

r2m

cui

ri1

ri2

ri3

ri4

…

rij

…

rim

…

…

…

…

…

…

…

…

…

cun

rn1

rn2

rn3

rn4

…

rnj

…

rnm

Collaborative filtering system uses an information that consumer rates the preference for web pages. Preference levels are represented on a scale of 0~1.0 in increments of 0.2, a total of 6 degrees, only when the value is higher than 0.5 is the consumer classified as showing interest. The web documents used in this paper are computerrelated documents gleaned by an http down loader. The features of web documents are extracted by association word mining described in section 2. rij in Table 2 is defined as a Equation (3). Namely, the element of matrix rij is in one of 6 degrees or no evaluation case.

r

ij

∈{φ,0,0.2,0.4,0.6,0.8,1}(i = 1,2,..., n)( j = 1,2,..., m)

(3)

In the Equation (3) φ means that collaborative filtering consumer i doesn’t rate document j. Table 3 shows the consumer’s preference to web documents in collaborative filtering system. The features of documents are composed of association words produced by the method described in section 2. In Table 3, ‘?’ means the automatic preference rating is required.

Prediction of Consumer Preference

33

Table 3. Consumer’s preference rating on web documents expressed in features d1 cu1

0.2

dj …

dm

1

…

0.4 0.6

cu 2

?

…

0.8

…

cu3

0.4

…

0.6

…

?

cun

0.4

…

?

…

?

3.2 Generating Collaborative Consumer Profile The profile of collaborative filtering consumer cui is generated based on the document features. In case a collaborative consumer rates preference low, the weight of rated document is given low. In case a collaborative consumer rates preference high, the weight of rated document is given high. Therefore, the preference of association words expressed in features is indicated various values according to the weight. As a collaborative consumer cui defines the preference rating rij on the document dj, the weight of each association word that is extracted from document dj is weighted as rij. The weight of association word AWijk are defined as c_wijk. In case that a collaborative th consumer cui rates for document dj ,AWijk is the k association word from Equation (2). Equation (4) is the equation that defines the initial weight AWijk , which are structural elements, to generate the consumer cui’s profile. The initial weight of AWijk of association words, c_wijk, is defined as the initial preference, the elements of {consumer-product} matrix. The preference that consumer rates directly is the most correct and important data for automatic preference rating.

c wijk = Preference ( AWijk ) = r

ij

(consumer:cui,1≤j≤m,1≤k≤p)

(4)

Table 4 shows the detailed calculating way to get the initial weight c_wijk obtained by the definition of Equation (4). Each of document d1, dj, dm is rated the value of 0.2, 0.8, 1 by a collaborative consumer cui in Table 4.The italic word in association word is a typical word. In Table 4 based on Equation (4), the weights of {AWi11…},{AWij1…},{AWim1…} are defined to be 0.2, 0.8, and 1, respectively. Although AWij1 ,AWim2, AWi1p in Table 4 are the same association word their initial weights are different like 0.2, 0.8, and 1. It needs combining these different initial weights to generate a collaborative consumer profile. The weight of the same association word is multiplied after retrieving all association words. Table 5 shows the detailed weighting ways and examples based on Table 4. For example, the final weights of {AWi12, AWijk}, c_w’i12 and c_w’ijk , are c_wi12xc_wijk because they are the same. Equation (5) based on Table 5 is the equation to change a weight according to the frequency of the association word extracted from all documents rated by consumer, after giving initial weight to association word by Equation (4) as in Table 4. All association words extracted from documents, which is rated by collaborative consumer cui, is saved in database(AWDB). Then, an association word(AWijk) is the same as another association word(Awij’k’) after retrieving AWDB , c_wijk is multiplied by c_wij’k’. The final weight of association word AWijk is defined to be c_w’ijk. In Equation (5), j ≠ j ′ork ≠ k ′ means that the same association word like AW111=AW111 is excluded from computing.

34

S.-J. Ko

c_wijk′ =

∏

AWijk ,AW ij’k’ ∈AWDB

c_wijk⋅c_wij′k′|(AWijk= AWij′k′ )(1≤ j, j′ ≤ m,1≤ k,k′ ≤ p) j ≠ j′ork ≠ k′,user : cui (5)

Table 6 defines the structure of a collaborative consumer profile to be CUi based on Table 5 and Equation (5). By definition in Equation (5), the final weight c_w’ijk is given to association word AWijk. Table 4. Giving initial weight for profile generation Document

Initial weight c_wi11 (0.2)

d1 (preference ri1=0.2)

c_wi12 (0.2) … c_wi1k (0.2) … c_wi1p (0.2) c_wij1(0.8)

AWij1 utilization&technology&development

dj (preference rij=0.8)

c_wij2 (0.8) … c_wijk(0.8) … c_wijp(0.8)

AWij2 game&organization&selection&rank … AWijk interior&newest&technology& installation … AWijp organization&user&rank

c_wim1(1.0)

AWim1provision&illustation&explanation

c_wim2(1.0)

AWim2utilization&technology&development

dm (preference rim=1.0)

Association word AWi11 game&configuration&user&selection AWi12 interior&newest&technology& installation … AWi1k figure&popularity&service&music … AWi1p utilization&technology&development

…

…

c_wimk(1.0) … c_wimp (1.0)

AWimkdevelopment&rank&sports … AWimpfigure&data&service&engine

Table 5. The final weight given to association words Association Word

Weight to association word c_w’ij1 @

> @

orders

PROJECT

>@

> @

EMPLOYEE

works for

Fig. 2. Project management

Although the two associations look identical in their graphical representation, there is some substantial difference in the semantics of each association. Indeed, every employee works on one project at a time, but over time employees can work on several projects consecutively. In other words, the association "works for" is modifiable. The "orders" association however, is not modifiable: a project is ordered by one customer, but this customer remains the same over time. Consequently the diagram in Fig. 2 can be considered to be semantically incomplete: some relevant statements about the domain have not been expressed. Therefore, in MERODE, it is required to transform a class diagram into an existence dependency graph (EDG). In such graph, all object types are only related through associations that express existence dependency. According to the formal definitions in MERODE, a class D is existence dependent of a class M if and only if the life of each occurrence of class D is embedded in the life of one single and always the same occurrence of class M. D is called the dependent class and is existence dependent of M, called the master class. A more informal way of defining existence dependency is as follows: if each object of a class D always refers to minimum one, maximum one and always the same occurrence of class M, then D is existence dependent of M. Notice that existence dependency is equivalent to the notion of weak entity as defined by Chen [7][4]. To avoid confusion with a standard UML class diagram, MERODE uses a proprietary notation with dots and arrows to define cardinality of the existence dependency relationship. This cardinality defines how many occurrences of the dependent object type can be dependent of one master object at one point in time. As the cardinality of the master class is always exactly one (every dependent is associated to exactly one master), only the cardinality for the dependent needs to be specified. An arrowhead means that the master can have several dependents simultaneously whereas a straight line limits the maximum cardinality to one. A white dot means that having a dependent is optional for the master, whereas a black dot imposes a minimum constraint of one (the master has at least one dependent at any time). The transformation of the class diagram of Fig. 2 results in the EDG of Fig. 3. The "orders" association expresses existence dependency: each project can only exist within the context of a customer and refers to exactly one and always the same

Consistency by Construction: The Case of MERODE

109

customer for the whole duration of its life. A customer on the contrary can exist on its own. He needs not to have a project in order to exist (optionality indicated by the white dot) and he can have many ongoing projects (arrowhead). The "works for" relationship does not represent existence dependency. An employee can exist outside of the context of a project and a project can exist outside of the context of an employee. When an association does not express existence dependency, the association is turned into an object type that is existence dependent of all the object types participating in the association. In this case this means that the "works for" association is turned into an object type ASSIGNMENT, which is existence dependent of PROJECT and EMPLOYEE. MERODE calls this type of intermediate class a "contract" class: it models what can happen during the period of time that a project and an employee are related to each other. Since a project can have zero to many employees, each project has zero to many assignments (white dot, arrow). And as each employee is assigned to exactly one project at a time, each employee has exactly one assignment at a time (black dot, straight line). CUSTOMER

PROJECT

ASSIGNMENT

EMPLOYEE

Fig. 3. Existence dependency graph for the project management example

2.2

The Object-Event Table

In the case of object-oriented conceptual modeling, domain requirements will be formulated in terms of business or enterprise object types, associations between these object types and the behavior of business object types. The definition of desired object behavior is an essential part in the specification process. On the one hand, we have to consider the behavior of individual objects. This type of behavior will be specified as methods and statecharts for object classes. On the other hand, objects have to collaborate and interact. Typical techniques for modeling object interaction aspects are interaction diagrams or sequence charts, and collaboration diagrams. In most object-oriented approaches events are considered as subordinate to objects, because they only serve as a trigger for an object’s method. The object interactions themselves are modeled by means of sequence and/or collaboration diagrams. In contrast, MERODE follows an event-driven approach that raises events to the same level of importance as objects, and recognizes them as a fundamental part of the structure of experience [8]. A business event is now defined as an atomic unit of action that represents something that happens in the real world, such as the creation of a new customer, an order placement, etc. The business events reflect how domain objects come into existence (the creating events), how domain objects are modified (the modifying events), and how they disappear from the universe of discourse (the ending events). Object interaction can now be modeled by defining which objects are concurrently involved in which events. Object-event participations are denoted by means of an object-event table (OET). When an object participates in an event, it implements a method that defines the effect of the event on the object. On occurrence of the event all corresponding methods in the participating objects are executed in parallel. Thus, instead of modeling a complex sequence of method invocations, it is

110

M. Snoeck, C. Michiels, and G. Dedene

now assumed that all methods are concurrently executed. The OET for the project management example is given in Table 1. The rules that govern the construction of this table are described in the next section. 2.3 The Finite State Machines Finally, the life cycle of every enterprise object class is modeled by means of a finite state machine (FSM). The events of the object-event table are used as triggers for the transitions in the finite state machine. As an example, Fig. 4 shows the FSM for EMPLOYEE. Similarly, a FSM can be defined for the classes PROJECT, ASSIGNMENT and CUSTOMER. Table 1. Object-event table for project management. U H

FUBFXVWRPHU PRGBFXVWRPHU

W F MH R U 3

P R W V X &

& 0

P Q LJ V V $

P (

HQGBFXVWRPHU

(

FUBSURMHFW

0

&

PRGBSURMHFW

0

0

HQGBSURMHFW

0

(

FUBHPSOR\HH

&

PRGBHPSOR\HH

0

HQGBHPSOR\HH

PRGBHPSOR\HH

W Q H

H H \ OR S

(

DVVLJQ

0

0

0

&

UHPRYH

0

0

0

(

FUBHPSOR\HH H[LVWV

HQGBHPSOR\HH UHPRYH

DVVLJQ DVVLJQHG

Fig. 4. Finite state machine for Employee

3 Consistency by Construction The construction of the OET is governed by a number of rules that ensure the consistency of the OET with the EDG. An algorithmic approach to consistency checking would verify the consistency after entering the specification. In this section we illustrate how many of the consistency rules allow to automatically generate some parts of the requirements, preventing in this way inconsistencies and incompleteness. 3.1 Alphabet Rule The alphabet of an object class is defined as the set of all event types that are marked for this object type in the OET. The Alphabet Rule states that each event can have only one effect on objects of a class: the event either creates, modifies or deletes objects. In addition, the rule states that each object class needs at least one event to create occurrences and one event to destroy occurrences in this class.

Consistency by Construction: The Case of MERODE

111

Rather than verifying post factum whether there is at least one creating and one ending event for each enterprise object type, the case-tool will automatically generate two business events when an object type is added to the EDG. The default names are the name of the object type preceded by "cr_" and "end_", but as shown in Fig. 5, the user can overwrite the names and decide not to generate one or both event types. Simultaneously, the OET is completed accordingly: a column is added for the object type, two rows are added for the event types and the participations are marked (see Fig 6.).

Fig. 5. Existence dependency graph

Fig. 6. Object-event table

3.2 Propagation Rule and Type of Involvement Rule A second rule in the construction of the OET is the propagation rule. The propagation rule states that when an object type D is existence dependent of an object type M, the latter is by default also involved in all event types D is involved in. This means that if an involvement is marked for an event type in the column of a dependent object type D, it must also be marked in the column of the master object type M. In addition, the type of involvement rule states that since an existence dependent object type cannot start to exist before its master, a creating event type for a dependent class is a creating or a modifying event type for the master class. A modifying event type for a dependent class is also a modifying event type for its master class. And finally, since a dependent cannot outlive its master, an ending event type for a dependent is an ending or modifying event type for its master. To discern the participations the master acquired from its dependents through the propagation rule from the event type participations that are proprietary to the master class, the former are preceded by a ’A/’ (from Acquired) and the latter by an ’O/’ (from Owned). Performing and verifying the propagation by hand is a time consuming task, especially for larger projects. A case-tool however, can easily generate all the propagated participations. For the project management example, the resulting OET after entering the four object types and the existence dependency relations is shown in Fig. 7.

112

M. Snoeck, C. Michiels, and G. Dedene

The OET can be modified independently from the EDG, but also in this case, consistency is automatically enforced whenever possible. Adding an object type in the OET will add the object type in the EDG as well, although it will not be related to other object types already in the EDG. Events can be added in the OET and for these events we can add owned methods, which will be automatically propagated. Acquired methods cannot be added or removed. The type of involvement can be modified, provided it follows the type of involvement rule.

Fig. 7. OET with propagated object-event participations

3.3 Detection of Possible Redundant Paths Joining paths in the EDG occur when a master can be reached from a dependent by following two different existence dependency paths transitively from dependent to master. Assume that the project management example is extended with invoicing as in Fig. 8. During his/her assignment to a project, each employee can register the hours performed for the project. This time registration is included on an invoice at the end of the month as an invoice line. &86720(5

352-(&7

$66,*1 0(17

,192,&(

,192,&( /,1(

7,0( 5(*,675$ 7,21

(03/2 0.15). Moreover, the skewness of the distribution is close to one (0.9601, p-value: 0.0578), meaning more observations in the left tail than normal. The kurtosis is even significantly larger than one (2.6891, p-value: 0.0381), meaning that more observations are clustered around the mean, with fewer in the tails, than normal. Because of these indications of a nonnormal distribution, we used the less powerful Wilcoxon signed rank test for the difference in medians, which is a non-parametric alternative to the paired samples ttest. The result of the one-tailed test (see table 4) allows rejecting the null hypothesis (H0a), meaning that we empirically corroborated that the accuracy of comprehension of the REA diagrams is significantly higher than that of the ER diagrams. Table 4. 1-tailed Wilcoxon’s signed rank test for differences in median correctness ratio (REA diagram versus ER diagram; = 0.05)

Response variable Accuracy of comprehension

Size of the effect detected 0.0625

95% CI 0.0208 to +∞

Wilcoxon’s W-statistic 116.5

1-tailed pvalue 0.0055

Table 5 shows descriptive statistics of the comprehension time data for the ER and REA diagrams. As the differences between the paired samples comprehension time data show a near-normal distribution (skewness (-0.1689, p-value: 0.7209), kurtosis (0.8532, p-value: 0.3145), Shapiro-Wilk W statistic (0.9412, p-value: 0.2300), Kolmogorov-Smirnov statistic (0.5863, p-value > 0.15)) we applied the one-tailed paired samples t-test for the difference in means, which is robust against minor deviations from normality [26]. The significance level was again set at 5 %, i.e. = 0.05. The results of the test are shown in table 6. On the basis of these results we cannot accept hypothesis H1b, meaning that there is no significant difference in comprehension time between REA diagrams and ER diagrams. Only if we would agree to a lower confidence level, like = 0.10 (i.e. the probability that we reject H0b when H0b is false, is at least 90%), then a significant difference is found, meaning that we found some empirical evidence that REA diagrams require less time to understand than ER diagrams.

Conceptual Modeling of Accounting Information Systems

161

Table 5. Descriptive statistics comprehension time data

Comprehension time data Number of observations Mean Standard deviation 95% confidence interval of mean Median Interquartile range (upper/lower quartile) 95% confidence interval of median

ER diagrams 21 27.6 5.89 24.9 to 30.3 28.0 9.0

REA diagrams 21 24.0 7.04 20.8 to 27.2 26.0 10.0

24.0 to 33.0

18.00to 28.00

Table 6. 1-tailed paired samples t-test for differences in mean comprehension time (REA diagram versus ER diagram; = 0.05)

Response variable Comprehension time

Difference between means -3.6

95% CI

t-statistic

-∞ to 0.6

-1.49

1-tailed pvalue 0.0757

4.2 Validity Evaluation The response variables used in the experiment, i.e. accuracy of comprehension and comprehension time, are only pseudo-measures of user comprehension, which relates to the pragmatic quality of the diagrams. We agree with Briand et al. [26] that it is impossible to capture all dimensions of a concept in a single controlled experiment. Correctness ratios and timings relative to comprehension questionnaires are frequently used measurements of user comprehension in this type of research [26-30]. One might object that the time to answer the questionnaire is not a good measure of user comprehension if the student’s performance was bad (i.e. a low correctness ratio). This threat to construct validity can be investigated with a correlational analysis. The Pearson correlation coefficient expressing the degree of association between the paired comprehension time and correctness values (42 observations, normally distributed) is close to zero (Pearson’s r: -0.08, p-value: 0.6131), meaning that there is no correlation between the two types of measurement. We already described how potential confounding effects like learning, instrumentation, ceiling, and fatigue effects were alleviated through counterbalancing and other operational procedures. A ‘real’ threat to the internal validity of the study’s results is the order of learning effect. Students studied the ER model first, before they were introduced to the REA model. Towards the end of the course, most example conceptual models were REA diagrams and this increasing familiarity with the REA model might explain why the experiment, organized after the course ended, showed a better comprehension of the REA diagrams. Switching the order of learning does of course not help, as the REA accounting model is based on the ER approach. A solution is to organize the experiment before the REA model is explained to students. We believe, however, that in that case the experimental comparison can only focus on the improved readability of REA diagrams due to the naming and placement

162

G. Poels

conventions, as the subjects would not be familiar with the more essential REA model features such as event categories, structured relationships and ontological axioms. The use of students as experimental subjects, though common in empirical software engineering research [31-32], is of course a threat to the external validity of the study if the population that is studied is mainly composed of professionals. According to Kitchenham et al. [33] this is not a major issue as students are the next generation of professionals and are close to the population under study. This certainly holds for our experiment where the subjects were graduate business administration students. A more serious threat is the choice of experimental objects. The diagrams used were models of a single isolated transaction cycle, whereas in practice the REA enterprise information architecture aims at the development of an enterprise-wide or inter-enterprise accounting data model and information system. Many of the claimed benefits therefore relate to the integration of models and systems and this is something we couldn’t evaluate in the experiment. The conclusions drawn in this paper therefore relate to a limited use of the REA accounting model, more in particular to the instantiation of the REA template to develop a conceptual model of a single transaction-oriented business process.

5 Conclusions The results of our experiment show that the user comprehension of REA diagrams that are used as conceptual representations of economic exchanges, is higher than that of standard ER diagrams. Especially the accuracy of comprehension, measured as the correctness of answering comprehension questions about the business process as modeled, is significantly higher (at the 5% significance level). The hypothesis that the time required to understand a REA diagram is less than that of an equivalent ER diagram could only be supported at the 10% level of significance. The empirical evidence gathered through this experiment validates to some extent the claimed pragmatic quality of conceptual representations that are developed using the REA accounting model. In this sense they demonstrate the effectiveness of REA modeling in the AIS domain. We should however be careful in drawing definite conclusions about the validity of the REA model. Several threats to the internal and external validity of this study have been identified in the paper. In particular the choice of experimental objects (limited in scope to a single transaction cycle) and experimental tasks (relatively simple comprehension questions, not going into the depth of details of the REA enterprise domain ontology), and the timing of the experiment warrant caution. Moreover, our focus on pragmatic quality and the consequent necessity to use only semantic equivalent models in the experiment (which constrained the way we derived ER diagrams from REA diagrams) prevented us from evaluating alleged benefits related to the semantic quality assurance properties of the REA model.

Conceptual Modeling of Accounting Information Systems

163

References [1] [2] [3]

[4] [5] [6] [7] [8] [9] [10] [11] [12]

[13] [14] [15] [16] [17] [18] [19]

Olivé, A.: Specific Relationship Types in Conceptual Modeling: The Cases of Generic and with Common Participants. Unpublished keynote lecture, 4th Int’l Conf. Enterprise Information Systems (ICEIS'02), Ciudad Real, Spain, April 2002. Moody, D.L., Shanks, G.G.: Improving the quality of data models: empirical validation of a quality management framework. Information Systems, 28 (6), 2003, 619–650. Krogstie, J., Lindland, O.I., Sindre, G.: Towards a Deeper Understanding of Quality in Requirements Engineering. In: Lecture Notes in Computer Science, 932. Proc. 7th Int’l Conf. Advanced Information Systems Engineering (CAiSE'95), Jyvaskyla, Finland, June 1995, 82–95. Nelson, H.J., Monarchi, D.E., Nelson, K.M.: Ensuring the "Goodness" of a Conceptual Representation. In: Proc. 4th European Conf. Software Measurement and ICT Control (FESMA'01), Heidelberg, Germany, May 2001. Poels, G., Nelson, J., Genero, M., Piattini, M.: Quality in Conceptual Modeling – New Research Directions. In: Proc. ER’02 Workshops, 1st Int’l Workshop on Conceptual Modeling Quality (IWCMQ’02), Tampere, Finland, October 2002, 1–8. Wynekoop, J.L, Russo, N.L.: Studying System Development Methodologies: An Examination of Research Methods. Information Systems, 7 (1), 1997, 47–66. McCarthy, W.E.: An Entity-Relationship View of Accounting Models. The Accounting Review, 54 (4), 1979, 667–686. McCarthy, W.E.: The REA Accounting Model: A Generalized Framework for Accounting Systems in a Shared Data Environment. The Accounting Review, 57 (3), 1982, 554–578. Chen, P.P.: The Entity-Relationship Model – Towards a Unified View of Data. ACM Transactions on Database Systems, 1 (1), 1976, 9–36. Hollander, A.S., Denna, E.L., Cherrington, J.O.: Accounting, Information Technology and Business Solutions, 1st edition, 1996. Irwin, Chicago. th Romney, M.B., Steinbart, P.J.: Accounting Information Systems, 8 edition, 2000. Prentice-Hall. McCarthy, W.E.: Semantic Modeling in Accounting Education, Practice, and Research: Some Progress and Impediments. In: Chen, P.P., Akoka, J., Kangassalo, H., Thalheim, B. (Eds.): Conceptual Modeling: Current Issues and Future Directions. Springer Verlag, Berlin, 1999, 144–153. Dunn, C.L., McCarthy, W.E.: The REA Accounting Model: Intellectual Heritage and Prospects for Progress. J. Information Systems, 11 (Spring), 1997, 31–51. Vaassens, J.: Accounting Information Systems: A Managerial Approach, 2002. John Wiley & Sons. Geerts, G.L., McCarthy, W.E.: The Ontological Foundation of REA Enterprise Information Systems. Paper presented at the American Accounting Association Conference, Philadelphia, USA, August 2000. Geerts, G.L., McCarthy, W.E.: Augmented Intensional Reasoning in Knowledge-Based Accounting Systems. J. Information Systems, 15 (Fall), 2001, 127–150. Geerts, G.L., McCarthy, W.E.: An Ontological Analysis of the Primitives of the Extended-REA Enterprise Information Architecture. Int’l J. Accounting Information Systems. 3, 2002, 1–16. Lindland, O.I., Sindre, G., Sølvberg, A.: Understanding Quality in Conceptual Modeling. IEEE Software, 11 (2), 1994, 42–49. McCarthy, W.E.: Inter-Enterprise REA Models. Unpublished keynote lecture, 3rd European Conf. Accounting Information Systems (ECAIS’00), Munich, Germany, March 2000.

164

G. Poels

[20] Krogstie, J., Lindland, O.I., Sindre, G.: Defining quality aspects for conceptual models. In: Proc. IFIP8.1 Working Conf. Information Systems. Marburg, Germany, 1995, 216– 231. [21] Wohlin, C. et al.: Experimentation in Software Engineering: An Introduction. Kluwer Academic Publishers, 2000. [22] Zelkowitz, M.V., Wallace, D.: Experimental Validation in Software Engineering. Paper presented at the 1st Int’l Conf Empirical Assessment & Evaluation in Software Engineering (EASE’97), Keele, UK, March 1997. [23] Krogstie, J., Jørgensen, H.D.: Quality of Interactive Models. In: Proc. ER’02 Workshops, 1st Int’l Workshop on Conceptual Modeling Quality (IWCMQ’02), Tampere, Finland, October 2002, 115–126. [24] Moody, D.L., Sindre, G., Brasethvik, T., Sølvberg, A.: Evaluating the Quality of Process Models: Empirical Testing of a Quality Framework. In: Lecture Notes in Computer Science, 2503. Proc. 21st Int’l Conf Conceptual Modeling (ER’02), Tampere, Finland, October 2002, 380–396. [25] Genero, M., Poels, G., Piattini, M.: Defining and Validating Measures for Conceptual Data Model Quality. In: Lecture Notes in Computer Science, 2348. Proc. 14th Int’l Conf. Advanced Information Systems Engineering (CAiSE'02), Toronto, Canada, May 2002, 724–727. [26] Briand, L.C., Bunse, C., Daly, J.W.: A Controlled Experiment for Evaluating Quality Guidelines on the Maintainability of Object-Oriented Designs. IEEE Transactions on Software Engineering, 27 (6), 2001, 513–530. [27] Agarwal, R., De, P., Sinha, A.: Comprehending object and process models: An empirical study. IEEE Transactions on Software Engineering, 25 (4), 1999, 541–555. [28] Briand, L.C., Bunse, C., Daly, J., Differding, C.: An Experimental Comparison of the Maintainability of Object-Oriented and Structured Design Documents. Empirical Software Engineering, 2 (3), 1997, 291–312. [29] Harrison, R., Counsell, S., Nithi, R.: Experimental assessment of the effect of inheritance on the maintainability of object-oriented systems. J. Systems and Software, 52 (2–3), 2000, 173–179. [30] Danoch, R., Shoval, P., Balaban, M.: Hierarchical ER Diagrams (HERD) – The Method and Experimental Evaluation. In: Proc. ER’02 Workshops, 1st Int’l Workshop on Conceptual Modeling Quality (IWCMQ’02), Tampere, Finland, October 2002, 23–34. [31] Briand, L.C. et al.: Empirical Studies of Object-Oriented Artifacts, Methods, and Processes: State of the Art and Future Directions. Empirical Software Engineering, 4 (4), 1999, 387–404. [32] Deligiannis, I.S., Shepperd, M., Webster, S., Roumeliotis, M.: A Review of Experimental Investigations into Object-Oriented Technology. Empirical Software Engineering, 7, 2002, 193–231. [33] Kitchenham, B.A. et al.: Preliminary Guidelines for Empirical Research in Software Engineering. IEEE Transactions on Software Engineering, 28 (8), 2002, 721–734.

Preface to AOIS 2003

Information systems have become the backbone of all kinds of organizations today. In almost every sector – manufacturing, education, health care, government, and businesses large and small – information systems are relied upon for everyday work, communication, information gathering, and decision-making. Yet the inflexibilities in current technologies and methods have also resulted in poor performance, incompatibilities, and obstacles to change. As many organizations are reinventing themselves to meet the challenges of global competition and e-commerce, there is increasing pressure to develop and deploy new technologies that are flexible, robust, and responsive to rapid and unexpected change. Agent concepts hold great promise for responding to the new realities of information systems. They offer higher level abstractions and mechanisms which address issues such as knowledge representation and reasoning, communication, coordination, cooperation among heterogeneous and autonomous parties, perception, commitments, goals, beliefs, intentions, etc. all of which need conceptual modeling. On the one hand, the concrete implementation of these concepts can lead to advanced functionalities, e.g., in inference-based query answering, transaction control, adaptive work flows, brokering and integration of disparate information sources, and automated communication processes. On the other hand, their rich representational capabilities allow more faithful and flexible treatments of complex organizational processes, leading to more effective requirements analysis, and architectural/detailed design. The workshop will focus on how agent concepts and techniques will contribute to meeting information systems needs today and tomorrow. To foster greater communication and interaction between the Information Systems and Agents communities, we are organizing the workshop as a bi-conference event. It is intended to be a single “logical” event with two “physical” venues. It is hoped that this arrangement will encourage greater participation from, and more exchange between, both communities. The first part of the workshop was held on the 14th of July at AAMAS’03 – The 2nd International Joint Conference on Autonomous Agents and Multi-Agent Systems, in Melbourne (Australia), 14-18 July 2003. The second part of the workshop will be held in October at the 22nd International Conference on Conceptual Modeling ER03, in Chicago. We would like to gratefully acknowledge all the contributions to the workshop: those by the authors, the participants, and the reviewers. We believe that these accepted papers reflect the field’s state of the art very well. Furthermore, we anticipate that they constitute an excellent basis for an in-depth and fruitful exchange of thoughts and ideas on the various issues of agent-oriented information systems. October 2003

Paolo Giorgini Brian Henderson-Sellers

Bringing Multi-agent Systems into Human Organizations: Application to a Multi-agent Information System Emmanuel Adam and René Mandiau LAMIH UMR CNRS 8530, University of Valenciennes Le Mont Houy - France – 59313 Valenciennes Cedex 9 {emmanuel.adam,rene.mandiau}@univ-valenciennes.fr

Abstract. Agents are increasingly used to search for information on the Internet. However, they are principally used as single agents and not as part of a multi-agent system. Indeed, few projects use agents that communicate or collaborate with each other. This lack of communication often causes users to be isolated in front of their computer. We think that it is necessary for the user of an information search system (like an actor of a technological watch cell or a researcher in a laboratory for example) to be aware of what his/her colleagues are searching for (or at least to have knowledge of a part of their searches). This should avoid redundancies of information and work, and should (re) generate a team feeling among the actors. That is why we propose, in this article, an multiagent information system, which is composed itself of multi-agent systems located on the computer of each actor of a technological watch department, and which collaborate with each other and with the actors. This multi-agent architecture has been chosen in agreement with actors in a real department, following the analysis and modeling of their activities. The method of integrating a multi-agent system into a human organization is also discussed in this article.

1 Introduction The boom in Internet technology and companies networks has contributed to completely changing a good number of habits, which have been well established in companies for several decades. Enterprises are now launched in a race for information: being the first to find the correct information has become an essential objective for competitive enterprises. It is therefore important to own a fast tool for information search and distribution. Admittedly, tools have already been suggested such as: search engines, meta-engines, tools for automatic search (which search at determined regular intervals), and, more recently, information agents, capable of searching for information, sorting and filtering it. The problem with these solutions is that they do not take into account the human factors, such as the notion of the group or even the man-machine co-operation. We previously developed a method (AMOMCASYS, meaning the Adaptable Modeling Method for Complex Administrative Systems) to design and set up multi-agent systems within human organization, more precisely in the cooperative processes of M.A. Jeusfeld and Ó. Pastor (Eds.): ER 2003 Workshops, LNCS 2814, pp. 168–179, 2003. © Springer-Verlag Berlin Heidelberg 2003

Bringing Multi-agent Systems into Human Organizations

169

these organizations [1]. We have reused this method to develop multi-agent systems intended to facilitate cooperative information management within a technological watch team. The main advantage of our method, and of our system, is that it takes into account the cooperation between the actors of workflow processes. Indeed, we have noticed that most human organizations follow a holonic model (each part of the organization is stable, autonomous and cooperative and is composed of sub holonic organizations for which it is responsible) and we have built our method by integrating these notions. This paper describes firstly the AMOMCASYS method that we built to model human processes and to design multi-agent systems, which interact with the actors of the processes. Then, the method that we propose for the design of such multi agent systems is shown in two steps: a step for the individual characterization of the agents; and a step for the design of the agents’ cooperative working method. Finally, this article presents an application of our method to the design of multi-agent systems that help actors of a technological watch cell to search for and exchange information.

2 An Adaptable Modeling Method for Complex Administrative Systems (AMOMCASYS) 2.1 Holonic Principles Before designing a multi-agent system to assist information management within a human organization, we think that it is necessary to understand its working mechanisms. This should allow us to take into account all of its characteristics and to integrate multi-agent systems into it in a pertinent way. We have shown in [1] that most of complex administrative systems follow the holonic model that was proposed by Arthur Koestler in 1969 [2]. A series of grouped rules defines holonic systems, which are called Open Hierarchical Systems (OHS). Here we propose to give an interpretation of these rules according to the multi-agent point of view. We can retain the following principles from the rules: A holonic system possesses a tree structure. In fact it can be seen as a set of interwoven hierarchies. Each holon is responsible for a layer of other holons. – A holon is a stable, autonomous and cooperative part of the system and can be considered as a whole. Holons unite themselves to form a whole and each one can be broken down into holonic agents, this corresponds to the recursive breakdown of the problem into sub problems [3]. – A holon obeys precise principles, but is able to adopt different strategies according to its need. – The complex activities and behavior are situated at the top of the hierarchy; the interactions with the environment and the “simpler” reactive acts are located at the base of the holarchy. – The communications must follow the hierarchy and, according to the direction, must be filtered or detailed. The hierarchy is defined by the responsibility that a holon, which composes the system, has on a process or a sub-process and, so, on the holons that act within it. For

170

E. Adam and R. Mandiau

example, an organization by projects can be represented with a tree structure where each node (each holon) is responsible of a part of the project. As the holonic architecture is a well adapted or human organization where actors exchange information between themselves, we have proposed to reuse this architecture to design a multi-agent system that has to manage and exchange data. Here we can find at least two of the characteristics of the agents in a MAS sense: autonomy and cooperation. The third characteristic, the capacity to adapt itself to an environment is suggested by stability. A holon can therefore be seen as an agent whose stability is an essential point1. Our aim is the design of a multi-agent organization providing assistance to the actors in cooperative processes. This organization must be fixed (which does not imply rigidity) in order to be able to meet the user demands as quickly as possible. This is why we have used the social rules defined in the holonic concept in order to simplify and to accelerate the design of a multi-agent society (in the [4] sense). In such holonic multi-agent system, agents located at the top of the system are more cognitive than agents located at its base, which are more reactive. We think that this holonic concept is especially useable and useful in structured and cooperative fields [3]. Indeed, we think that systems composed of reactive agents, or systems that have a less rigid structure, are more dedicated to more flexible environments. However, before setting up a multi agent system, and more generally software, it is necessary to model the human organization in which it has to work. That is why we have proposed a method adapted to human organizations in which actors have different levels of responsibility and have to cooperate around documents. 2.2 Use of AMOMCASYS AMOMCASYS (meaning the Adaptable Modeling Method for Complex Administrative Systems) was designed for the modeling of the cooperative functioning of procedures in holonic human organization by integrating several software engineering methods after their comparisons [5]. Indeed, as our aim is to bring multi-agent system into human organizations, we wanted to imply their actors during the modeling step. So, we have built a benchmark to compare six families of method generally used in industry. As none of the compared methods totally fulfills our needs, we have integrated the most relevant parts of the methods to build the AMOMCASYS method. The integration allows, in a relatively short period of time, to build a method suited to its needs, in our case: to have a clear method, allowing explicit description of cooperation (communication, coordination, and collaboration) and the degrees of responsibility of the actors. AMOMCASYS is made up of four models, a data model, a data flow model, a processing model and a dynamic model: – Concerning the data model, the one proposed by UML makes it possible to represent the structure of the information (documents, …) and their relations 1

However, if the holons are stable, they do not have to be rigid. Indeed, the stability of the whole system is more important than the stability of each of its parts. So, it is sometimes necessary for some holons to be temporarily destabilized so that the whole system can take more long-term protection strategies.


171

(inheritance, …). Regarding the specification of MAS, this model makes it possible to define agents’ structure. – The dataflow model, which represents the information flows between actors, is based on the SADT dataflow model (IDEFO)) which we have adapted to represent the levels of responsibility of the actors and of the agents. – Although the activity model represents all possible flows between actors, it does not represent the conditions for which the information follows a particular path. That is why it is sometimes necessary to use a processing model. This model has to also represent cooperation between actors and their hierarchy and/or responsibility relationships. For this, we use the data processing model of the OSSAD method. Like the previous model, this model can be used for MAS design, the actors having different responsibility levels are replaced by agents having different responsibility levels too. These two models allow us to check if the specified organization follows holonic communication rules. However, although these models can be reused in the low-level design step, they are not sufficient to lead to the high-level one. Indeed, there is an considerable lack relating to the dynamics of the human organization being studied. – So, the AMOMCASYS method is composed of a dynamic model which uses parameterized Petri nets. This model implies the definition of three levels of rules of the process working method: global rules, local rules and personal rules. This model is not yet used in MAS design but only in the modeling of human organizations. This method, supported by a CASE tool (Visual Basic layer based on the commercial software VISIO), enabled us not only to reveal the key points of the procedures where the multi-agent systems should be brought in, but also to improve them in an organizational way2. Three steps are necessary to set up a MAS with AMOMCASYS: firstly, the processes in which the multi-agent has to be set up are modeled by using the data model and the dataflow model; secondly, the agents are introduced into the processes with the dataflow model and in cooperation with the process actors (fig. 1); and finally, the data exchanges and the working mechanism of the multi-agent system are modeled with the processing model. Figure 1 presents the integration of software agents in the information retrieval process of a technological watch team. Each agent is linked to an actor. Agents search for information, filter it, compare it and transmit it to actors that check it to record it in a database. The integration of agents in the process has been done in cooperation with the actors by using the dataflow model and corresponds to the second step of our method. That is why our MAS are designed in two steps: a step to design the roles that the agents play; and a step to design the cooperative working of the agents and their interactions with human actors of the process in which they are integrated.

2

For example, the time for dealing with one procedure involving about 15 actors was halved (at least 20 days of processing have been suppressed), by improving cooperation and increasing the responsibilities of the actors. The simplifications of the procedure have been proposed by its actors and are concretely applied in their department.

172


Fig. 1. Example of integration of software agents into a cooperative processes

3 Design of a MAS in a Human Organization Although the definition of our MAS structure has been facilitated by the use of holonic principles, the modeling of the system organization and the characterization of the functionality of the agents remain problematic. Indeed, the research published on this subject is mainly theoretical. Only a few recent research projects allow organization modeling with an application goal, but they are mainly devoted to the field of collective robotics [6] [7]. Here, we propose modeling and specification in two stages: the first stage concerns the individual functioning of each type of holonic agent; the second concerns the functioning of the group, describing communications between agents and between actors and agents. 3.1 Individual Design of the Holonic Agents In order to describe the general characteristics of various types of agent, we use a grid adapted from Ferber [8]. This grid gives a description in three dimensions, instead of the five dimensions initially suggested (cf. table 1). The physical dimension, which is too dependent upon the system, does not appear in the description of general characteristics. The relational dimension is attached to the social dimension. Concerning the organization functions, the conative and organizational functions, dealing with planning, have been grouped together; the conative function being more oriented towards needs, desires and urges which our holonic agents do not have, at least for the time being. This grid enables us to define the functions for each holonic agent relating to: knowledge (the representational function also describes the non-procedural knowledge); action planning (the organizational function); interactions; maintenance (the preservation function) and the actions specific to the role of the agent (the productive function). These functions are described in relation to the agent’s environment, the other agents and the agent itself.


173

This grid is applied to design the different types (different roles) of agents. However, it is interesting to note that a MAS can be considered as one single holonic agent. So it is possible to use this grid to define, at a higher abstraction level, the different functions of the multi-agent system. Table 1. Design grid adapted from Ferber’s analysis grid [8] Dimensions \ Function

Social

Environmental

Personal

Representation of the group (of Representation of the world Representation of itself, the other roles) of its capacities Planning of social actions, Planning of actions in the Planning control, metaOrganizational communications environment planning Description agent-society Perception and action Auto-communication Interactive interaction, performative mechanisms in relation to the Auto-action environment Management, coordination and Analysis, modification and Auto-modification, Productive negotiation tasks creation tasks learning Preservation of the society, the Preservation of resources, Self-preservation, repair, Preservation relations, the network of defense and maintenance of maintenance contacts territory

Representational

But, even though this grid enables us to have a clear view of the agents’ actions according to the environment and according to the other agents, it does not allow a definition of the functioning of the whole organization. Indeed, it does not allow the design of the cooperative functioning of the whole multi-agent organization. So, it is necessary to use a method, such as AMOMCASYS, which allows us to do this. 3.2 Cooperative Working of the Holonic Agents Regarding the design of the MAS to be integrated to the human process, AMOMCASYS data model allows us to represent the principal Holon class (which describes the general agent structure) as well as the classes associated to the knowledge concerning the environment (representation of the process, the actor, the workstation, the agent responsible, the subordinates). Each holonic agent has five main functions: to plan its action according to the process and its current state (corresponding to the organizational function); to receive and send to other holonic agents (corresponding to the interaction function); to act (corresponding to the productive function, to the specialty of the agent) and to manage the links between the agent responsible and the subordinates (corresponding to the preservation function). Of course each holonic agent has an implicit function: ’initialize’ (enabling it to acquire knowledge concerning the MAS). The four main functions (the organizational, interactive, productive and conservative functions) imply co-operations between holonic agents and sometimes between the agents and the actors (the users). The processing model of the AMOMCASYS method can model these co-operations, as we will see in the following case study.

174


4 Bringing an Information Multi-agent System into a Technological Watch Department The case study presented in this article has been performed in a technological watch department of a large company. In this application, we have designed a MAS in order to assist the actors of a technological watch department [9] in their tasks. This specification has been done following the analysis and the modeling of the department processes. In these processes, actors (called watchmen) have to collect information on the Internet, manage it and distribute it to their clients. So we had to design a MAS for information retrieval. 4.1 Structure of the Multi-agent Information System A multi-agent information system (IMAS) is generally composed of information agents that search, on the basis of requests that are sent to them (directly or indirectly through a database) for information on databases (local or distributed) or on Internet sites.Information agents’ activities are often coordinated through coordinator agents. These agents own knowledge on information agents (such as their addresses, their search domains for example) to which they send requests (in a targeted way if they own knowledge on their competences or by broadcast techniques). Coordinator agents have to gather collected information, in order to check it, compare it or filter it. Most information multi-agents systems are directly in touch with the user, upstream (to receive new requests) and/or downstream (to display search results). In order to have a reactive interface and distribute to the users, some IMAS propose the use of interface agents acting as interfaces between the users and the system. In our IMAS, called CIASCOTEWA (meaning CO-operative Information Agents’ System for the COoperative TEchnological WAtch), each agent proposed in the second step of our method (figure 1) is in fact a IMAS composed of agents. So, we associate an information agents system (a CIASTEWA, for CO-operative Information Agents’ System for the TEchnological WAtch) to each actor of the watch team. This allows the global system to have greater flexibility and allows each watchman to have more autonomy. Indeed, it is easier to integrate a new watchman by adding a sub-system to the global system than by reconfiguring a centralized system. A CIASTEWA is a sub holonic multi agent system, which has to search for information, collect it, sort it and communicate relevant information to other CIASTEWAs. The architecture of a CIASTEWA is shown in figure 2. Each of these sub systems is made up of: – a local database that contains the user requests, their results and information on the user, – an interface agent that assists the users to express their requests and allows them the interaction with the results provided by information agents, or with the other users of the group, – a coordinator agent that has the task of coordinating the actions of the other agents,


– – –

175

an agent responsible for information, which distributes the requests that are recorded in the local database to the request agents according to a search strategy, request agents that distribute the request for which they are responsible to search engine agents search engine agents that have to find information on the Internet.

So, each CIASCOTEWA helps the user, to whom it is dedicated, to search for relevant information and to communicate it with other actors. In order to maintain or create the feelings of community or group among the actors, which is often forgotten with the use of new technologies (the individuals are isolated with their workstation), we have proposed the development of self-organizing capacities in order to generate communities of CIASTEWA, which have to answer the same kinds of request. This reorganization is indicated to users in order to encourage them to cooperate, if they want to do it, with other users who have the same centers of interests. 4XHULHV &RRUGLQDWRU $JHQW

*URXS

5HVXOWV

4XHULHV ,QWHUIDFH $JHQW

8VHU

5HTXHVW $JHQW

6HDUFK HQJLQH $JHQW

6HDUFK HQJLQH $JHQW

5HVXOWV

,QIRUPDWLRQ 5HVSRQVLEOH $JHQW

5HTXHVW $JHQW

6HDUFK HQJLQH $JHQW

6HDUFK HQJLQH $JHQW

6HDUFK HQJLQH $JHQW

Fig. 2. Architecture of a CIASTEWA

In fact, works on the generation or identification of communities in IMAS have appeared recently, like in [10] where the agentification of a Web server requires the creation of agent communities. And, we think that “to find the right person” who may know where the answers are located is the best way of finding the correct information. Some work has been carried out in this direction by [11] and [12]. For example, in a large laboratory, it is frequent for researchers to have the same center of interest momentarily without knowing it. In our system, they are informed of this and so encouraged to exchange their information. The cooperation between several agents of a CIASTEWA is organized around a user database that contains data about the user, the user requests and the corresponding results. A CIASTEWA is also linked to a group database (fig. 2). This database collects requests of the group that are qualified as public by the users, and, temporarily, the results of these requests during the comparison step (fig. 1). This knowledge about the team allows coherence in the global MAS and allows a cooperative watch by the watchmen.

176


As an example of cooperation accorded by the system, we have the fact that each watchman knows what the others have collected; this avoids redundancy in the management and storing of information. Of course, the redundancies of stored information could be suppressed by the use of a centralized system (like a proxy server or a centralized agent) but these centralized systems are less flexible than a distributed system regarding the addition or suppression of an actor from the information retrieval system. Another example: when a user adds a new request, this should be compared to the others, if it is a subset of an existing one, the results are extracted by its results and the watchman is informed by its MAS that he has analog request to another actor (we suppose that this should make cooperation easier between watchmen having the same centers of interest). But, the cooperative functioning of the different agents of the CIASCOTEWA can only be specified after the definition of the different roles composing a CIASTEWA. The Individual design of the CIASTEWA agents are made by reusing design grid of the five roles that have to be designed in a CIASCOTEWA: the interface agent role ( ), the coordinator agent role ( ), the agent responsible for information role ( ), the request agent role ( ), and the search engine agent role ( ). 4.2 Cooperative Functioning of a CIASTEWA After having described the individual roles of agents that compose a CIASTEWA, we have to define the cooperative interactions between them. For that, we use the processing model of the AMOMCASYS method. For example, we have design the recording of a new request in a CIASTEWA. The user adds a request to its CIASTEWA through the interface agent. The interface agent asks the coordinator agent if the request exists in the group database, if it is a subset of another one or if it includes requests of other actors. In all these cases, a message is displayed to the user. The request is then recorded in the user database and a message is sent to the agent responsible for information. This message asks it to execute the requests not yet carried out. For this, the agent creates a request agent for each request. Each request agent creates a search engine agent for each search engine specified in the request. Each of these request agents connects to Internet in order to find results and send them to the agent responsible for it. When the request agent has received the results of each of its subordinates, it filters them (it deletes the doubles) and sends the results to the agent responsible for it. When the agent responsible for information has received a response from each of the request agents, it compares results with the group results, given by the coordinator agents, and writes in the result characteristics, which the actor has also received. Thanks to the AMOMCASYS method, we have defined other interactions between users and agents of the CIASCOTEWA such as the mailing, the annotation, the deletion of a result, the modifying and the deletion of a request. 4.3 Application of the CIASCOTEWA A prototype has been built from these specifications and is currently used within our laboratory.


177

Firstly, we have to define the structure of a CIASTEWA by creating the agents and by defining their links (their acquaintances). Then, as each CIASTEWA is composed of five kinds of agent, we have to implement five behaviors. Each behavior contains attributes relative to the knowledge of the agent to which it is associated and functions that the acquaintances of the agent can call. Thanks to the MAGIQUE platform [13], which is a java library that is dedicated to the design of hierarchical multi-agent systems, we have set up a multi-agent prototype. Indeed, MAGIQUE is dedicated to Hierarchic Multi-Agent Systems in which agents own competencies that they can learn or lose at run-time. By default, all the MAGIQUE agents contain a communication skill, which allow them to communicate with the other agents of their hierarchies. To define a CIASTEWA with MAGIQUE, we created five empty agents to which we added skills (sets of behaviors): SupSkill for the coordinator agent; AgentFrameSkill for the interface agent; LaunchSearchSkill for the agent responsible for information; SearchSkill for the request agents; OneEngineSearchSkill for the search engine agents. When an agent responsible for information creates request agents, it creates empty agents and informs them that it is their superior. Then it asks them to learn their competency (SearchSkill) and to perform their tasks. We have the same working when a request agent creates search engine agents. These search engine agents could create other sub agents if necessary (to interrogate different addresses of the same search engine for example) without having an impact on the working of the global system. Indeed, if the search engine agents return the required data, it is not important that it does it alone or with the help of sub agents. This principle is applicable for all the agents as it is specified by the holonic model. In our prototype, each CIASTEWA (fig. 3) allows an actor to: consult results of a request; modify, delete, add a request; send a result to other actors in its neighborhood; add notes to a result; delete a result. The user has the following information about a result: its address; its page name; its summary; its size; its date; its owner; the names of actors who own it (for that, the CIASTEWA communicates with the other CIASTEWAs in the group); the requests of the user that are linked to the result; and the search engines that have given the result. A Magique agent can communicate with others only if they are members of the same hierarchy. In order to make communication possible between CIASTEWAs, which are each linked to an actor in the technological watch cell, and thus to create a CIASCOTEWA, it was necessary to create a "super agent" which is the superior of all the coordinator agents. This agent has the BossSkill skill that contains no functions (the interactive function and the knowledge all the agents of the CIASCOTEWA system are implicit in a Magique agent). At the present time, the subgroups of CIASTEWA are created by the users through a configuration file that defines, amongst other things, the coordinator agent’s acquaintances of a CIASTEWA. Indeed, a set of xml files is necessary to initialize a CIASTEWA. Each of them corresponds to a knowledge: agent-config.xml corresponds to the personal knowledge; actors.xml, wastebasket.xml, search-engines.xml correspond to the environment knowledge; requests.xml and results.xml correspond to the user database.

178


Fig. 3. Screen copy of a CIASTEWA.

We have added some capacities to the BossSkill skill, which should allow the system to be able to self-organize itself according to the interest centers of the user, by using Kohonen self-organizing maps and interest distances, but we have not yet examined the results of this functionality. This reorganization of the system aims at decreasing number of communications between CIASTEWAs and increasing the team feeling amongst the users by informing them that they have interest centers in common (we think that this should increase cooperation between them).

5 Conclusion In order to specify a co-operative information agents’ system into a human organization, we have defined a method composed of three steps: a step of analysis and modeling of the human organization; a step of modeling the insertion of agent systems into the human organization; and a step of design of the multi-agent system. Our work uses the AMOMCASYS method that we have defined to analyze and model complex administrative system, and is backed up by the holonic model. Indeed, this model allows us to understand human organizations and we have shown (Adam, 2000) that this model allows the design of multi-agent systems particularly adapted to the human organizations being studied. Currently, the first prototype of the CIASCOTEWA system that we have proposed has been set up for short-term use (a few months), and has been particularly well accepted by actors, thanks to the participative design that we proposed. At this time, the CIASCOTEWA is being used in our laboratory in order to develop capacities of self-organization. A centralized CIASCOTEWA, accessible by


179

jsp pages is currently being set up in a knowledge management department of a large company. We aim at developing a more automatic design of the agent, by using our latest work, such as the deployment of a holonic multi-agent system in a network from a xml configuration file and by the use of the MAGIQUE platform.

References 1.

2. 3. 4. 5. 6. 7. 8. 9. 10.

11. 12.

13.

Adam, E., Modèle d'organisation multi-agent pour l'aide au travail coopératif dans les processus d'entreprise : application aux systèmes administratifs complexes (in French), PhD Thesis, Université de Valenciennes et du Hainaut-Cambrésis, (2000). Koestler, A. The Ghost in the Machine. Arkana Books, London, (1969). Gerber, C., Siekmann, J., Vierke, G. Holonic Multi-Agent Systems. Research report, RR99-03, March, DFKI GmbH, Germany, (1999). Mandiau, R., Le Strugeon E. & Agimont G. Study of the influence of organizational structure on the efficiency of a multi-agent system. Networking and Information Systems Journal, 2(2), (1999), 153–179. Adam, E., Kolski, C. Etude comparative de méthodes de génie logiciel utiles au développement de systèmes interactifs dans les processus administratifs complexes. Génie Logiciel, 49, (1999), 40–54. Collinot, A., Drogoul, A. Approche orientée agent pour la conception d'organisations: application à la robotique collective. Revue d'intelligence artificielle, 12 (1), (1998), 125– 147. Burckert, H.-J. and Fischer, K. and Vierke, G.Transportation Scheduling with Holonic MAS The TeleTruck Approach, Proceedings of the Third International Conference on Practical Applications of Intelligent Agents and Multiagents (PAAM'98), (1998). Ferber, J. Les systèmes multi-agents, Vers une intelligence collective. InterEditions, Paris, (1995). Jonnequin, L., Adam, E., Kolski, C., Mandiau, R. Co-operative Agents for a Co-operative Technological Watch, CADUI'02 – 4th International Conference on Computer-Aided Design of User Interfaces, University of Valenciennes, (2002). Helmy, T., Amamiya, S., Mine, T., Amamiya M.: An Agent-Oriented Personalized Web Searching System. In: Giorgini, P., Lespérance, Y., Wagner, G., Yu, E. (eds.): Proceedings of the Fourth International Bi-Conference Workshop on Agent-Oriented Information Systems (AOIS-2002 at AAMAS*02), Bologna, Italy, (2002). Jie, M., Karlapalem, K., Lochovsky, F.: A Multi-agent framework for expertise location. In Wagner, G., Lesperance, Y. and Yu, E. (eds.). Agent-Oriented Information Systems 2000, iCue Publishing, Berlin, (2000). Kanawati, R., Malek, M.: A Multiagent for collaborative Bookmarking. In: Giorgini, P., Lespérance, Y., Wagner, G., Yu, E. (eds.): Proceedings of the Fourth International BiConference Workshop on Agent-Oriented Information Systems (AOIS-2002 at AAMAS*02), Bologna, Italy, (2002). Routier, J.C., Mathieu P., Secq, Y., Dynamic Skills Learning: A Support to Agent Evolution, Proceedings of AISB'01, York, (2001), pp 25–32.

Reconciling Physical, Communicative, and Social/Institutional Domains in Agent Oriented Information Systems – A Unified Framework Maria Bergholtz, Prasad Jayaweera, Paul Johannesson, and Petia Wohed Department of Computer and Systems Sciences Stockholm University and Royal Institute of Technology Forum 100, SE-164 40 Kista, Sweden {maria,prasad,pajo,petia}@dsv.su.se

Abstract. One of a business system’s roles is to provide a representation of a Universe of Discourse, which reflects its structure and behaviour. An equally important function of the system is to support communication within an organisation by structuring and co-ordinating the actions performed by the organisations agents. These two roles of a business system may be represented in terms of business and process models, i.e. the separation of the declarative aspects from the procedural control flow aspects of the system. Although this separation of concerns has many advantages, the differences in representation techniques and focus of the two model types constitutes a problem in itself. The main contribution of this paper is a unified framework based on agent oriented concepts to facilitate the analysis and integration of business models and process models in e-Commerce in a systematic way. The approach suggested bridges the gap between the declarative and social/economic aspects of a business model and the procedural and communicative aspects of a process model. We illustrate how our approach can be used to facilitate integration, process specification, process pattern interpretation and process choreography.

1 Introduction Agent-oriented concepts have recently been applied to the area of information systems design. One of the most promising applications of agent-orientation could be in the development of e-Commerce systems. In e-Commerce, systems design is based on two fundamental types of models, business models and process models. A business model is concerned with value exchanges among business partners [10], while a process model focuses on operational and procedural aspects of business communication. Thus, a business model defines the what in an e-Commerce system, while a process model defines the how. This means that the process of designing eCommerce systems consists of two main phases. First, a business requirement capture phase focusing on value exchanges, and secondly, a phase focused on operational and procedural realisation. In the business requirement capture phase, coarse-grained views of business activities as well as their relationships and arrangements in business collaborations are represented by means of business model constructs at an abstract level. In contrast, M.A. Jeusfeld and Ó. Pastor (Eds.): ER 2003 Workshops, LNCS 2814, pp. 180–194, 2003. © Springer-Verlag Berlin Heidelberg 2003

Reconciling Physical, Communicative, and Social/Institutional Domains

181

the specification of a process model deals with more fine-grained views of business transactions, their relationships and choreography in business collaborations. Although the two phases in e-Commerce design, and their related models, have different focuses, there is clearly a need for integrating them. A unified framework covering coarse-grained business modelling views to fine-grained process specification views provides several benefits. It can be used for supporting different user views of the system being designed, and it can form the basis of a precise understanding of modelling views and their inter-relationships. It can also provide a basis for design guidelines that can assist in developing process models. The purpose of this paper is to propose a framework integrating the contents of business models and process models. The framework is based on agent-oriented concepts, like agent, commitment, event, action, etc., [18]. We use ebXML [8] and UMM [3] as the basis of our framework, more specifically the UMM Business Requirements View (BRV) for business models and the UMM Business Transaction View (BTV) for process models. UMM BRV already includes a number of agentoriented concepts, which we extend by adding a number of constructs for bridging business and process models, in particular speech acts. The work presented in this paper builds on [4], where Speech Act Theory (SAT)[17] and the language/action approach [6], are used for analysing processes, as well as for clarifying the relationships between agents in business and process models. The rest of the paper is organised as follows. Section 2 gives an overview of related research and introduces informally the basic concepts. Section 3 introduces the UMM BRV and BTV. Section 4 contains the main contribution of the paper and presents the integrated framework. Section 5 illustrates two applications of the introduced framework, and the analysis and design of business process patterns. Section 6 introduces rules for governing the choreography of transactions and collaborations. Section 7, finally, concludes the paper and discusses the results.

2 Basic Concepts and Related Research A starting point for understanding the relationships between business models and process models is the observation that a person can carry out several different actions by performing one single physical act. An everyday example could be a person who turns on the water sprinkler and thereby both waters the lawn and fulfils the promise to take care of the garden – one physical act (turning on the sprinkler), which can be viewed as “carrying” two other actions (watering the lawn and fulfilling a promise). Relationships like these are particularly common for communicative actions, which are carried out by means of physical actions. One way to look at the role of communicative actions and their relationships to other actions is to view human actions as taking place in three different domains: * *

The physical domain. In this domain, people carry out physical actions – they utter sounds, wave their hands, send electronic messages, etc. The communicative domain. In this domain, people express their intentions and feelings. They tell other people what they know, and they try to influence the behaviour of other actors by communicating with them. People perform such communicative actions by performing actions in the physical domain.

182

*

M. Bergholtz et al.

The social/institutional domain. In this domain, people change the social and institutional relationships among them. For example, people become married or they acquire possession of property. People change social and institutional relationships by performing actions in the communicative domain.

Using this division, business models can be seen as describing the social/institutional domain, in particular economic relationships and actions like ownership and resource transfers. Process models, on the other hand, describe the communicative domain, in particular how people establish and fulfil obligations. The three-fold division above is based on an agent-oriented approach to information systems design, [19], [20]. A key assumption of this approach is that an enterprise can be viewed as a set of co-operating agents that establish, modify, cancel and fulfil commitments and contracts [7]. In carrying out these activities, agents rely on so called speech acts, which are actions that change the universe of discourse when a speaker utters it and a recipient grasps it. A speech act may be oral as well as written, or even expressed via some other communication form such as sign language. The feasibility of speech act theory for electronic communication systems is supported by several researchers, see [16] for a review. The work reported on in this paper differs from these approaches since it uses SAT for analysing and integrating different modelling domains in e-Commerce, rather than facilitating electronic message handling per se. One of the pioneers in the development of a theory of speech acts is John Searle, [17], who introduced a taxonomy of five different kinds of speech acts: assertive, directive, commissive, expressive, and declarative, also called illocutionary points. An assertive is a speech act the purpose of which is to convey information about some state of affairs of the world from one agent, the speaker, to another, the hearer. A commissive is a speech act, the purpose of which is to commit the speaker to carry out some action or to bring about some state of affairs. A directive is a speech act, where the speaker requests the hearer to carry out some action or to bring about some state of affairs. A declarative is a speech act, where the speaker brings about some state of affairs by the mere performance of the speech act, e.g. “I declare you husband and wife”. Finally, an expressive is a speech act, the purpose of which is to express the speaker’s attitude to some state of affairs. In addition to its illocutionary point, a speech act also has a propositional content. The speech acts “I hereby pronounce you husband and wife” and “You are hereby divorced”, which are both declaratives, have different propositional contents. A speech act is often viewed as consisting of two parts, its propositional content and its illocutionary force. The illocutionary force is the illocutionary point together with the manner (for example ordering, asking, begging) in which the speech act is performed and the context in which it occurs.

3 UMM Business and Process Models – BRV and BTV The Resource-Event-Agent (REA) [15] framework has recently been applied in the UN/CEFACT Modelling Methodology (UMM) for business process modelling [3]. The scope of UMM is to provide a procedure for specifying, in a technology-neutral and implementation-independent manner business processes involving information


183

exchange. In UMM, a number of meta-models are defined to support an incremental model development and to provide different levels of specification granularity. • A business meta-model, called the Business Operations Map (BOM) partitions business processes into business areas and business categories. • A requirements meta-model, called the Business Requirements View (BRV) specifies business processes and business collaborations. • An analysis meta-model, called the Business Transaction View (BTV) captures the semantics of business information entities and their flow of exchange between business partners as they perform business activities. • A design meta-model, called the Business Service View (BSV) models the network components services and agents and their message (information) exchange. The two meta-models relevant for our work are BRV and BTV (see [Fig. 1]) and we describe them briefly in the following sub sections. 3.1

Business Requirements View

As it is based on REA, BRV models EconomicEvents, the Resources transferred through the EconomicEvents, and the Agents, here called Partners between whom the Economic Events are performed. An EconomicEvent is the transfer of control of a Resource from one Partner to another. Each EconomicEvent has a counterpart, i.e. the EconomicEvent that is performed in return and realising an exchange. For instance the counter part of a goods transfer economic event could be a payment, i.e. a transfer of money economic event. This connection between two economic events is modelled through the relationship duality. Furthermore, an EconomicEvent fulfils an Economic Commitment. An EconomicCommitment can be seen as the result of a commissive speech act and is intended to model an obligation for the performance of an Economic Event. The duality between EconomicEvents is inherited into the Economic Commitments, where it is represented by the relationship reciprocal. In order to represent collections of related commitments, the concept of Economic Contract is used. A Contract is an aggregation of two or more reciprocal Economic Commitments. An example of a Contract is a purchase order composed of one or more order lines, each one representing a corresponding EconomicCommitment in the contract. The product type specified in each line is the Resource Type that is the subject for the EconomicCommitment. EconomicContracts are often made within the boundaries of different Agreements. An Agreement is an arrangement between two Partners that specifies the conditions under which they will trade. 3.2

Business Transaction View

The Business Transaction View (BTV) specifies the flow of business information between business roles as they perform business activities. A BusinessTransaction is a unit of work through which information and signals are exchanged (in agreed format, sequence and time interval) between two business partners. These information exchange chunks, called BusinessActions, are either Requesting Business

184

M. Bergholtz et al.

Activities or Responding Business Activities (depending on whether they are performed by a Partner Role who is requesting a business service or whether they are the response to such a request). A transaction completes when all the interactions within it succeed, otherwise it is rolled back. Furthermore, the flow between different Business Transactions can be choreographed through BusinessCollaborationProtocols. > 1..* BusinessT ransactionActivity 0..* +transaction 1

> BusinessProcessActivityModel

1 > BusinessCollaboration1 * 1 1..* 1 1 > 1 BusinessCollaborationTask +realize 1..* forms +role +realization 1 governs > > BusinessCollaboration´ PartnerType 1 UseCase 2..* * +collaboration +participation1..*

1

participation

> BusinessCollaborationProtocolUseCase

1 * >* Agreement

* > resultsIn 1 > 0..1 EconomicEvent EconomicEffect * 1..* +te duality +ini fulfills rmi tiat nat or > or EconomicResourceT ype * reciprocal > 1 1 EconomicContract 1 Resource Flow specifies 1 classifies + * * 1 > establish Economic 2..* > resultsIn Commitment 1 EconomicResource *

1

> BusinessCollaborationProtocol 0..1 2..*+partner > BusinessPartner

> 1 BusinessTransaction 1

1

1 > Requesting BusinessActivity

1

> Responding BusinessActivity

0..1 1..* +role >+performedBy 1..* > AuthorizedRole 1..* +performs BusinessAction

>

Document Envelope

1 1 +source +target 0..1 1 1 1 ObjectFlow 1..2 State +type

>

Business Document >*+header InformationEntity 1 * +body +hadPart * +p 11 art Of > > UnstructuredDocument StructuredDocument

Fig. 1. UMM Business Requirement and Business Transaction Views

4 An Agent-Oriented Integration Framework In terms of the three domains introduced in Section 2, UMM explicitly addresses only the physical and the social/institutional domains. The physical domain is modelled through classes like BusinessTransaction and BusinessAction, while the social/institutional domain is modelled through EconomicCommitment, EconomicEvent, and other classes. The details of the communicative domain, however, are not explicitly modelled. This state of affairs causes two main problems. First, the relationship between the physical and the social/institutional domains is very coarsely modelled; essentially the UMM only states that a completed collaboration may influence objects in the social/institutional world, but it does not tell how the components of a collaboration affect the social/institutional objects. Secondly, there is no structured or systematic way of specifying how events in the physical domain influence the social/institutional domain. These problems can be overcome by introducing the communicative domain as an additional layer in the UMM, thereby creating a bridge between the physical and social/institutional domains.


185

As a preparation to modelling the communicative domain, a minor modification to UMM BRV is made, see [Fig. 2]. A class EconomicEffect is introduced as a superclass of EconomicCommitment, Agreement, and EconomicEvent. The power type [14] of EconomicEffect, called EconomicEffectType, is also added for the purpose of differentiating between the modelling of concrete, tangible objects in a domain, and the abstract characteristic categories of these objects. These modifications will allow for a more concise representation of the effects of communicative actions. In addition to these changes, the classes BusinessActionEnactment and BusinessTransactionEnactment are added. These represent the actual execution of a business action or business transaction, respectively. E conom icC om m itm ent

E conom icEvent

R ole

A greem ent

#baseC lass: string

2 * Econom icEffectT ype

1

E conom ic Effect

0..*

m easurem ent: string

+effectT ype: string

*

* * Pragm aticAction illocution: string action: string tim eT oP erform :

1 1

0..*

Pragm aticActionEnactm ent perform edT im e:

* * 1

* BusinessA ction

1

+isN onR epudiationR equired: +tim eToP erform :

0..*

BusinessActionEnactm ent perform edT im e:

1..2

1..*

1

1

BusinessT ransaction #baseC lass: A ctivityG raph

1..*

1

C ollaboration +beginsW hen: string +endsW hen: string

Fig. 2. Extended Business Requirement View

The basic notions introduced for modelling the communicative domain are those of a pragmatic action and its execution, i.e. PragmaticAction and PragmaticActionEnactment, see Fig. 2. A pragmatic action is a speech act as introduced in Section 2. It consists of three parts, denoted as a triple: Intuitively, these components of a pragmatic action mean the following: • EffectType specifies an EconomicEffectType, i.e. it tells what kind of object the pragmatic action may affect • Action is the type of action to be applied – create, change, or cancel • Illocution specifies the illocutionary force of the pragmatic action, i.e. it tells what intention the actor has to the Action on the EffectType Formally, Intention and Action are defined through enumeration: Action = {create, change, cancel, none} Illocution = {propose, accept, reject, declare, query, reply, assert} The meanings of the illocutions are as follows: propose – someone proposes to create, change, or cancel an object accept – someone accepts a previous proposal

186

M. Bergholtz et al.

reject – someone rejects a previous proposal declare – someone unilaterally creates, changes, or cancels an object query – someone asks for information reply – someone replies to a previous query assert – someone makes a statement about one or several objects For ‘query’, ‘reply’, and ‘assert’, there is no relevant Action involved, so only the “dummy” ‘none’ can be used. The class PragmaticActionEnactment is used to represent the actual executions of pragmatic actions. A PragmaticActionEnactment specifies a PragmaticAction as well as an EconomicEffect, i.e. the agreement, commitment, or economic event to be affected. Some examples of PragmaticActions are: “Query status of a sales order” would be modelled as “Request purchase order” would be modelled as <propose, create, purchaseOrder>, where ‘salesOrder’ and ‘purchaseOrder’ are EconomicEffectTypes 4.1

Integrated View of Process and Business Models

The glue between the physical domain and the communicative domain is made up by the associations between the classes BusinessAction and PragmaticAction, and BusinessActionEnactment and PragmaticActionEnactment. These associations express that a business action can carry one or more pragmatic actions, i.e. by performing a business action, an actor simultaneously performs one or several pragmatic actions. Often, only one pragmatic action is performed, but in some cases several can be performed, e.g. when creating a commitment and its contract at the same time. The global integrated view of BRV and BTV is shown graphically in [Fig. 3]. The original BTV-parts are grouped within the darker (lower) grey area boundary, BRVparts are grouped within the lighter grey area and the new parts introduced in this chapter are depicted in the white area.

5 Application/Analysis of Transaction and Collaboration Patterns In this section, a number of applications of the proposed framework with respect to business modelling patterns are introduced. A pattern is a description of a problem, its solution, when to apply the solution, and when and how to apply the solution in new contexts [12]. First, we discuss how the framework can be used for analysing the semantics of UMM business transaction patterns. Secondly, different collaboration patterns for incremental development are suggested. 5.1

Analysing UMM Business Transaction Patterns

UN/CEFACT has defined a number of business transaction patterns as part of UMM with the intention of providing an established semantics of frequently occurring business interactions. Below, we list a number of these patterns and show how they can be understood based on the framework introduced in the previous section.


187

participatetion

participatetion

typifies

> Agreement

> EconomicCommitmentType

typifies

> EconomicCommitment

> EconomicEventType

typifies

> EconomicEvent

> AgreementType

typifies

> EconomicEffectType

2..*

> EconomicEffect

resultsIn

2

> BusinessPartner

> PragmaticAction

> PragmaticActionEnactment

> BusinessAction

> BusinessActionEnactment

0..1 1..* +role > AuthorizedRole

+performedBy 1..* 1..* +performs

> RequestingBusinessActivity

> RespondingBusinessActivity

1

1

1

+transaction

1

> BusinessTransaction 1

0..*

> BusinessTransaction Activity

> BusinessCollaboration

Fig. 3. Integrated Global view

Design patterns are defined as “descriptions of communicating objects and classes customised to solve a general design problem in a particular context” [9]. We will adopt this definition to the UMM transaction patterns and view a transaction pattern as a template of exactly one pair of a Requesting and Responding Business Activity customised to encode the intentions and effects of a business interaction in a context . Definition: A transaction pattern (TP) is an activity diagram with two states designating the Requesting and Responding Business Activity. Every other state is an end state. All state transitions are labelled by pragmatic actions, carried by the Requesting and Responding Business Activity, see [Fig. 4]-[Fig. 5] and [Table 1] below. The analysis suggests one way to interpret the definitions of the UMM transaction patterns, but it does not make any claims to be the final, “correct” interpretation of these definitions. This is not an achievable goal as the definitions are only formulated in natural language, sometimes quite vaguely. The value of the analysis is that it provides explicit interpretations that can be judged for their validity, and thereby can help in formulating more precise and unambiguous definitions of the patterns. Another use of the analysis is to suggest additional patterns than those already present in UMM. The Fulfilment, ContractProposal,, Bilateral and Unilateral Cancellations (from [Table 1]) are obvious candidates for business transaction patterns.

188

M. Bergholtz et al. Table 1. Analysis of transaction patterns in terms of pragmatic actions

TP

Definition

Analysis

Commercial (Offer/ Accept)

“This design pattern is best used to model the ‘offer and acceptance’ business transaction process that results in a residual obligation between both parties to fulfil the terms of the contract. The pattern specifies an originating business activity sending a business document to a responding business activity that may return a business signal or business document as the last responding message.” [3]. “The query/response design pattern specifies a query for information that a responding partner already has e.g. against a fixed data set that resides in a database. The response comprises zero or more results each of which meets the constraining criterion in the query.” [3].

Request

“The request/confirm activity pattern shall be used for business contracts when an initiating partner requests confirmation about their status with respect to previously established contracts or with respect to a responding partner’s business rules.” [3].

Request

Query/ Response

Request/ Confirm

Request/ Response

Information Distribution

Notification

1

2

“The request/response activity pattern shall be used for business contracts when an initiating partner requests information that a responding partner alread6y has and when the request for business information requires a complex interdependent set of results [3]. “This pattern specifies the exchange of a requesting business document and the return of an acknowledgement of receipt signal. The pattern is used to model an informal information exchange business transaction that therefore has no nonrepudiation requirements.” [3]. “This pattern specifies the exchange of a requesting business document and the return of an acknowledgement of receipt signal. The pattern is used to model a formal information exchange business transaction that therefore has nonrepudiation requirements.” [3].

Response or

Request Response

Response Request Response 1 Request Response Carries no pragmatic action Request Response Carries no pragmatic action 2.

Note that the analysis fails to make a distinction between the query/response and the request/response patterns; the reason for this is that the difference between the patterns does not reside in different business effects but in different ways of computing the responses. The motivation for this analysis is that a notification results in a binding specification of business conditions for the initiating partner and, thus, in a (partial) agreement.


189

TP

Definition

Analysis

Fulfilment

The fulfilment pattern specifies the completion of an Economic Event [Fig. 4].

Contract Proposal

The Contract Proposal Transaction Pattern is a variation of the aforementioned Offer-Accept transaction pattern where the Partners does not have to make their assertions of intentions legally binding.[Fig.4] The Bilateral Cancellation transaction pattern refer to the bilateral cancellation of an Economic Contract or to Commitment(s) within an Economic Contract. See left part of [Fig. 5].

Request Response or Request

Bilateral Cancellatio n

Unilateral Cancellatio n

The Unilateral Cancellation transaction pattern refers to the unilateral cancellation of an Economic Contract or to Commitment(s) within an Economic Contract. See right part of [Fig. 5].

Response or Request Response or Request Response Carries no pragmatic action Contract Proposal Transaction Pattern

FulFillment Transaction Pattern Initiating Agent

Responding Agent

Response

>

Faliure

e, non act"> ntr ject, t rea t, c Even jec a, P where a denotes an event term (the triggering event), C denotes a logical formula (the mental state condition), a denotes an action term (the triggered action), and P denotes a logical formula (the mental effect or postcondition). 2.2 Example: Communicating Elevators pr

We consider an example scenario where two elevators operate in the same shaft and must take care to avoid collisions. For simplicity, we restrict our consideration to the case with three floors. Elevator A is serving floor 1 and 2, and elevator B is serving floor 2 and 3, and hence the critical zone, requiring coordination, is floor 2. This scenario is depicted in Fig.3.

208

G. Wagner and F. Tulba

B

A

Fig. 3. Two elevators operating in one shaft.

The external AOR diagram in Fig. 4 models this scenario. Shaft 1 2

Elevator reqPerm

HaltFloor[0..1] DownFloor[0..1] UpFloor[0..1]

reqTransp

ElevatorUser

TargetFloor move Direction move

grantPerm grantPerm

Direction arriveAt Floor halt

Fig. 4. An external AOR diagram modeling the elevator scenario from Fig. 3. There are two agent types (Elevator and ElevatorUser), one object type (Shaft), three message types (reqTransp, reqPerm, grantPerm), one non-action event type (arriveAt), two noncommunicative action event types (move, halt) and commitment/claim types for move and for grantPerm.

2.3 Modeling and Simulating Communication and Perception For modeling and simulating communication between agents, we do not consider nonverbal communication and abstract away from the physical layer, where the speaker realizes a communication act (or, synonymously, sends a message) by performing some physical action (such as making an utterance), and the listener has to perceive this action event, implying that, due to the physical signal transmission, there can be noise in the listener’s percepts referring to the message send event. In general, for each (external) event E and each agent, the simulation system has to compute a corresponding potential percept PP (according to physical laws), from which the

Agent-Oriented Modeling and Agent-Based Simulation

209

actual percept AP has to be derived according to the perceptive capability of the agent. The mental availability of the actual percept, then, is the (internal) perception event corresponding to the external event. There are two options, how to simplify the E—>PP—>AP chain: we can either assume that 1. E = PP = AP, so we don’t have to distinguish between an external event and the corresponding perception event; or 2. PP = AP, that is, all agents have perfect perception capabilities. For communication events, it makes sense to assume that E = PP = AP, i.e. the message received is equal to the corresponding message sent. Yet, there may be a delay between these two events, depending on the type of the message transport channel and the current physical state of the speaker and listener. For the perception of a non-communicative action event such an assumption may be not justified and would mean a severe simplification. However, the less severe simplification expressed by the assumption that PP = AP may be justified for many purposes.

3 Agent-Based Discrete Event Simulation In Discrete Event Simulation (DES) systems are modeled in terms of system states and discrete events, i.e. as discrete dynamic systems. Since a system is composed of entities, its state consists of the combination (Cartesian product) of all states of its entities. All state changes are brought about by events. DES is a very generally applicable and powerful approach, since many systems, in particular technical and social systems, can be viewed as discrete dynamic systems. In event-driven DES, the simulated time is advanced according to the occurrence time of the next event. In time-driven DES, the simulated time is advanced in regular time steps. In many ABS approaches, the basic DES model is refined in some way by making certain additional conceptual distinctions, including the fundamental distinction between interacting agents and passive objects. These simulation approaches may be classified as Agent-Based Discrete Event Simulation (ABDES). In our version of ABDES, extending and refining the basic DES model into a model of Agent-Object-Relationship Simulation (AORS), we start with time-driven DES (since we need small regular time steps for simulating the perception-reaction cycle of agents) and adopt a number of essential ontological distinctions from AORML: The enduring entities of a system (also called endurants in foundational ontrologies) are distinguished into agents and objects. Agents maintain beliefs (referring to the state of their environment) and process percepts (referring to events). Events can be either action events or non-action events. Action events can be either communicative (messages) or non-communicative.

210


SysTF: S,e

S’

e SysTF

S

Event OccurenceTime

SystemState S’

System 1

Time

*

*

* *

*

EntityState

Entity

Time

*

1

Fig. 5. The basic model of discrete event simulation as a UML class diagram. As a system consists of a number of entities, its state consists of all the states of these entities. A system transition function SysTF takes a system state S and a state-changing event e and determines the resulting successor state S’.

SysTF: S,e

S’

e SysTF

S SystemState S’

System 1

*

Time

*

1 ActionEvent

*

NonActionEvent

*

*

Message

TimeEvent

EntityState

Entity 0..1

Event OccurenceTime

1

*

Time

*

*

NonCommActEvt

sends 1

*

1 Object

1

receives

Agent creates

1

{xor}

* EntityBelief

Percept * *

Fig. 6. A UML class diagram describing the basic ontology of AORS: Agents maintain beliefs about entities, send/receive messages, and process percepts, which refer either to a noncommunicative action event or to a non-action event. Notice that communication (sending/receiving messages) is separated from perception (perceiving non-communicative action events and non-action events).

In addition to these conceptual distinctions from AORML, we need to introduce the notion of exogenous events, which drive the simulation and which are generated at random. Exogenous events are either non-action events that are not caused by other events or exogenous action events in the sense that their actor is not included in the simulation. Fig. 6 shows the AORS extension of the basic DES model of Fig. 5.


211

3.1 An Abstract Architecture and Execution Model for ABDES Systems In ABDES, it is natural to partition the simulation system into 1. 2.

the environment simulator responsible for managing the state of all external (or physical) objects and for the external/physical state of each agent; a number of agent simulators responsible for managing the internal (or mental) state of agents.

The state of an ABDES system consists of: the simulated time t the environment state representing − the environment (as a collection of objects) and − the external states of all agents (e.g., their physical state, their geographic position, etc.) the internal agent states (e.g., representing perceptions, beliefs, memory, goals, etc.). a (possibly empty) list of future events A simulation cycle consists of the following steps: 1. At the beginning of a new simulation cycle, say at simulated time t, the environment simulator determines the current events comprising a) all the events of the future events list whose occurrence time is now b) exogenous events whose occurrence time is now (e.g. stochastic non-action events or events created by actors that do not belong to the simulated system) 2. The environment simulator computes, on the basis of the current environment state and the current events, a) a new environment state, b) a set of successor events to be added to the future events list at different moments (representing events physically caused by the current events), c) for each agent, its perception events 3. Each agent simulator computes, on the basis of the current internal agent state and its current perceptions, a) the new internal agent state, b) a set of action events representing the actions performed by this agent (which are added to the future events list with time stamp t+1) 4. The future events list is updated by removing all the processed events and adding the computed action and successor events 5. The environment simulator sets the simulated time t to t+1, and starts over with step 1 of the simulation cycle. The simulation ends when the future events list is empty. This abstract architecture and execution model for ABDES systems can be instantiated by different concrete architectures and systems. In section 4, we present a Prolog program, which implements an AORS system and instantiates this architecture.

212


The AORML distinction between external and internal models provides the means needed for modeling both the environment and the agents involved in a simulation scenario. An external model describes the perspective of the environment simulator, whereas the internal models derived from the external one describe the perspectives of the involved agents. This suggests the following methodology for developing an AOR simulation model: 1. In the domain analysis of the simulation problem, develop an external AOR model of the scenario from the perspective of an external observer. This model is the basis both for designing the environment simulation and for deriving the specification of the involved agent simulators. 2. For each involved agent, transform the external AOR model of the simulation scenario into an internal AOR model for specifying the corresponding agent simulator. 3.2 Advantages of ABDES and AORS ABDES and AORS support structure-preserving modeling and closer-to-reality simulation: − Passive entities with certain properties are modeled as objects with corresponding attributes. − Interactive entities (actors) are modeled as agents, which have beliefs and perceptions , and interact with each other and with their environment. functionally distributed simulation where any of the participating simulators (the environment simulator and all involved agent simulators) may be deployed to different threads or processes, possibly running on different machines (realizing vertical distribution). interactive simulation where any of the involved agent simulators may be replaced by its real counterpart. modeling and simulating pro-active behavior, in addition to the basic reactive behavior.

4 A Prolog Prototype of an AORS System Implemented as a Prolog program, the AORS simulation cycle yields the following procedure: 1: 2: 3: 4: 5: 6: 7: 8:

cycle( _, _, _, []) :- !. cycle( Now, EnvSt, IntAgtSts, EvtList) :extractCrtEvts( Now, EvtList, CrtEnvEvts, CrtPercEvts), envSimulator( Now, CrtEnvEvts, EnvSt, NewEnvSt, TranslCausEvts), agtsSimulator( Now, CrtPercEvts, IntAgtSts, NewIntAgtSts, TranslActEvts), computeNewEvtList( EvtList, CrtEnvEvts, TranslCausEvts, TranslActEvts, NewEvtList), NextMoment is Now+1, cycle( NextMoment, NewEnvSt, NewIntAgtSts, NewEvtList).


213

Line 1 represents the exit condition (when the future events list is empty). In line 3, the current environment events (steps 1a and 1b of the simulation cycle) and also the current perception events are extracted from the future events list. Lines 4 and 5 simulate the system in the current cycle by first calling the environment simulator and then calling all agents simulators. In line 6, the future events list is updated (step 4). The last two lines update the time and start a new cycle (step 5). NewEnvSt and NewIntAgtSts stand for the new environment state and the new internal states of agents. We represent physical causality as a transition function, which takes an environment state and an event and provides a new environment state and a set of caused events. This function is specified as a set of reaction rules for the environment simulator in the form of rrEnv( RuleName, Now, Evt, Cond, CausEvt, Eff)

with obvious parameter meanings. Agent behavior, as a function from a mental state and a perception event to a new mental state and a set of action events, is also specified by a set of reaction rules: rr( AgentName, RuleName, OwnTime, Evt, Cond, ActEvt, Eff)

For processing these rules we use two meta-predicates: 1. prove( X, P) where X is a list of atomic propositions (representing an environment state or an internal agent state) and P is a proposition. 2. update( X, P, X’) where X’ is the new state resulting from updating X by assimilating P (in our simple example this means asserting/retracting atoms). When E is a current event, and there is an environment simulator rule, whose event term matches E such that prove( EnvSt, Cond) holds, then the specified CausEvt is added to the caused events list of step 2c) and the environment state is updated by performing update( EnvSt, Eff, NewEnvSt)

In a similar way, the reaction rules of each agent are applied, updating its internal state by update( IntAgtSt, Eff, NewIntAgtSt)

Concerning step 2c), notice that if there are only communication events (messages), then the perceptions of an agent are the messages sent to it. We now present the environment simulator: 1: 2:

envSimulator( Now, CrtEvts, EnvSt, NewEnvSt, TranslCausEvts) :findall( [CausEvt, Eff], ( member( Evt/_, CrtEvts), rrEnv( RuleName, Now, Evt, Cond, CausEvt,

Eff),

3: 4: 5: 6:

prove( EnvSt, Cond) ), ListOfResults), extractEffects( ListOfResults, Effects), computeNewEnvState( EnvSt, Effects, NewEnvSt), extractEvents( ListOfResults, CausEvts), translateCausEvts( Now, CausEvts, TranslCausEvts).

214


In line 2 all events (and their accompanying effects) that are caused by an event from the CrtEvts list are collected in ListOfResults. Based on the effects of the current environment events (extracted on line 3) the new environment state is determined (line 4). After extracting also the caused events from ListOfResults (in line 5), their absolute time stamp is computed with respect to the current moment (line 6). A similar procedure is performed for each agent: 1:

agtSimulator( AgtName, Now, CrtPercEvts, IntAgtSt, NewIntAgtSt, ActEvts) :2: timeFunction( AgtName, Now, OwnTime), 3: findall( [ActEvt, Eff], ( member( Evt, CrtPercEvents), rr( AgtName, RuleName, OwnTime, Evt, Cond, ActEvt, Eff), prove(IntAgtSt, Cond), ), ListOfResults), 4: extractEvents( ListOfResults, ActionEvents), 5: extractEffects( ListOfResults, Effects), 6: computeNewState( IntAgtSt, Effects, NewIntAgtSt).

5 Simulating the Communicating Elevators Scenario When making a simulation model, we have to draw a boundary around those entities we want to include in our simulation and those we want to exclude. In our simulation of the communicating elevators we choose not to include the shaft and the elevator users depicted in Fig. 4. This modeling decision turns the reqTransp messages of Fig. 4 into exogenous action events, which have to be generated at random (on the basis of some probability distribution). In AORS, a simulation model is expressed by means of 1. a model of the environment (obtained from the external AOR model of the scenario), consisting of a state structure model specifying all entity types, including exogenous event types a causality model, which is specified by means of reaction rules 2. a model for each involved agent (obtained from internalizing the external AOR model of the scenario into a suitable projection to the mental state of the agent under consideration), consisting of a state structure model a behavior model, which is specified by means of reaction rules 3. a specification of the initial states for the environment and for all agents The environment and agent models can be defined visually by means of AORML diagrams. Also the initial states can be defined by means of instance diagrams (similar to UML object diagrams). The encoding of a simulation model by means of a


215

high-level UML-based modeling language provides a platform-independent representation and allows to generate platform-specific code automatically. In the case of our Prolog simulation platform, we have to generate Prolog predicates from the AOR agent diagram shown in Fig. 6. We also have to generate the reaction rules for specifying causality and agent behavior in the format of the simulator. Please consult the web page http://tmitwww.tm.tue.nl/staff/gwagner/AORS

for obtaining further information about AORS and for downloading our Prolog AORS system.

6 Related Work Agent-Based Simulation is being used in various research areas, today. E.g., in

-

Biology, e.g. for investigating eco-systems or in population ethology (especially with respect to ants and other insects), see, e.g., [Klü01]; Engineering: for analyzing and designing complex (socio-) technical systems, such as Automatically Guided Vehicle Transport Systems [RW02];

-

Economics: e.g. in the simulation of auctions and markets (see Trading Agent Competition [TAC02]) and in the simulation of supply chains [LTFE03]; - Social Sciences: e.g. in [CD01] the phenomena of social monitoring and normbased social influence and in [Hal02] the cooperation in teams is studied. Some well-known platforms for Agent-Based Simulation are Swarm [MBLA96], SDML [MGWE98], Sesam [Klü01], MadKit [MadKit00] and CORMAS [Cormas01, Cormas00]. A particularly interesting class of simulation systems is formed by international technology competitions such as RoboCup [Robo98] and Trading Agent Competition (TAC) [TAC02]. Both RoboCup and TAC can be classified as interactive agent-based realtime simulation systems.

7 Conclusions We have presented a general approach to modeling and simulating scenarios of interacting systems as multiagent systems, based on the Agent-Object-Relationship (AOR) modeling language. Although there is a large body of work on agent-based simulation, our AORS approach appears to be the first general UML-based declarative approach to agent-based discrete event simulation. Our Prolog implementation of the AOR simulation system is still in an early prototype stage. In the future, we will transfer it to the Java platform.

216


References [Boo99] [Cormas00]

[CD01] [Dav02] [Dav02] [Den71] [EW02] [FG98]

[Hal02] [HLA02] [Jac94] [Klü01] [LTFE03] [MadKit00] [MBLA96] [MGWE98] [Robo98] [TAC02] [Wag03]

G. Booth: CourseWare Programmer’s Guide, Yale Institute for Biospheric Studies, 1999. C. Le Page, F. Bousquet, I. Bakam, A. Bah, C. Baron: CORMAS: A multiagent simulation toolkit to model natural and social dynamics at multiple scales. In Proceedings of Workshop "The ecology of scales", Wageningen (The Netherlands), 2000. Rosaria Conte and Frank Dignum. From Social Monitoring to Normative Influence. Journal of Artificial Societies and Social Simulation 4:2 (2001). Paul Davidsson. Agent Based Social Simulation: A Computer Science View. Journal of Artificial Societies and Social Simulation, 5:1 (2002). [http://jasss.soc.surrey.ac.uk/5/1/7.html] A. Davies: EcoSim: An Interactive Simulation, Duquesne Universität, Pittsburgh, 2002. D.C. Dennett: Intentional Systems. The Journal of Philosophy, 68 (1971). B. Edmonds, S. Wallis: Towards an Ideal Social Simulation Language, Manchester Metropolitan University, 2002. J. Ferber, O. Gutknecht: A meta-model for the analysis and design of organizations in multi-agent systems. Proceedings of the Third International Conference on Multi-Agent Systems (ICMAS´98), IEEE Computer Society Press, pp. 128–135, 1998. D. Hales: Evolving Specialisation, Altruism and Group-Level Optimisation Using Tags. Presented to the MABS'02 workshop at the AAMAS 2002 Conference. Springer-Verlag, LNCS, 2002. Defense Modelling and Simulation Office: High Level Architecture, 2002. I. Jacobson. The Object Advantage. Addison-Wesley, Workingham (England), 1994. F. Klügl: Multiagentensimulation, Addison-Wesley Verlag, 2001. O. Labarthe, E. Tranvouez, A. Ferrarini and B. Espinasse. A Heterogeneous Multi-Agent Modeling for Distributed Simulation of Supply Chains. Proc. of HOLOMAS 2003 Workshop. J. Ferber, O. Gutknecht, F. Michel: MadKit Development Guide, 2002. N. Minar, R. Burkhart, C. Langton, M. Askenazi: The Swarm Simulation System: A Toolkit For Building Multi-Agent Simulations, 1996. Moss, Scott , Helen Gaylard, Steve Wallis and Bruce Edmonds, SDML: A MultiAgent Language for Organizational Modelling, Computational and MathematicalOrganization Theory 4:1 (1998), 43–70. I. Noda, H. Matsubara, K. Hiraki, I. Frank: Soccer Server: a tool for research on multiagent systems, Applied Artificial Intelligence, 12:2-3 (1998). SICS: Trading Agent Competition 2002. See http://www.sics.se/tac/. G. Wagner. The Agent-Object-Relationship Meta-Model: Towards a Unified View of State and Behavior, Information Systems 28:5 (2003), 475–504.

REF: A Practical Agent-Based Requirement Engineering Framework Paolo Bresciani1 and Paolo Donzelli2 1

2

ITC-irst Via Sommarive 18, I-38050 Trento-Povo (Italy) [email protected] Department of Computer Science – University of Maryland College Park - MD (USA) [email protected]

Abstract. Requirements Engineering techniques, based on the fundamental notions of agency, i.e., Agent, Goal, and Intentional Dependency, have been recognized as having the potential to lead towards a more homogeneous and natural software engineering process, ranging from high-level organization needs to system deployment. However, the availability of simple representational tools for Requirements Engineering still remains a key factor to guarantee stakeholders involvement, facilitating their understanding and participation. This paper introduces REF, an agent-based Requirements Engineering Framework designed around the adoption of a simple, but effective, representational graphical notation. Nevertheless, a limited expressiveness of the graphical language may constrain the analysis process, reducing its flexibility and effectiveness. Some extensions are proposed to enhance REF capability to support requirements engineers in planning and implementing their analysis strategies, without affecting however REF clarity and intuitiveness.

1

Introduction

Agent- and goal-based Requirements Engineering (RE) approaches have the potential to fill the gap between RE and Software Engineering [5,4]. The concepts of Agent, Goal, and Intentional Dependency, in fact, applied to describe the social setting in which the system has to operate, lead towards a smooth and natural system development process, spanning from high-level organizational needs to system deployment [4]. Goals are valuable in identifying, organizing and justifying system requirements [14,2], whereas the notion of agent provides a quite flexible mechanism to model the stakeholders. However, the concrete application of such approaches has been until now limited only to few case studies. Several causes of this still immature adoption of agent- and goal-based paradigms for RE may be identified. Below we consider only two of them. First, although the notion of goal is central in some RE consolidated approaches like i* [15], GBRAM [1,2], and KAOS [8], an integrated and comprehensive requirements analysis methodology, clearly linked, or link-able, to the subsequent phases of software development, still is an open issue. At best of our knowledge, only the Tropos methodology [5,4] fully addresses this issue. Yet, not full consideration has been given by Tropos ´ Pastor (Eds.): ER 2003 Workshops, LNCS 2814, pp. 217–228, 2003. M.A. Jeusfeld and O. c Springer-Verlag Berlin Heidelberg 2003

218

P. Bresciani and P. Donzelli

itself to the design of a precise process for the RE phases (early requirements and late requirements), due to the wide set of aspects that have to be captured. Second, concerning the RE component of Tropos (or i*, to which Tropos RE is largely inspired), it is worth noticing that its considerably rich modeling framework, although promises to be capable of capturing several aspects relevant for the following phases, shows a certain level of complexity, so resulting understandable to only a strict group of practitioners. When the use of an i*-like modeling language has to be extended to non-technical stakeholders, it may be appropriate to give up with the full language expressiveness and modeling flexibility, in favor of a more straightforward and simple way to communicate with the stakeholders. In such a perspective, the paper introduces an agent- and goal-based RE Framework (called REF) previously applied to an extensive project for the definition of the requirements of a simulation environment [11]. Simple, yet reasonably expressive, REF allows non technical stakeholders to elicitate requirements, in collaboration with a requirements engineer which, at the same time, is provided with an effective methodology and process for requirements acquisition, analysis and refinement, and for communicating, in an easily intelligible way, the results of her analysis to the stakeholders. In the following, after a brief introduction to REF (Sections 2), a case study, is adopted (Sections 3) to critically revise the analysis process underlying the current methodology, to point out some of its current limits, and, finally, to suggest some notational and methodological extensions (Section 4). The tradeoff between the REF simplicity (and usability) and its expressiveness is carefully analyzed. Finally, observed advantages are discussed in the conclusive Section.

2

REF

REF is designed to provide the analysts and the stakeholders with a powerful tool to capture high-level organizational needs and to transform them into system requirements, while redesigning the organizational structure to better exploit the new system. The framework tackles the modeling effort by breaking the activity down into more intellectually manageable components, and by adopting a combination of different approaches, on the basis of a common conceptual notation. Agents are used to model the organization [9,11,16]. The organizational context is modeled as a network of interacting agents (any kind of active entity, e.g., teams, humans and machines, one of which is the target system), collaborating or conflicting in order to achieve both individual and organizational goals. Goals [9,11,8] are used to model agents relationships, and, eventually, to link organizational needs to system requirements. According to the nature of a goal, a distinction is made between hard-goals and softgoals. A goal is classified as hard when its achievement criterion is sharply defined. For example the goal “document be available” is a hard-goal, being easy to check whether or not it has been achieved (i.e., is the document available, or not?). For a soft-goal, instead, it is up to the goal originator, or to an agreement between the involved agents, to decide when the goal is considered to have been achieved. For example, the goal “document easily and promptly available” is a soft-goal, given that when we introduce concepts such as easy and prompt , different persons usually have different opinions.

REF: A Practical Agent-Based Requirement Engineering Framework

Soft Goal Modelling

Hard Goal Modelling

goal modeling phase

hard goals, constraints

mapping to organisation

hard goals

RE Framework

219

Development Flow

soft goals

Elicitation & Validation Flow

stakeholders & analysts

organization modeling phase

Organisation Modelling

start−up phase

Fig. 1. The Requirements Engineering Framework (REF)

REF, tackles the modeling effort by supporting three inter-related activities as listed below (see also Figure 1). The three modeling activities do not exist in isolation, rather they are different views of the same modeling effort, linked by a continuous flow of information, schematized as Development, and Elicitation & Validation flows. Organization Modeling, during which the organizational context is analyzed and the agents and their goals identified. Any agent may generate its own goals, may operate to achieve goals on the behalf of some other agents, may decide to collaborate with or delegate to other agents for a specific goal, and might clash on some other ones. The resulting goals will then be refined, through interaction with the involved agents, by hard- and soft-goal modeling. Hard-Goal Modeling seeks to determine how an agent can achieve a received hardgoal, by decomposing it into more elementary subordinate hard-goals, tasks1 , and resources2 . Supported by the REF graphical notation, the analyst and the agent will work together to understand and formalize how the agent thinks to achieve the goal, in terms of subordinate hard-goals and tasks that he or she will have to achieve and perform directly, or indirectly, by passing them to other agents. Soft-Goal Modeling aims at producing the operational definitions of the soft-goals, sufficient to capture and make explicit the semantics that are usually assigned implicitly by the involved agents [3,6,7]. Unlike for an hard-goal, for a soft-goal the achievement criterium is not, by definition, sharply defined, but implicit in the originator intentions. The analyst’s objective during soft-goal modeling is to make explicit 1 2

A task is a well-specified prescriptive activity. A resource is any concrete or information item necessary to perform tasks or achieve goals.

220


such intentions, in collaboration with the goal originator. However, depending on the issue at hand, and the corresponding role played by the two agents (i.e., the originator and the recipient) within the organization, also the recipient may be involved in the process, to reach a sharply defined achievement criterium upon which both of them can agree. Again, the analyst and the agents will cooperate through the support of the REF graphical notation. In the three modeling activities, REF uses a diagrammatic notations which immediately convey the dependencies among different agents and allow for a detailed analysis of the goals, upon which the agents depend. The adopted graphical notation is widely inspired by the i*framework [15] for RE [16] and business analysis and re-engineering [17], and thus open to be integrated in or extended by the Tropos methodology. An important aspect of REF is to adopt the i* notational ingredients at a basic and essential level, in favor of a higher usability and acceptability by the stakeholders. In the next Sections the notation and the methodology is briefly introduced by means of a case study. Mainly, Soft-Goal Modeling will be considered. The main aim during Soft-Goal Modeling is to iteratively refine each soft-goal in terms of subordinate elements, until only hard-goals, tasks, resources, and constraints are obtained (that is, until all the soft aspects have been dealt with) or each not refined soft-goal is passed on another agent, in the context of which will then be refined. Constraints may be associated with hard-goals, tasks, and resources to specify the corresponding quality attributes. Thus, the resulting set of constraints represents the final and operationalized views of the involved quality attributes, i.e., the quality models that formalize the attributes for the specific context [3, 6].

3 The Case Study We refer to an on-going project aiming at introducing an Electronic Record Management System (ERMS) within a government unit. The impact of such a system on the common practices of the communities and sub-communities of knowledge workers is quite relevant. A ERMS is a complex Information and Communication Technology (ICT) system which allows for efficient storage and retrieval of document-based unstructured information, by combining classical filing strategies (e.g., classification of documents on a multi-level directory, cross-reference between documents, etc.) with modern information retrieval techniques. Moreover, it usually provides mechanisms for facilitating routing and notification of information/document among the users, and supporting interoperability with similar (typically remote) systems, through e-mail and XML. Several factors (international benchmarking studies, citizens demand, shrink budgets, etc.) called for the decision of leveraging new technologies to transform the organization into a more creative, and knowledgeable environment. The initial organization model is shown in Figure 2. Circles represent agents, and dotted lines are used to bound the internal structure of complex agents; that is, agents containing other agents. In Figure 2, the complex agent Organization Unit corresponds to the organizational fragment into which it is planned to introduce the new ERMS, whereas the Head of Unit is the agent, acting within the Organizational Unit, responsible for achieving the required


legend Agent

resource

221

soft goal

hard goal

task

exploit ICT to increase performance while avoiding risks

cost/effective and quick solution

dependency link

Head of Unit

Organisational Unit

Fig. 2. Introducing the ERMS: the initial organization model

organizational improvement (modeled by the soft-goals exploit ICT to increase performance while avoiding risks, and cost/effective and quick solution). Goals, tasks, resources and agents (see also next Figures) are connected by dependency-links, represented by arrowhead lines. An agent is linked to a goal when it needs or wants that goal to be achieved; a goal is linked to an agent when it depends on that agent to be achieved. Similarly, an agent is linked to a task when it wants the task to be performed; a task is linked to an agent when the agent is committed at performing the task. Again, an agent is linked to a resource when it needs that resource; a resource is linked to an agent when the agent has to provide it. By combining dependency-links, we can establish dependencies among two or more agents. As mentioned, the soft-goals modeling process allow the analysts and the stakeholders to operationalize all the soft aspects implicitly included in the meaning of the soft-goal. Thus, for example, Figure 3 describes how the soft-goal exploit ICT to increase performance while avoiding risks is iteratively top-down decomposed to finally produce a set of tasks, hard-goals, and constraints that precisely defines the meaning of the soft-goal, i.e., the way to achieve it. Figure 3, in other terms, represents the strategy that the Head of Unit (as result of a personal choice or of a negotiation with the upper organizational level) will apply to achieve the assigned goal. Again, the arrowhead lines indicate dependency links. A softgoal depends on a sub-ordinate soft-goal, hard-goal, task, resource or constraint, when it requires that goal, task, resource or constraint to be achieved, performed, provided, or implemented in order to be achieved itself. These dependency links may be seen as a kind of top-down decomposition of the soft-goal. Soft-goals decompositions may be conjunctive (all the sub-components must be satisfied, to satisfy the original soft-goal), indicated by the label A on the dependency link, or disjunctive (it is sufficient that only one of the components is satisfied), indicated by the label O on the dependency link (see Figure 5).

222



Head of Unit

A

A A

increase personal performance

soft goal

hard goal

resource

task

increase productivity

A

A

avoid risks due to new technology A

A

A

easy document access

multi−channel access

O A dependency link

be more productive

A

increase process visibility

A

no filters from secretary

A

A

A

e−document as paper document

A

.....

guarantee smooth transaction

provide process performance

A PDA for reading documents

guarantee security

A

A A

constraint

(And / Or decomposition)

legend

reduce process constraints

A A

..... notify new documents by SMS

A

provide employee’s performance A A

weekly update

.....

A

daily update

number of documents waiting

mantain process structure

provide employee’s number of documents

Fig. 3. The “exploit ICT to increase performance while avoiding risks” Soft-Goal Model

According to Figure 3, the Head of Unit has to increase personal performance, to increase productivity of the whole unit, and also to avoid risks due to new technology. Let’s consider in details only the first sub-soft-goal, i.e., increase personal performance. It spawns two subordinate soft-goals, easy document access, for which the Head of Unit will require a multi-channel access system in order to be able to check and transfer the documents to the employees also when away from the office, and increase process visibility, to take better informed decisions. In particular, the soft-goal increase process visibility will eventually lead to the identification of some tasks (functionalities) the system will have to implement in order to collect and make available some data about the process (e.g., number of documents waiting) and about the employees (provide employee’s number of documents that have been assigned), and of some associated constraints, represented by a rounded-rectangle with a horizontal line, characterizing such data. In Figure 3, for example, they specify the frequency of update: daily for the process data and weekly for the employee’s ones.

4 Adding Special Links to Support the Analysis Process As described in Section 2, REF aims at providing a representational framework for requirements discovery and analysis, characterized by a sufficiently expressive graphical notation that, at the same time, be simple enough to be easily and quickly grasped by the stakeholders, even by those unfamiliar with RE. Indeed, these are very important aspects that, as demonstrated by several case studies [13,12,11], make REF applicable


223


Head of Unit

A

A A



A

A

avoid risks due to new technology A

A

A

easy document access

be more productive

A

multi−channel access



A

A

A

A

e−document as paper document

A

.....

A PDA for reading documents

..... notify new documents by SMS

guarantee security

A A

no filters from secretary

A

A

provide process performance provide employee’s performance

guarantee smooth transaction

A

..... A mantain process structure

A A

number of documents waiting

A

provide employee’s number of documents

twice a week update

Fig. 4. The “exploit ICT to increase performance while avoiding risks” Soft-Goal Model revised

..... Head of Unit

A


A A

provide employee’s performance

S

provide process performance

Fig. 5. A sharing between goals of the same agent

to real projects, and ensure a concrete involvement of the stakeholders, allowing for a quicker and more effective knowledge acquisition and requirements elicitation process. REF simplicity and effectiveness is mainly based on two key points: 1) the use of only one type of link (the dependency link); 2) the focus, during both hard- and soft-goal analysis, on only one goal (and the involved agents) at time; this leads to the drawing of very simple goal analysis diagrams, strictly generated by a top-down process. These two aspects make REF different from other approaches, in particular from i*. Indeed, we believe that these two characteristics allow for a very easy reading of the goal models; the second feature, in particular, allows the stakeholders (and the analysts, as well) to

224


concentrate the attention on one problem at time, and not being worried about the order in which different analyses of different sub-diagrams should be interleaved in order to obtain different possible diagrams: the goal diagram is always a tree, and it is always generate in the same shape, whatever node expansion sequence is followed. In the following, we analyze whether or not these two very important REF characteristics may represent a limit to some relevant aspects of the process of domain description. In particular, we tackle two possible cases in which the present version of REF tends to show some limits, describe them by means of our case study ERMS, and propose simple extensions to REF, to allow for a finer control during the process of model description and requirements elicitation. 4.1

Sharing Goals (Tasks, Constraints . . . )

Let us here analyze the fact that REF produces only trees, as goal diagrams. Thus, there is no the explicit possibility to deal with sub-goals (or constraints, or tasks, or resources) that may be shared by different upper-level goals. This situation may be further distinguished in at least three different sub-cases. First Case: top-down tree expansion and analysis induces at introducing different subgoals (or constraints, or tasks, or resources) for any different goal that is found during the goal analysis activity, even if different goals could be satisfied by the same sub-goal (or constraint, or task, or resource). For example, in Figure 3, two distinct constraints have been introduced for satisfying the two soft-goals provide process performance and provide employee’s performance, namely the constraints daily update and weekly update. Instead, for example, the Head of Unit could have accepted that the two softgoals, rather than requiring two different specialized constraints (as in Figure 3), would have shared the same constraint, e.g., a twice a week update (as in Figure 4). After all, according to REF, any sequence may have been followed in analyzing the two softgoals, and the two constraints may have been introduced in two very different moments, making it very difficult to spot that a common (although slightly different) constraint could have been adopted. This compromise, instead, could have been identified and judged as acceptable if considered by the analyst together with the Head of Unit at the proper moment during the design activity. The difference between Figure 3 and Figure 4 is minimal, regarding only leaf nodes, as highlighted by the dotted circle. Thus, Figure 4 can be obtained as a simple transformation of Figure 3. But let us consider a more complex hypothetical case, in which the two nodes collapsing in one are non-leaf nodes, with possibly deep trees expanding from them: relevant parts of the two sub trees, rooted in the two nodes, would have to be revised, in order to consider an alternative non-tree-based analysis. Thus, in this case, it would be strategic to be able to introduce a common child for the two different nodes before proceeding with the analysis of the nodes sub-trees. It is clear that, now, different diagram evolution strategies, and thus development sequences, may lead to quite different results or, even when producing the same result, this may be obtained with different degrees of efficiency. For example, a top-down bread-first diagram expansion could be probably preferred to a top-down depth-first strategy. In this way, it may appear appropriate to develop a shared sub-tree


225

only once, with two advantages: 1) at the design level, the analysis has not to be carried out twice; 2) at the implementation level, the complexity of the system to be implemented will be reduced, being two potentially different requirements, and all the derived artifacts – from architectural design down to implemented code – collapsed in one. Second Case: as a specialization of the first one, we can consider the case in which the similar sub-goal sharing happens among goals attached to the same agent already since its introduction in the organizational model, and not as a result of goal modeling. In this case, the REF methodology would lead the analyst to duplicate the sub-goal in two different diagrams, possibly with slightly different labels, although with the same semantics. Catching these cases as early as possible is very important in order to avoid duplicated analysis and assign higher priority and relevance to the analysis of the shared items. Third Case: a more intriguing situation may arise when the very same sub-goal can be shared among two different agents, as a consequence of two different and autonomous analyses of two different goals of the two agents (there is no room here to present the figures to illustrate an example in this case, but see [10]). Again, the analysis of such a shared soft-goal immediately assume a higher relevance and priority over the analysis of other goals. Its satisfaction is desired by two agents! For example, leading to the selection, among all the possible available tools on the market, of only one kind of mobile access channel able to satisfy both the agents. From the analysis of the previous three cases, clearly emerges the need of introducing in REF some mechanism to support the analysts during their refinement and navigation through the goals and sub-goals models. In particular, we propose to provide the analysts with a specific notation to be used to highlight situations where they believe that some commonalities could be hidden, i.e., that shared-goals could arise during the analysis. In other terms, to introduce in REF a notation suitable to act as a high-level reasoning support tool to enable the analysts to record their intuitions while building the goals models, by making notes to drive their strategies, e.g., to highlight where a top-down breath-first diagram expansion may be preferable to a top-down depth-first one. As such a notation, to denote situations in which a possible sharing could, we introduce what we call the S-connection, a link that does not have arrows, being the relationship perfectly symmetric, and that is marked by the label “S”, that stands for Sharing. Figure 5 shows a fragment of Figure 3 where the S-connection has been adopted. In particular, it shows how the S-connection could have been used during the analysis of the soft-goal to highlight in advance the possibility of sharing between the softgoals provide employee performance and provide process performance (the first example case previously analyzed). In the same way, in Figure 6 is depicted the use of the S-notation to highlight, within the soft-goal analyzed in Figure 3, a possible sharing between the soft-goal increase personal performance, that the Head of Unit wants to achieve, and the soft-goal be more productive, that the Head of Unit imposes, transfers to, the Employee (the third example case previously analyzed). It is worth noting how the S-notation is only a reasoning support mechanism that tend to disappear once the

226


Head of Unit


A

A


A

avoid risks due to new technology


A

S

A


be more productive

Fig. 6. A sharing between goals of different agents


Head of Unit be more productive guarantee security

PDA for reading documents

provide employee’s performance

.....

Archivist

H

cost/effective and quick solution

ERMS

Employee protect my privacy

IT easy to integrate

Organisational Unit apply public administration standard

Fig. 7. A conflict between goals of different agents

analysis proceeds. In other terms, the S-notation purpose is to mark possible sharing situations to drive the analysis (e.g., bread-first, multi-agents analysis, repeated back to back comparisons, and so on), but does not have any reason to exist any more once the goals have been exploded: the initial REF notation, with its simplicity, is sufficient for that regard.


4.2

227

Clashing Goals (Tasks, Constraints . . . )

Another common situation regards the possibility of efficiently dealing with clashing needs (goals, or constraints, tasks, and resources). As well as during a top-down analysis a top-down introduced sub-goal may be recognized as helpful for another goal (possibly of another agent), similarly, the introduced sub-goal may be recognized as (possibly) harmful for another goal. In addition, during the analysis, new agents may have to be introduced into the context (e.g., the Head of Unit requires the Employee to be more productive), and such new agents may express their own needs by introducing new goals that may very easily clash with other goals already in the context. Indeed, REF already provide the tool for the detailed recognition of such situations. In fact, when fully operationalized in terms of tasks and constraints, goals models can be adopted to detect clashing situations and to resolve them. Nevertheless, it is critical to foresee such possibly clashing situations as early as possible, even only at a very qualitative level. To enable the analysts to mark possible conflicting situations (and build their refinement strategy to deal with them), we introduce the H-connection (“H” for hurting). This is a powerful tool to detect possible conflicts and try to reconcile different stakeholders points of view, allowing to evolve the analyses only along the most promising alternatives. An example of application is given in Figure 7, where a Hconnection is used to highlight a possible conflict between two goals before proceeding in their analysis (i.e., the soft-goal provide employee’s performance is not broken down into tasks before taking into account the protect my privacy one – see also [10]).

5

Conclusions

The paper introduced an agent-oriented Requirements Engineering Framework (REF), explicitly designed to support the analysts in transforming high-level organizational needs into system requirements, while redesigning the organizational structure itself. The underlying concepts and the adopted notations make of REF a very effective and easy to deal with (usable) tool, able to tackle complex real case situations, while remaining simple enough to allow a concrete and effective stakeholders involvement. REF is strongly based upon i*, the modeling framework suggested by Eric Yu [15, 17,16]. However, it introduces some simplifications and tends to adopt a more pragmatic approach in order to obtain a greater and more active involvement of the stakeholders during the requirements discovery, elicitation and formalization process. However, we felt that REF could be improved with the regard to the support it provided to the analysts in dealing with more complex, and system/organizational design related issues, such as shared and clashing stakeholders needs. In both cases, an early detection of such a situation could lead to better analysis results: shared needs could be objects of a more intensive analysis effort to exploit commonalities to reduce complexity and increase re-usability; clashing needs could be solved at a very early stage, to focus then the analysis only towards the most promising alternatives. Two graphical notations (i.e., the S-connection and the H-connection) have therefore been introduced to allow the analysts to mark such situations and better reason about how to build their strategy, while performing them.

228


References 1. A. I. Antón. Goal-based requirements analysis. In Proceedings of the IEEE International Conference on Requirements Engineering (ICRE ’96), Colorado Springs, USA, Apr. 1996. 2. A. I. Antón and C. Potts. Requirements for evolving systems. In Proceedings of the International Conference on Software Engineering (ICSE ’98), Kyoto, Japan, Apr. 1998. 3. V. R. Basili, G. Caldiera, and H. D. Rombach. Encyclopedia of Software Engineering, chapter The Goal Question Metric Approach. Wiley&Sons Inc, 1994. 4. P. Bresciani, P. Giorgini, F. Giunchiglia, J. Mylopoulos, and A. Perini. TROPOS: An agentoriented software development methodology. Autonomous Agents and Multi-Agent Systems, 2003. in Press. 5. P. Bresciani, A. Perini, , F. Giunchiglia, P. Giorgini, and J. Mylopoulos. A Knowledge Level Software Engineering Methodology for Agent Oriented Programming. In Proceedings of the Fifth International Conference on Autonomous Agents, Montreal, Canada, May 2001. 6. G. Cantone and P. Donzelli. Production and maintenance of goal-oriented software measurement models. International Journal of Knowledge Engineering and Software Engineering, 10(5):605–626, 2000. 7. L. K. Chung, B. A. Nixon, E. Yu, and J. Mylopoulos. Non-Functional Requirements in Software Engineering. Kluwer Publishing, 2000. 8. A. Dardenne, A. van Lamsweerde, and S. Fickas. Goal-directed requirements acquisition. Science of Computer Programming, 20(1–2):3–50, 1993. 9. M. D’Inverno and M. Luck. Development and application of an agent based framework. In Proceedings of the First IEEE International Conference on Formal Engineering Methods, Hiroshima, Japan, 1997. 10. P. Donzelli and P. Bresciani. Goal-oriented requirements engineering: a case study in egovernment. In J. Eder and M. Missikoff, editors, Advanced Information Systems Engineering (CAiSE’03), number 2681 in LNCS, pages 605–620, Klagenfurt/Velden, Austria, June 2003. Springer-Verlag. 11. P. Donzelli and M. Moulding. Developments in application domain modelling for the verification and validation of synthetic environments: A formal requirements engineering framework. In Proceedings of the Spring 99 Simulation Interoperability Workshop, LNCS, Orlando, FL, 2000. Springer-Verlag. 12. P. Donzelli and R. Setola. Putting the customer at the center of the IT system – a case study. In Proceedings of the Euro-Web 2001 Conference – The Web in the Public Administration, Pisa, Italy, Dec. 2001. 13. P. Donzelli and R. Setola. Handling the knowledge acquired during the requirements engineering process. In Proceedings of the Fourteenth International Conference on Knowledge Engineering and Software Engineering (SEKE), 2002. 14. A. van Lamsweerde. Goal-oriented requirements engineering: A guided tour. In Proceedings of RE’01 – International Joint Conference on Requirements Engineering, pages 249–263, Toronto, aug 2001. IEEE. 15. E. Yu. Modeling Strategic Relationships for Process Reengineering. PhD thesis, University of Toronto, Department of Computer Science, University of Toronto, 1995. 16. E. Yu. Why agent-oriented requirements engineering. In Proceedings of 3rd Workshop on Requirements Engineering For Software Quality, Barcelona, Catalonia, June 1997. 17. E. Yu and J. Mylopoulos. Using goals, rules, and methods to support reasoning in business process reengineering. International Journal of Intelligent Systems in Accounting, Finance and Management, 1(5):1–13, Jan. 1996.

Patterns for Motivating an Agent-Based Approach Michael Weiss School of Computer Science, Carleton University, Ottawa, Canada [email protected]

Abstract. The advantages of the agent-based approach are still not widely recognized outside the agent research community. In this paper we use patterns as a way of motivating the use of agents. Patterns have proven to be an effective means for communicating design knowledge, describing not only solutions, but also documenting the context and motivation for applying these solutions. The agent community has already started to use patterns for describing best practices of agent design. However, these patterns tend to pre-suppose that the decision to follow an agent approach has already been made. Yet, as this author has experienced on many occasions, that is usually far from a given. There is a need for guidelines that summarize the key benefits of the agent approach, and serve as a context for more specific agent patterns. Our response to this need is a pattern language – a set of patterns that build on each other – that introduces the concepts of agent society, roles, common vocabulary, delegation, and mediation. We also argue that authors of agent patterns should aim to organize their patterns in the form of pattern languages, and present a template for pattern languages for agents.

1

Introduction

Agents are rapidly emerging as a new paradigm for developing software applications. They are being used in an increasing variety of applications, ranging from relatively small systems such as personal assistants to large and open mission-critical systems such as switches, electronic marketplaces, or health care information systems. There is no universally accepted definition of the notion of an agent. However, the following four properties are widely accepted to characterize agents: autonomy, social ability, reactivity and proactiveness [29]. Agents are autonomous computational entities (autonomy), which interact with their environment (reactivity) and other agents (social ability) in order to achieve their own goals (proactiveness). Agents typically represent different users, on whose behalf they act. Most interesting agent-based systems are thus collections of collaborating autonomous agents (typically referred to as multi-agent systems), each representing an independent locus of control. Multiple agents can, of course, be acting on behalf of the same user. Agents also provide an appropriate metaphor for conceptualizing certain applications, as the behavior of agents more closely reflects that of the users whose work they are delegated to perform or support. This reflects the fact that most complex software systems support the activities of a group of users, not individual users. Such agents are then treated as actors in a system that comprises human actors as well. M.A. Jeusfeld and Ó. Pastor (Eds.): ER 2003 Workshops, LNCS 2814, pp. 229–240, 2003. © Springer-Verlag Berlin Heidelberg 2003

230

M. Weiss

The following domain characteristics are commonly quoted as reasons for adopting agent technology: an inherent distribution of data, control, knowledge, or resources; the system can be naturally regarded as a society of autonomous collaborating entities; and legacy components must be made to interoperate with new applications [24]. However, the advantages of the agent-based approach are still not widely recognized outside the agent research community. While there are several papers discussing the differences between agents and objects [12, 13], on the one side, and agents and components [8], on the other, these papers do not provide actual guidelines for assessing whether a particular development project can benefit from using an agent-based approach. Patterns, on the other hand, are an effective way of guiding non-experts [11]. In this paper we use patterns as a way of motivating the use of agents. In the following sections we first summarize related work on agent patterns, then document a set of patterns that introduce key agent concepts. These serve as a conceptual framework and context for documenting more specific agent patterns (eg security patterns). We also make a case that agent pattern authors should organize their patterns in the form of pattern languages, and present a template for pattern languages for agents.

2

Related Work

Patterns are reusable solutions to recurring design problems, and provide a vocabulary for communicating these solutions to others. The documentation of a pattern goes beyond documenting a problem and its solution. It also describes the forces or design constraints that give rise to the proposed solution [1]. These are the undocumented and generally misunderstood features of a design. Forces can be thought of as pushing or pulling the problem towards different solutions. A good pattern balances the forces. There is by now a growing literature on the use of patterns to capture common design practices for agent systems [2, 12, 9, 16, 28]. The separate notion of an agent pattern can be justified by differences between the way agents and objects communicate, their level of autonomy, and social ability [7]. Agent patterns are documented in a similar manner as other software patterns, except for the structure of an agent pattern where we will make use of role models [20, 13]. The distinction between role models and collaboration diagrams is the level of abstraction: a collaboration diagram shows the interaction of instances, whereas a role model shows the interaction of roles. Aridor and Lange [2] describe a set of domain-independent patterns for the design of mobile agent systems. They classify mobile agent patterns into traveling, task, and interaction patterns. Kendall et al [12] use patterns to capture common building blocks for the architecture of agents. They integrate these patterns into the Layered Agent pattern, which serves as a starting point for a pattern language for agent systems based on the strong notion of agency. Schelfthout et al [21], on the other hand, document agent implementation patterns suitable for developing weak agents. Deugo et al [9] identify a set of patterns for agent coordination, which are, again, domain-independent. They classify agent patterns into architectural, communication, traveling, and coordination patterns. They also describe an initial set of global forces

Patterns for Motivating an Agent-Based Approach

231

that push and pull solutions for coordination. Kolp et al [15] document domainindependent organizational styles for multi-agent systems using the Tropos methodology. On the other hand, Kendall [13] reports on work on a domain-specific pattern catalog developed at BT Exact. Several of these patterns are described in the ZEUS Agent Building Kit documentation [5] using role models. Shu and Norrie [22], and Weiss [27] have also documented domain-specific patterns, respectively for agent-based manufacturing, and electronic commerce. However, unlike most other authors, their patterns are organized in the form of a pattern language. This means that the patterns are connected to each other in such a way that they guide a developer through the process of designing a system. Lind [16], and Mouratidis et al [18] suggest that we can benefit from integrating patterns with a development process, while Tahara et al [25], and Weiss [28] propose pattern-driven development processes. Tahara et al [25] propose a development method based on agent patterns, and distinguish between macro and micro architecture patterns. Weiss [27] documents a process for mining for, and applying agent patterns. Lind [16] suggests a view-based categorization scheme for patterns based on the MASSIVE methodology. Mouratidis et al [18] document a pattern language for secure agent systems that uses the modeling concepts of the Tropos methodology. As the overview of related work has shown, most agent patterns are documented in the form of pattern catalogs. Usually, the patterns are loosely related, but there is a lack of cohesion between them. Such collections of patterns provide point solutions to particular problems, but do not guide the developer through the process of designing a system using those patterns. This can only be achieved by a pattern language. We argue that agent pattern authors have to put more emphasis on organizing their patterns in the form of pattern languages for them to become truly useful. We therefore next suggest a template for pattern languages. A secondary goal of our pattern language for motivating the use of agents is to illustrate the use of that template.

3

Template for Pattern Languages

Patterns are not used in isolation. Although individual patterns are useful at solving specific design problems, we can benefit further from positioning them among one another to form a pattern language. Each pattern occupies a position in a network of related patterns, in which each pattern contributes to the completion of patterns “preceding” it in the network, and is completed by patterns “succeeding” it. A pattern language guides developers through the process of generating a system. Beck and Johnson [3] describe this generative quality of patterns: “Describing an architecture with patterns is like the process of cell division and specialization that drives growth in biological organisms. The design starts as a fuzzy cloud representing the system to be realized. As patterns are applied to the cloud, parts of it come into focus. When no more patterns are applicable, the design is finished.” Unlike a pattern catalog that classifies patterns into categories, the goal of a good pattern language is, foremost, to create cohesion among the patterns. We want the patterns to be closely related to each other. References to patterns should therefore largely be to other patterns in the same pattern language; and the patterns should be

232

M. Weiss

organized from higher-level to lower-level patterns in a refinement process. We can also expect a pattern language to have a reasonable degree of coverage of its application domain. We want to be able to generate most of the possible designs. Finally, the goal of a pattern language is to make the links between the patterns easy to use and understandable. This we refer to as the navigability of a pattern language. A pattern language should contain: a roadmap, a set of global forces, references to other patterns in the language in the context section of each pattern, and a resulting context section in each pattern. The roadmap shows the structure of the pattern language. The arrows in the roadmap point from a pattern to a set of patterns that system designers may want to consult next, once this pattern has been applied. It is also often useful to identify the forces that need to be resolved in the design of the systems targeted by the pattern language. These global forces establish a common vocabulary among the patterns, and can be used to summarize their contributions. The context section of each pattern in the pattern language describes a (specific) situation in which the pattern should be considered. In particular, the context includes references to other patterns in the language in whose context the pattern can be applied (“You are using pattern X, and now wish to address concern Y”). More than just referring to “related patterns” (usually external to this set of patterns), the resulting context section of each pattern similarly refers to other patterns in the same language that should be consulted next, together with a rationale (that is, the trade-off addressed) for each choice (“Also consult pattern X for dealing with concern Y”).

4

Pattern Language for Motivating an Agent-Based Approach

The structure of our pattern language is shown in the roadmap in Figure 1. The starting point (root) for the pattern language is AGENT SOCIETY. This pattern depends on AGENT AS DELEGATE, AGENT AS MEDIATOR, and COMMON VOCABULARY. The nature of these dependencies, that is, the rationale for applying each of these patterns, is documented in the Related Context section of the AGENT SOCIETY pattern.

Fig. 1. Roadmap for the pattern language


233

Although we lack the space for a full description of the global forces involved in agent-based design, we can identify the following trade-offs between: • the autonomy of an agent and its need to interact • the user’s information overload and the degree of control the user has over agents acting on his behalf • the openness of an agent society and the resulting dynamicity and heterogeneity • the need for intermediaries to facilitate agent interactions and the concern for the privacy of users’ sensitive data (one form of trust) • the heterogeneity and concern for quality of service (another form of trust) The patterns themselves will elaborate on these trade-offs in more detail. However, it is important to note that these trade-offs are independent of the application domain. As a motivating example for using the pattern language, consider the design of information agents for filtering news. The example uses AGENT AS DELEGATE in as much users are represented by USER AGENTS that maintain their profiles, and filter search results against them. Search requests are represented by TASK AGENTS: this includes one-shot searches, as well as subscriptions to periodically repeated searches. The example also requires AGENT AS MEDIATOR in as much users can obtain recommendations from each other. A recommender agent mediates between the users. Together these agents form an AGENT SOCIETY, and they therefore need to agree on a COMMON VOCABULARY to communicate with each other, and with news sources. AGENT SOCIETY Context Your application domain satisfies at least one of the following criteria: your domain data, control, knowledge, or resources are decentralized; your application can be naturally thought of as a system of autonomous cooperating entities; or you have legacy components that must be made to interoperate with new applications. Problem How do you model systems of autonomous cooperating entities in software? Forces • The entities are autonomous in the sense that they do not require the user’s approval at every step of executing their tasks, but can act on their own. • However, they rely on other entities to achieve goals that are outside their scope or reach, and need to cooperate with each other. • They also need to coordinate their behaviors with those of others to ensure that their own goals can be met, avoiding interference with each other. Solution Model your application as a society of agents. Agents are autonomous computational entities (autonomy), which interact with their environment (reactivity) and other agents (social ability) in order to achieve their own goals (proactiveness). Often,

234

M. Weiss

agents will be able to adapt to their environment, and have some degree of intelligence, although these are not considered mandatory characteristics. These computational entities act on behalf of users, or groups of users [17]. Thus agents can be classified as delegates, representing a single user, and acting on her behalf, or mediators, acting on behalf of a group of users, facilitating between them. The key differentiator between agents and objects is their autonomy. Autonomy is here used in an extended sense. It not only comprises the notion that agents operate in their own thread of control, but also implies that agents are long-lived (they execute unattended for long periods), they take initiative (they do not simply act in response to their environment), they react to stimuli from the environment as guided by their goals (the receiving agent decides whether and how to respond to a stimulus), and interact with other agents to leverage their abilities in support of their own as well as collective goals. Active objects, on the other hand, are autonomous only in the first of these senses. They are not guided by individual, and/or collective goals. A society of agents can be viewed from two dual perspectives: either a society of agents emerges as a result of the interaction of agents; or the society imposes constraints and policies on its constituent agents. Both perspectives, which we can refer to as micro and macro view of the society, respectively, mutually reinforce each other, as shown in Fig. 2. Specifically, emerging agent specialization leads to the notion of roles. Roles, in turn, impose restrictions on the possible behaviors of agents [10].

Fig. 2. Micro-macro view of an agent society

This suggests two approaches to systematically designing agent societies. In the first approach, we identify top-level goals for the system, and decompose them recursively, until we can assign them to individual agents (as exemplified by the Gaia methodology [30]). In the second approach, we construct an agent society incrementally from a catalog of interaction patterns, as exemplified by [13]. These interaction patterns are described in terms of roles that agents can play and their interactions, and may also specify any societal constraints or policies that need to be satisfied. Roles are abstract loci of control [13, 10, 20]. Protocols (or patterns of interaction) describe the way the roles interact. Policies define constraints imposed by the society on these roles. As an example of a policy, consider an agent-mediated auction, which specifies conventions specific to its auction type (for example, regarding the order of


235

bids; ascending in an English auction, descending in a Dutch auction) that participating agents must comply with in order for the auction to function correctly. Roles and their subtypes can be documented in a role diagram, using the notation introduced in [13]. Role diagrams are more abstract than class diagrams. Each role in the diagram defines a position, and a set of responsibilities. A role has collaborators – other roles it interacts with. Arrows between roles indicate dynamic interactions between roles; the direction of an arrow represents the direction in which messages are sent between the roles. The triangle indicates a subtyping relationship between roles; subtypes inherit the responsibilities of their parent roles Many types of applications, such as call control [19], groupware [26], or electronic commerce applications [27] can be modeled using user, task, service, and resource roles, and their subtypes. The user role encapsulates the behavior of managing a user’s task agents, and controlling access to the user’s data. The task role represents users in a specific task. This is typically a long-lived, rather than one-shot, transaction. Agents in the service role typically provide a service to a group of users. They mediate the interaction between two or more agents through this service. The resource role abstracts information sources. These could be legacy data sources wrapped by “glue” code that converts standardized requests to the API of the data source. Resulting Context • For members of an agent society to understand each other, they need to agree on common exchange formats, as described in Common Vocabulary. • If you are dealing with an agent that acts on behalf of a single user, consult Agent as Delegate. • For the motivation of agents that facilitate between a group of users, and their respective agents, refer to Agent as Mediator. AGENT AS DELEGATE Context You are designing your system as a society of autonomous agents using AGENT SOCIETY, and you wish to delegate a single user’s time-consuming, peripheral tasks. Problem How do you instruct agents on what to do? How much discretion (authority) should you give to an agent? How do agents interact with their environment? Forces • Repetitive, time-consuming tasks should be delegated to agents that can perform the tasks on behalf of their users, and require only minimal intervention. • However, when delegating a task to an agent, users must be able to trust the agent to perform the task in an informed and unbiased manner. • The user also wants to control what actions the agent can perform on the user’s behalf and which it cannot (its degree of autonomy).

236

M. Weiss

Solution Use agents to act on behalf of the user performing a specific task. Such user agents manage a user’s task agents, and control access to the user’s data. The structure of this pattern is shown in Fig. 3. Task agents represent the user in different task contexts. For example, in the call control domain, a user placing, and a user receiving a call could both be represented as task agents. Each Concrete Task is a subtype of the generic Task role. The generic role contains beliefs and behaviors common to all concrete tasks. In the electronic commerce domain, we might have a Trader role, and Buyer and Seller roles that share the common belief of a desired price.

Fig. 3. Role diagram for AGENT AS DELEGATE

Resulting Context • For organizing the interaction with the user to gather their requirements and feedback on the performance of a task, consult User Agent1. • Also consult User Agent for measures to control access to user data. • For the design of task agents, consult Task Agent1. AGENT AS MEDIATOR Context You are designing your system as a society of autonomous agents using AGENT SOCIETY, and you wish to facilitate between a group of users, and their agents. Problem How can agents find each other, and coordinate their behaviors? Forces • In a closed agent society of known composition, agents can maintain lists of acquaintances with whom they need to interact (to obtain data or services). • However, in an open agent society, whose composition changes dynamically, agents need help locating other agents with which they can interact. • Agents that have no mutual history of interaction may need the help of trusted intermediaries to protect sensitive data, and ensure service quality. 1

Descriptions of USER AGENT and TASK AGENT have been omitted from this paper for space restrictions, but will be included in a future version of this language.


237

• Sometimes, the agents do not simply need to locate each other, but their interaction needs to follow a coordination protocol (for example, in an auction). • A special case is that agents need to gain access to resources, which creates the need for an intermediary that can find, and forward queries to relevant resources. Solution Use a mediator to facilitate between the members of a group of agents. Examples of mediators are directories, translators, market makers, and rating services. We distinguish two cases, one where task agents need to locate other task agents, and another where task agents need to gain access to relevant resources [14, 26]. The mediator can either just pair up agents with each other (task agent-task agent, or task agent-resource agent), or coordinate their interactions beyond the initial introduction.

Fig. 4. Role diagram for AGENT AS MEDIATOR

In [14], mediators are referred to as middle-agents. Different types of middleagents can be distinguished based what services they provide. Basic mediation services comprise matching agents based on search criteria, and translation services. Interaction services include the capability to coordinate the behaviors of task agents according to given protocols, conventions, and policies, for example, the rules of an auction. Finally, reliability services comprise trustworthiness (of the mediator itself), and quality assurance (of the mediated services and data, as well as of the mediator). Resulting Context • The Agent as Mediator pattern is a starting point for many specific agent patterns, such as for search agents, recommender systems, or auctions. COMMON VOCABULARY Context When agents in an AGENT SOCIETY interact, they need to agree on common exchange formats. One scenario is that you are using agents to represent users in individual, long-living transactions as described in TASK AGENT. These task agents (for example, buyer and seller agents) need to understand each other in order to exchange messages with one other (for example, when negotiating about a price).

238

M. Weiss

Problem How do you enable agents (for example, task agents) to exchange information? Forces • Agents may use different internal representations of concepts. • To exchange information, agents need to agree on common exchange formats. • However, common exchange formats must be widely adopted to be useful. • If an agent needs to use multiple exchange formats to interact with different agents, it may not be able to perform all the required mappings itself. Solution For agents to understand each other, they need to agree on a common message format that is grounded in a common ontology. The ontology defines concepts that each party must use during the interaction, their attributes and valid value ranges. The purpose of this ontology is agent interaction, and it does not impose any restrictions on the internal representations of the agents. In a heterogeneous, open environment, agents may even need to use multiple ontologies to interact with different agents. The structure of this pattern is shown in Fig. 5.

Fig. 5. Role diagram for COMMON VOCABULARY

It is generally impractical to define general-purpose ontologies for agent interaction. These are unlikely to include the intricacies of all possible domains. Instead, the common ontology will be application-specific. Given such a shared ontology, the communicating agents need to map their internal representations to the shared ontology. Much progress has been made on XML-based ontologies, for example, in the e-commerce domain xCBL, cXML, and RosettaNet are quite popular [4]. If agents need to interact with many agents using different common ontologies, it becomes impractical for the agent to be aware of all the different mappings. In this case, the need for translation agents arises that can map between ontologies on behalf of other agents. Fortunately, these are relatively straightforward to build using XSLT, a language for transforming XML documents into other XML documents.


239

Resulting Context • If agents need to interact with many agents using different common ontologies, apply Agent as Mediator to the construction of translation agents.

5

Conclusion

There are three main take home messages from this paper: • As the advantages of the agent-based approach are still not widely recognized outside our community, we need to educate non-experts in its use. • We need guidelines for non-agent technology experts that summarize the key benefits of the agent approach, and agent patterns can provide that guidance. • This pattern language at the center of this paper is intended to provide such guidelines, and serve as a starting point for more specific pattern languages. We also urge authors of agent patterns to organize their patterns in the form of pattern languages. To this end a template for pattern languages has been provided.

References 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14.

Alexander, C., A Pattern Language, Oxford University Press, 1977 Aridor, Y., Lange, D., Agent Design Patterns: Elements of Agent Application Design, Second Intl. Conference on Autonomous Agents, IEEE, 1998 Beck, K., and Johnson, R., Patterns Generate Architectures, European Conference on Object Oriented Programming (ECOOP), 139–149, 1994 Carlson, D., Modeling XML Applications with UML : Practical e-Business Applications, Addison-Wesley, 2001 Collis, J., and Ndumu, D., The ZEUS Role Modelling Guide, BT Exact, 1999 Coplien, J., Software Patterns, SIGS Books, 1996 Deugo, D., and Weiss, M., A Case for Mobile Agent Patterns, Mobile Agents in the Context of Competition and Cooperation (MAC3) Workshop Notes, 19–22, 1999 Deugo, D., Oppacher, F., Ashfield, B., Weiss, M., Communication as a Means to Differentiate Objects, Components and Agents, Technology of Object-Oriented Languages and Systems Conference (TOOLS), IEEE, 376–386, 1999 Deugo, D., Weiss, M., and Kendall, E., Reusable Patterns for Agent Coordination, in: Omicini, A., et al (eds.), Coordination of Internet Agents, Springer, 2001 Ferber, J., Multi-Agent Systems: An Introduction to Distributed Artificial Intelligence, Addison-Wesley, 13–16, 1999 Fernandez, E., and Pan, R., A Pattern Language for Security Models, Conference on Pattern Languages of Programming (PLoP), 2001 Kendall, E., Murali Krishna, P., Pathak, C., et al, Patterns of Intelligent and Mobile Agents, Conference on Autonomous Agents, IEEE, 1998 Kendall, E., Role Models: Patterns of Agent System Analysis and Design, Symposium on Agent Systems and Applications/Mobile Agents (ASA/MA), ACM, 1999 Klusch, M., and Sycara, K., Brokering and Matchmaking for Coordination of Agent Societies: A Survey, in: Omicini, A., et al (eds.), Coordination of Internet Agents, Springer, 2001

240

M. Weiss

15. Kolp, M., Giorgini, P., and Mylopoulos, J., A Goal-Based Organizational Perspective on Multi-Agent Architectures, Workshop on Agent Theories, Architectures, and Languages (ATAL), 2001 16. Lind, J., Patterns in Agent-Oriented Software Engineering, Workshop on Agent-Oriented Software Engineering (AOSE), 2002 17. Maes, P., Agents that Reduce Work and Information Overload, Communications of the ACM, 31–41, July 1994 18. Mouratidis, H., Giorgini, P., Schumacher, M., and Weiss, M., Integrating Security Patterns in the Development of Secure Agent-Based Systems, submitted, 2003 19. Pinard, D., Gray, T., Mankovski, S., and Weiss, M., Issues in Using an Agent Framework for Converged Voice and Data Applications, Conference on Practical Applications of Agents and Multi-Agents (PAAM), 1997 20. Riehle, D., and Gross, T., Role Model Based Framework Design and Integration, Conference on Object-Oriented Programs, Systems, Languages, and Applications (OOPSLA), 1998 21. Schelfthout, K., Coninx, T., et al, Agent Implementation Patterns, OOPSLA Workshop on Agent-Oriented Methodologies, 2002 22. Shu, S., and Norrie, D., Patterns for Adaptive Multi-Agent Systems in Intelligent Manufacturing, Intl. Workshop on Intelligent Manufacturing Systems (IMS), 1999 23. Silva, A., and Delgado, J., The Agent Pattern, European Conference on Pattern Languages of Programming and Computing (EuroPLoP), 1998 24. Sycara, K., Multiagent Systems, AI Magazine, 79–92, Summer 1998 25. Tahara, Y., Oshuga, A., and Hiniden, S., Agent System Development Method Based on Agent Patterns, Intl. Conference on Software Engineering (ICSE), ACM, 1999 26. Voss, A., and Kreifelts, T., SOaP: Social Agents Providing People with Useful Information, Conference on Supporting Groupwork (GROUP), ACM, 1997 27. Weiss, M., Patterns for e-Commerce Agent Architectures: Using Agents as Delegates, Conference on Pattern Languages of Programming (PLoP), 2001 28. Weiss, M., Pattern-Driven Design of Agent Systems: Approach and Case Study, Conference on Advanced Information System Engineering (CAiSE), Springer, 2003 29. Wooldridge, M., and Jennings, N., Intelligent Agents: Theory and Practice, The Knowledge Engineering Review, 10(2):115–152, 1995 30. Wooldridge, M., Jennings, N., et al, The Gaia Methodology for Agent-oriented Analysis and Design, Journal Autonomous Agents and Multi-Agent Systems, 2002

Using Scenarios for Contextual Design in Agent-Oriented Information Systems Kibum Kim, John M. Carroll, and Mary Beth Rosson Departement of Computer Science and Center for Human-Computer Interaction, Virginia Tech, Blacksburg, 24061, USA {kikim,carroll,rosson}@vt.edu

Abstract. In this position paper, we argue that current agent-oriented development methodologies are limited in their ability to model social aspects of the agents and human-software agent interactions. We identify how these limitations can be rectified using scenarios for contextual design in agentoriented information systems (AOIS).

1 Limitations in Traditional Approach for Agent Oriented Development Methodologies Currently, the primary approaches to agent-oriented development methodologies involve either adopting conventional software engineering methodologies—for example, Agent UML (AUML) and Methodology for Engineering Systems of Software Agents (MESSAGE)—or extending knowledge-engineering methodologies with agent-related concepts such as Conceptual Modeling of Multi-Agent Systems (CoMoMas), Gaia, and MAS-CommonKads [1]. However, because these approaches possess system-centric rather than user-centric natures or designs, they remain inappropriate for dealing with the important developmental process of humansoftware agent interactions, as well as the human factors for developing interactive agents. Most problems associated with the adoption of conventional software engineering methodologies derive from the essential differences between distributed objects and agents. In particular, although objects are not social, agents are characterized by their social aspects, and existing software development techniques usually do not adapt to this purpose. In addition, while extending knowledge-engineering methodologies to agent development can provide techniques for modeling agent knowledge, they do not effectively deal with the distributed or social aspects of the agents, or modeling such social interactions. Therefore, theoretical frameworks must be presented that will analyze how people communicate and interact with the variety of agents that constitute their work environments.

M.A. Jeusfeld and Ó. Pastor (Eds.): ER 2003 Workshops, LNCS 2814, pp. 241–243, 2003. © Springer-Verlag Berlin Heidelberg 2003

242

K. Kim, J.M. Carroll, and M.B. Rosson

2 Scenario-Based Contextual Design in Agent Oriented Information Systems Increasingly, information technology companies are moving from being technologydriven to being customer-driven, focused on ensuring system functions and structures that will work for the customer. A broad range of analysis involving human-computer interaction has already recognized that system design can profit from explicitly studying the context in which users work [2]. To achieve systems that are more “customer-centered,” one must depend upon Contextual Design as the backbone methodology for front-end design [3]. AOIS can achieve its goal most effectively when its design methodology takes into account what customers need, as well as how human-software agent interactions and social interactions between agents are structured within a usage context. Scenarios—descriptions of meaningful usage episodes—have become key to making abstract models understandable. They help designers examine and test their ideas in practice, with each narrative created to evoke an image of people doing things, pursuing goals, and using technology to support these goals. In Scenario-Based Design (SBD), designers also evaluate scenarios through claims analysis, wherein the positive and negative implications of design features are considered through “what if” discussions and the scenarios serve as a usage context for considering interaction options. Scenarios and claims analysis are useful in describing initiatives or actions taken by a software agent and considering their usability implications and emphasize the context of work in the real world. The Point-of-View Scenarios (POVs) describe each agent’s responsibilities in the scenario, including the extent of its collaboration with other agents [4]. Creating POVs encourages software designers to anthropomorphize agents and their behaviors, as a heuristic for reasoning about what the agent could or should do to support user needs (See Table 1). They help designers construct an initial analysis of the attributes and responsibilities of individual agents, which might lead them to consider how different agents might influence what users will be able to do with the system. In light of the POV analysis described in Table 1, Table 2 depicts usability tradeoffs that must be considered. Table 1. Points of veiw scenarios (POVs) created from the perspective of a software agent Scenario Agent Social Network Visualizer

Point of View Scenarios I was created based on Mrs. Parry’s constant email correspondence with her colleagues. When she first opened me, I asked a database manager for information about her social networks and displayed her personal connections and groups. When she sent email to a new person, I worked with it to set up a new relationship in her social network. Whenever I was asked display myself, I made sure all my nodes and links were shown correctly within the frame.

Using Scenarios for Contextual Design in Agent-Oriented Information Systems

243

Table 2. Examples of usability tradeoffs to consider in light of the POV analysis Scenario Feature Automatical creation based on email correspondece

Possible Positive (+) and Negative (-) Attributes of Feature + assists users by maintaining networks of colleagues, acquaintances, and friends based on their personal histories or behaviors. + can quickly and conveniently explore who knows whom through social networks. - user intentions might be incorrectly determined by the agent. - there is potential for losing user’s control, predictability, and comprehensibility.

3 Conclusion Developing a sound solution for well-defined agent-oriented development methodologies is not an easy task. We have briefly explored the possible contributions that a scenario-based contextual design approach for AOIS might provide to designers interested in pursuing a methodology that can facilitate modeling social aspects of the agents and human-software agent interactions. For AOIS development, we recommend using Point-of-View Scenarios created with the aid of anthropomorphism, because they can envision a user’s task in terms of a usage context relevant to the problem domain. Several usability issues are also raised by these POVs. Such a methodology will provide a well-organized framework within which software engineers can more effectively develop agent-oriented information systems.

References 1. Iglesias, C.A., Garijo, M., Gonzalez, J.C.: A Survey of Agent-Oriented Methodologies. In 2. 3. 4.

Intelligent Agents V: Proceedings of the ATAL’98, LNAI, vol. 1555. Springer, Berlin Heidelberg NY. (1999) Nardi, B.A.: Context and Consciousness: Activity Theory and Human Computer Interaction, 1st edn. MIT Press, Cambridge, MA. (1996) Beyer,H., Holtzblatt, K.: Contextual Design: Defining Customer-Centered Systems, Morgan Kaufmann Publishers, Inc., San Francisco, CA. (1998) Rosson, M. B., Carroll, J. M.: Scenarios, objects, and points-of-view in user interface design, In M. van Harmelen (ed.): Object modeling and user interface design. AddisonWesley, London. (2000)

Dynamic Matchmaking between Messages and Services in Multi-agent Information Systems Muhammed Al-Muhammed and David W. Embley Department of Computer Science, Brigham Young University Provo, UT 84602 USA {mja47, embley}@cs.byu.edu

1 Problem Statement Agents do not work in isolation; instead they work in cooperative groups to accomplish their assigned tasks. In a multi-agent information system, we assume that each of the agents has and acquires knowledge. We further assume that it is important and useful to be able to share this knowledge and to provide useful knowledge sources to enable activities such as comparison shopping, meeting scheduling, and supply-chain management. In order for agents to cooperate, they need to be able to communicate with one another. Communication essentially needs mutual understanding among agents. To achieve this mutual understanding among agents, researchers frequently make three assumptions: 1. Agents share ontologies that define the concepts used by these agents; 2. Agents communicate with the same agent communication language so that they can understand the semantics of the messages; and 3. Agents pre-agree on a message format so that they can correctly parse and understand the communicative act. These three assumptions are sufficient for agents to communicate; however, they impose many problems. First and foremost, they imply that agents cannot communicate (interoperate with one another) without agreeing in advance on these three assumptions. Hence, these assumptions preclude agents from interoperating on the fly (without a-priori agreement). Second, they explicitly mean that unless one designer or a group of designers (with full communication among them) develops these agents, the communication among agents is not likely to succeed because all or some of the three assumptions will not hold. Third, the assumptions require a designer who develops a new agent for a multi-agent system to know what ontologies other agents in that system use, what language they speak, and what message format they use. This imposes a stiff requirement on an outside developer. The importance of making agents interoperate on the fly becomes paramount. Indeed, in an interesting paper on agent interoperability, Uschold says that “the holy grail of semantic integration in architectures” is to “allow two agents to generate needed mappings between them on the fly without a-priori agreement and without them having built-in knowledge of any common ontology.” Consequently, in our research we are working on eliminating all three assumptions and allowing agents to interoperate on the fly without having to share knowledge of any ontology, language, or message format. M.A. Jeusfeld and Ó. Pastor (Eds.): ER 2003 Workshops, LNCS 2814, pp. 244–246, 2003. © Springer-Verlag Berlin Heidelberg 2003

Dynamic Matchmaking between Messages and Services

245

2 Research Questions To achieve interoperability among agents on the fly, we must have answers to five major research issues: (1) translating between different ontologies, (2) mapping between services and messages, (3) reconciling differences in data formats, (4) reconciling type mismatches, and (5) handling outputs of services so that only the information requested by a message is provided. 1. Given that agents do not share ontologies – they may represent the same concept using different vocabularies – how can translation among different ontologies be done? 1.a. How can the concepts of independent ontologies – related to the same domain – be matched? In particular, answers for the following subquestions are vital. 1.a.1. How can the semantically related concepts be determined? 1.a.2. How can the concepts whose names are the same but whose semantics are different be distinguished? 1.b. What is the information needed to make the translation work? 1.b.1. Can this information be extracted from the agents themselves? 1.b.2. What other resources are needed? 1.b.3. How much information from a multi-agent system is sufficient to do the translation correctly? 2. How can a message be mapped to an appropriate service? 2.a. How can the semantics of a message be captured? 2.a.1. What is the provided information – what will be the values for the service input parameters? 2.a.2. What is the required output—what is the message is asking for? 2.a.3. What are the constraints imposed on input (output) parameters? 2.b. How can a message be mapped to some service provided by a receiving agent? This question requires the ability to know the semantics of a service, which requires answers to the following. 2.b.1. What are the semantics of the input parameters and how do these input parameters match with those of a message? 2.b.2. What are semantics of the outputs and do the outputs constitute an answer for a message? 2.b.3. What are the constraints imposed on inputs and outputs and how can the mismatches between service input (output) constraints and message input (output) constraints be resolved? 3. How can differences among data formats (different date formats, time formats, currencies) of the communicating agents be recognized the and then converted correctly? 3.a. What are the problems that arise when converting from one format to another and how can they be resolved? 3.b. How can alternative value representations be guaranteed to match under various conversions?

246

M. Al-Muhammed and D.W. Embley

4. How can the mismatches between types be handled? 4.a. Can the proper conversion be guaranteed? 4.b. Can the loss of precision be recovered? 5. How can any unwanted output of a service be filtered out? 5.a. How can the expected output, which is represented in the local vocabulary of a receiving agent, be recognized? 5.b. Then, how can this output be sifted so that only the required information is delivered to a requesting agent? We realize that fully resolving these questions is an extremely hard problem. Nevertheless, we believe that the benefits of resolving these issues are of great value. These benefits include: (1) easy development of agents because developers need not be concerned with ontologies other agents use, agent communication languages other agents use, and the format of messages and (2) increased interoperability as agents can generate needed mappings on the fly. Thus, tackling these questions and solving the heterogeneous, semantic-mapping problem would be of great benefit.

Preface to XSDM 2003

XSDM’03 (XML Schema and Data Management) was the first International workshop held in conjunction with 22nd International Conference on Conceptual Modeling on 13th October in Chicago, USA. Web data management systems are rapidly being influenced by XML technologies and are driven by their growth and proliferation to create next generation web information systems. The purpose of XSDM workshop is to provide a forum for the exchange of ideas, and experiences among the theoretical and practitioners of XML technologies, who are involved in design, management and implementation of XML based web information systems. Topics of interest in XSDM’03 include, but were not limited to – – – – – –

XML schema discovery, XML data integration Indexing XML data, XML query languages XML data semantics, Semantic web and XML Mining of XML data, XML change management XML views and data mappings, Securing XML data XML in new domains- sensor and biological data management

The workshop received overwhelming response from many different countries. We received a total of 30 papers and the international program committee members reviewed the papers and finally selected 12 full papers and 2 short papers for the presentation and inclusion in the proceedings. The workshop program consisted of 4 sessions, XML Change Management and Indexing, Querying and Storing XML Data, XML Transformation and Generation, and XML Mapping and Extraction. I thank all the authors who contributed to these workshops. I also thank the program committee members of the workshop who selected such quality papers, which resulted in an excellent technical program. I thank to workshop co-chairs Manfred Jeusfeld and Oscar Pastor for selecting this workshop, and for excellent co-ordination and co-operation during this period. I would also like to thank ER Conference organization committees for their support and help. Finally, I would like to thank the local organizing committee for wonderful arrangement and all the participants for attending these workshops and stimulating the technical discussions. We hope that all participants enjoyed the workshops and local sightseeing. You should shortly introduce all papers presented at your workshop.

October 2003

Sanjay Madria

A Sufficient and Necessary Condition for the Consistency of XML DTDs Shiyong Lu, Yezhou Sun, Mustafa Atay, and Farshad Fotouhi Department Of Computer Science Wayne State University Detroit, MI 48202 {shiyong,sunny,matay,fotouhi}@cs.wayne.edu

Abstract. Recently, XML has emerged as a standard for representing and exchanging data on the World Wide Web. As a result, there is a trend of increasing amount of XML documents that publish information on the Web from various data sources. A Document Type Definition (DTD) describes the structure of a set of similar XML documents and serves as the schema for XML documents. The World Wide Web Consortium has defined the grammar for specifying DTDs; however, even a syntactically correct DTD might be inconsistent in the sense that there exist no XML documents conforming to the structure imposed by the DTD. In this paper, we formalize the notion of the consistency of DTDs, and identify a sufficient and necessary condition for a DTD to be consistent.

1

Introduction

XML [2] is rapidly emerging on the World Wide Web as a standard for representing and exchanging data. In contrast to HTML, which describes how data should be displayed to humans, XML describes the meanings and structures of data elements themselves, and therefore makes data self-describing and interpretable to programs. Currently, XML has been used in a wide range of applications as this is facilitated by standard interfaces such as SAX [13] and DOM [1], and the development of techniques and tools for XML such as XSL (Extensible Stylesheet Language) [5], XSLT (XSL Transformation) [3], XPath [4], XLink [11], XPointer [12] and XML parsers. It is well recognized that XML will continue to play an essential role in the development of the Semantic Web [9], the next generation web. XML Document Type Definitions (DTDs) [2] describe the structure of XML documents. With a DTD, independent groups of people can agree to use a common DTD for interchanging data. In addition, an application can use a standard DTD to verify if the data that the application receives from the outside world is valid or not. The World Wide Web Consortium has defined the grammar for DTDs [2]. Essentially, a DTD defines the constraints on the logical structure of XML documents, and an XML document is valid if it has an associated DTD and if the document complies with the constraints expressed in the DTD. Unfortunately, a syntactically correct DTD might be inconsistent in the sense ´ Pastor (Eds.): ER 2003 Workshops, LNCS 2814, pp. 250–260, 2003. M.A. Jeusfeld and O. c Springer-Verlag Berlin Heidelberg 2003

A Sufficient and Necessary Condition for the Consistency of XML DTDs

251

]>

Fig. 1. A DTD example

that there exist no XML documents conforming to the structure imposed by the DTD. Figure 4 shows some of such inconsistent DTDs. Inconsistent DTDs are of course useless, and they should be avoided. In practice, the consistency of small DTDs can be ensured by careful observation based on common sense; tools of checking consistency of DTDs might be desirable for large DTDs or for DTDs that are generated automatically from other data models such as the ER model and the relational model. In this paper, we first formalize the notion of consistency of an XML DTD: an XML DTD is consistent if and only if there exists at least one XML document that is valid w.r.t. it. We then identify a sufficient and necessary condition for an XML DTD to be consistent, and this condition also implies an algorithm for checking the consistency of XML DTDs. We believe that these results are fundamentally important for the XML theory and the semistructured data model. Organization. The rest of the paper is organized as follows. Section 2 gives a brief overview of XML Document Type Definitions (DTDs), and formalizes the notion of consistency of DTDs. Section 3 identifies a sufficient and necessary condition for the consistency of DTDs, which also implies an algorithm that checks the consistency of DTDs. Section 4 discusses our implementation of the algorithm and related work. Finally, Section 5 concludes the paper.

2

Consistency of DTDs

XML Document Type Definitions (DTDs) [2] describe the structure of XML documents and are considered as the schemas for XML documents. A DTD example is shown in Figure 1 for memorandum XML documents. In this paper, we model both XML elements and XML attributes as XML elements since XML attributes can be considered as XML elements without further nesting structure. A DTD D is modeled as a set of XML element definitions {d1 , d2 , · · · , dk }. Each XML element definition di (i = 1, · · · , k) is in the form of ni = ei , where ni is the name of an XML element, and ei is a DTD expression.

252

S. Lu et al.

Each DTD expression is composed from XML element names (called primitive DTD expressions) and other DTD subexpressions using the following operators: – Tuple operator. (e1 , e2 , · · · , en ) denotes a tuple of DTD subexpressions. In particular, we consider (e) is a singleton tuple. The tuple operator is denoted by “,”. – Star operator. e∗ represents zero or more occurrences of subexpression e. – Plus operator. e+ represents one or more occurrences of subexpression e. – Optional operator. e? represents an optional occurrence (0 or 1) of subexpression e. – Or operator. (e1 | e2 | · · · | en ) represents one occurrence of one of the subexpressions e1 , e2 , · · ·, en . We ignore the encoding mechanisms that are used in data types PCDATA and CDATA and model both of them as data type string. The DOCTYPE declaration states which XML element will be used as the schema for XML documents. This XML element is called the root element. We define a DTD expression formally as follows. Definition 1. A DTD expression e is defined recursively in the following BNF notation where n range over XML element names and e1 , · · ·, en range over DTD expressions. e ::= string | n | e+ | e∗ | e? | (e1 , · · · , en ) | (e1 | · · · |en ) where the symbol “::=” should be read as “is defined as” and “|” as “or”. Example 1. With our modeling notations, the DTD shown in Figure 1 can be represented as the following set of XML element definitions { memo = (to, from, date, subject?, body, security, lang), security = string, lang = string, to = string, from = string, date = string, subject = string, body = (para+), para = string }. Informally, an XML document is valid with respect to a DTD if the structure of the XML document conforms to the constraints expressed in the DTD (see [2] for a formal definition of validity), and invalid otherwise. Figure 2 shows a valid XML document with respect to the DTD shown in Figure 1. Figure 3 illustrates an invalid XML document with respect to the same DTD since XML element cc is not defined in the DTD and the required element date is missing. However, some DTDs are inconsistent in the sense that there exist no XML documents that are valid with respect to them. For example, the four DTDs shown in Figure 4 are inconsistent. DTD1 is inconsistent since it requires that element a contains an element b and vice versa. This is impossible for any XML document with a finite size. For similar reasons, other DTDs are inconsistent. We formalize the consistency of DTDs as follows. Definition 2. A DTD is consistent if and only if there exists at least one XML document that is valid with respect to it; otherwise, it is inconsistent.


253

<memo security="5" lang="eng"> yezhou mustafa October 27, 2002 <subject> Why XML <para> Is XML really better than HTML?

Fig. 2. A valid XML document

<memo security="5" lang="eng"> mustafa shiyong yezhou <subject> Re: Why XML What HTML are you talking about?

Fig. 3. An invalid XML document

3

Condition for the Consistency of DTDs

Obviously, inconsistent DTDs are not useful in real life although they are syntactically correct. Hence, it is important to characterize consistent DTDs. To do this, we introduce the notion of DTD graph which graphically represents the structure of a DTD. In this graph, nodes represent XML elements and edges represent operators over them and are labeled by the corresponding operators. An edge in a DTD graph is called a ?-edge, ∗-edge, +-edge, ,-edge or |-edge respectively according to the operator label on it. In this paper, we call ,- and +-edges hard edges, and other kinds of edges soft edges. Similarly, a path (or a cycle) p = e1 → e2 → · · · en is hard if each edge ei → ei+1 (for i = 1, · · ·, n-1) in p is hard, and soft otherwise. For the brevity of a graph, the label of a ,-edge is omitted and made implicit. Nodes without incoming edges are called sources and nodes without outgoing edges are called terminals. Note that our notion of DTD graph differs from the one defined in [14] in which operators are also treated as nodes of the graph.

254

S. Lu et al.

DTD1: ]>

DTD2: ]>

DTD3: ]>

DTD4: ]>

Fig. 4. Inconsistent DTDs

a

a

c +

+ b

b

DTD1

DTD2

d e

a

a

+

+ c

b

c

b

*

f

g DTD3

DTD4

Fig. 5. Cyclic and inconsistent DTD Graphs

Definition 3. Given a DTD graph g, we say XML element e leads to f if e = f or if there is a hard path from e to f in g. Definition 4. XML element e leads to a cycle c if e leads to any XML element in c. Example 2. The DTD graphs for the DTDs in Figure 4 are illustrated in Figure 5. In all of these graphs, the root element a leads to a cycle. All of the four DTDs are inconsistent.

A Sufficient and Necessary Condition for the Consistency of XML DTDs DTD5: ]>

DTD6: ]>

DTD7: ]>

DTD8: ]>

255

Fig. 6. Consistent DTDs despite mutual recursions

However, not all cycles imply the inconsistency of DTDs since soft edges do not cause the inconsistency of a DTD. This is illustrated by the following example. Example 3. The DTDs shown in Figure 6 are all consistent although they contain mutual recursions and their corresponding DTD graphs (Figure 7) have cycles. In particular, DTD5 and DTD7 contain soft cycles, but DTD6 and DTD8 contain hard cycles.

a

a

b

b

? DTD5

c DTD6 |

a

a

d

|

* b

c

b

c

? DTD7

DTD8

Fig. 7. Cyclic but consistent DTD Graphs

256

S. Lu et al.

The operator | introduces complexity. For example, in DTD8 of Figure 7, the DTD is consistent despite the presence of a hard cycle since XML document Hello world! is valid w.r.t. it. In the following, to identify the condition for a DTD to be consistent, we first consider those DTDs that do not involve operator “|”. We call these DTDs |-free DTDs. We will consider DTDs that involve operator “|” later. In Figure 6, DTD5, DTD6 and DTD7 are all |-free DTDs while DTD8 is not. The following lemma identifies a necessary condition for an |-free DTD to be consistent. Lemma 1. A |-free DTD D with root r is consistent only if r does not lead to a hard cycle in its DTD graph. Proof. Suppose r leads to a hard cycle e0 → e1 → · · · → en → e0 , then any XML document that is valid w.r.t. D must contain all the elements in this cycle. In the following, we prove by contradiction that D is inconsistent. Suppose D is consistent, then there exists an XML document x that is valid w.r.t. D. x must contain all the elements e0 , · · ·, en . Let ei be the innermost element of x in the sense that ei does not contain any other elements from e0 , · · · , en . The finite size of x implies that such an innermost element ei exists. However, the edge from ei to ei+1 is either a ,-edge or a +-edge. This implies that ei must contain at least one occurrence of ei+1 . This contradicts the assumption that ei is the innermost element of x. Therefore, there exists no XML document x that is valid w.r.t. D and hence D is inconsistent. The following lemma identifies a sufficient condition for a DTD (including |-free DTDs) to be consistent. Lemma 2. A DTD D with root r is consistent if r does not lead to any hard cycle. Proof. Suppose r does not lead to any hard cycle in the DTD graph of D. In the following, we prove D is consistent by constructing an XML document that is valid w.r.t. D. First, we convert D into another DTD D using the following transformation rules where empty represents the empty element, i.e., e? ≡ e | empty, and e, e1 , · · · , en range over DTD expressions: (1) e+ ⇒ e; (2) e∗ ⇒ empty; (3) e? ⇒ empty. (4) (e1 | · · · | en ) ⇒ e1 . Obviously, the DTD graph of D only contains ,-edges. In addition, the strongly connected subgraph g of the DTD graph of D that contains r must be acyclic since r does not lead to any hard cycle in the DTD graph of D. Based on the acyclicity of g , it is straightforward to create an XML document x that conforms to g (we leave the detail of the creation to the readers) and thus is valid w.r.t. to D . Since any document that is valid w.r.t. D must also be valid w.r.t. D, x is valid w.r.t. D as well. Hence, D is consistent. Based on Lemma 2 and 1, the following theorem states a sufficient and necessary condition for an |-free DTD to be consistent. Theorem 1. An |-free DTD with root r is consistent if and only if r does not lead to any hard cycle.


257

Proof. According to Lemmas 1 and 2. With the above theorem, the algorithm to decide if an |-free DTD D with root r is consistent or not is straightforward: create the DTD graph for D and do a depth first traversal of this graph starting at r following only hard edges. Each node is marked as “visited” the first time it is reached. If a node is visited twice during this traversal, then r leads to a hard cycle and D is inconsistent; otherwise, D is consistent. Example 4. Consider the DTDs and their DTD graphs shown in Figure 6 and 7 respectively. DTD5 is consistent since there is no hard cycle present in its DTD graph; DTD6 is consistent although there is a hard cycle in its DTD graph since the root element a does not lead to that hard cycle; DTD7 is also consistent since the cycle is not a hard one. The consistency problem of DTD8 will be discussed later. Theorem 2 (Complexity). The time complexity of checking whether an |-free DTD D is consistent or not is O(n) where n is the size of D. Proof. Both the creation of the DTD graph for D and the checking of the cyclicity of this graph can be done in O(n) where n is the size of D. We leave the detail of the proof to the readers. To deal with | operators, we split a DTD D to a set of |-free DTDs D1 , D2 , · · · , Dm such that D is consistent if and only if one of D1 , D2 , · · · , Dm is consistent. We first formalize the notion of a split of a DTD expression, and then we extend this notion to one for XML element definition and one for XML DTD. Definition 5. The split of a DTD expression e is a set of |-free DTD expressions split(e) that is defined recursively by the following rules: – – – – – – –

split(string) = {string}. split(n) = {n}. split(e+ ) = {g + | g ∈ split(e)}. split(e∗ ) = {g ∗ | g ∈ split(e)}. split(e?) = {g? | g ∈ split(e)}. split((e1 , e2 , · · · , en )) = {(g1 , g2 , · · · , gn ) | gi ∈ split(ei ) f or i = 1, · · · , n}. split((e1 | e2 | · · · | en )) = split(e1 ) ∪ split(e2 ) ∪ · · · ∪ split(en ).

Example 5. split((a+, b?)*) = {(a+, b?)*} and split((a?, b* | c , d | e)) = {(a?, b*, d), (a?, b*, e), (a?, c, d), (a?, c, e)}. The following lemma indicates that |-free DTD expressions are fixpoints of the split transformation function. Lemma 3. All DTD expressions in split(e) are |-free and for each |-free DTD expression e, we have split(e) = e.

258

S. Lu et al.

Proof. We can easily prove it by applying a structural induction on e. We extend the notion of split to one for an XML element definition. Definition 6. The split of an XML element definition di (with the form ni = ei ) is a set of |-free XML element definitions split(d) that is defined as split(ni = ei ) = {ni = g | g ∈ split(ei )}. Example 6. split(n=(a+, b?)*) = {n=(a+, b?)*} and split(n=(a?, b* | c , d | e)) = {n=(a?, b*, d), n=(a?, b*, e), n=(a?, c, d), n=(a?, c, e)}. The following lemma indicates that |-free XML element definitions are fixpoints of the split transformation function. Lemma 4. All XML element definitions in split(d) are |-free and for each |-free XML element definition d, we have split(d) = d. Proof. It follows from Lemma 3 and the notion of split for XML element definitions in Definition 6. Finally, we extend the notion of split to one for a DTD. Definition 7. The split of an XML DTD D = {d1 , d2 , · · · , dk } is a set of |free DTDs split(D) that is defined as split({d1 , d2 , · · · , dk }) = {{d1 , d2 , · · · , dk } | di ∈ split(di ) f or i = 1, 2, · · · , k}. Example 7. DTD8 in Figure 6 can be represented as D = {a = b | d, b = c, c = b, d = string} and we have split(D) = {{a = b, b = c, c = b, d = string}, {a = d, b = c, c = b, d = string}}. Notice that the first DTD in split(D) is inconsistent since root a leads to a hard cycle, but the second one is consistent since root a does not lead to any hard cycle (despite the presence of a hard cycle). The following theorem indicates that |-free DTDs are fixpoints of the split transformation function. Theorem 3. All DTDs in split(D) are |-free and for each |-free DTD D, we have split(D) = D. Proof. It follows from Lemma 4 and the notion of split for XML DTDs in Definition 7. Finally, we identify a sufficient and necessary condition for an arbitrary DTD to be consistent. Theorem 4. A DTD D with root r is consistent if and only if at least one of the DTDs in split(D) with root r is consistent. Proof. It follows from the split notions defined in Definition 5, 6 and 7.


259

For example, DTD8 shown in Figure 7 is consistent as one of its splits, the one with the cycle of b and c deleted, is consistent. With the above theorem, the algorithm to decide if an arbitrary D with root r is consistent or not is straightforward: calculate split(D) which is a set of |-free DTDs, and check if there exists a DTD in split(D) that is consistent using the decision procedure for an |-free DTD. The following theorem states the complexity of checking the consistency of an arbitrary DTD; it shows that | might increase the complexity in an exponential fashion. Theorem 5 (Complexity). The time complexity of checking whether a DTD D is consistent or not is O(n ∗ 2m ) where n is the size of D and m is the number of | in D. Proof. Based on Theorem 2 and the fact that split(D) contains at most 2m |-free DTDs, each of which has the size of n.

4

Implementation and Related Work

We have implemented our algorithm in Java and have used it in one of our XML projects. The source code is downloadable at http://database.cs.wayne.edu/download/dtd con.zip. Recently, we noticed that Fan and Libkin have studied the consistency problem in a more general context in which the structural constraints enforced by XML DTDs might interact with integrity constraints [7] [8]. They have shown that the consistency problem of a DTD can be reduced to the emptiness problem for a context free grammar, which is decidable in linear time. In contrast, our algorithm is linear for |-free DTDs but exponential for DTDs in which |’s are present. Our contribution is that we identified an explicit sufficient and necessary condition for the consistency of XML DTDs in terms of the cyclicity of DTD graphs. Therefore, our algorithm and implementation are useful in a context in which DTD graphs are used, such as [14] [10]. An experimental comparison study between these two approaches is an interesting future work but is beyond the scope of this paper.

5

Conclusions and Future Work

In this paper, we have formalized the notion of consistency of XML DTDs and identified a sufficient and necessary condition for a DTD to be consistent. This condition implies an algorithm for checking the consistency of DTDs. We have implemented the algorithm with Java and have used it in one of our projects on XML. We expect that this algorithm might be integrated into various XML tools where the consistency of XML DTDs is critical. XML Schema [15] is a recent W3C standard. In addition to the features in DTDs, XML Schema supports typing of values and set size specification. There are several studies to convert a DTD into an XML Schema [6]. A DTD still needs

260

S. Lu et al.

to be consistent before it is converted to an XML Schema, and the algorithm introduced in this paper will be applicable to XML Schema with minor extension as a future work. Acknowledgements. We are grateful to the anonymous reviewers for their helpful comments and suggestions. We are also thankful to Lei Chen who implemented our proposed consistency checking algorithm in Java.

References 1. Document Object Model (DOM), October 1998. http://www.w3.org/DOM/. 2. T. Bray, J. Paoli, C. Sperberg-McQueen, and E. Maler. Extensible Markup Language (XML) 1.0, October 2000. http://www.w3.org/TR/REC-xml. 3. J. Clark. XSL Transformation (XSLT) Version 1.0., November 1999. http://www.w3.org/TR/xslt. 4. J. Clark and S. DeRose. XML Path Language (XPath) Recommendation. http://www.w3.org/TR/xpath. 5. S. Deach. Extensible Stylesheet Language (XSL) Specification. http://www.w3.org/TR/xsl. 6. R. dos Santos Mello and C. A. Heuser. A rule-based conversion of a DTD to a conceptual schema. Lecture Notes in Computer Science, 2224:133, 2001. 7. W. Fan and L. Libkin. On XML integrity constraints in the presence of DTDs. In In: Proc. ACM SIGACT-SIGMOD-SIGART Symp. on Principles of Database Systems, pages 114–125, Santa Barbara, California, May 2001. 8. W. Fan and L. Libkin. On XML integrity constraints in the presence of DTDs. Journal of the ACM, 49(3):368–406, 2002. 9. S. Lu, M. Dong, and F. Fotouhi. The semantic web: Opportunities and challenges for next-generation web applications. International Journal of Information Research, 7(4), 2002. Special Issue on the Semantic Web. 10. S. Lu, Y. Sun, M. Atay, and F. Fotouhi. A new inlining algorithm for mapping XML DTDs to relational schemas. In Proc. of the 1st International Workshop on XML Schema and Data Management (Lecture Notes in Computer Science), Chicago, Illinois, USA, October 2003. To appear. 11. E. Maler and S. DeRose. XML Linking Language (XLink). http://www.w3.org/TR/xlink. 12. E. Maler and S. DeRose. XML Pointer Language (XPointer). http://www.w3.org/TR/xptr. 13. D. Megginson. SAX – The Simple API for XML. http://www.saxproject.org/. 14. J. Shanmugasundaram, K. Tufte, C. Zhang, G. He, D. DeWitt, and J. Naughton. Relational databases for querying XML documents: Limitations and opportunities. The VLDB Journal, pages 302–314, 1999. 15. C. Sperberg-MCQueen and H. Thompson. W3C XML Schema, April 2000. http://www.w3.org/XML/Schema.

Index Selection for Efficient XML Path Expression Processing Zhimao Guo, Zhengchuan Xu, Shuigeng Zhou, Aoying Zhou, and Ming Li Dept. of Computer Science and Engineering, Fudan University, China {zmguo,zcxu,sgzhou,ayzhou,mingli}@fudan.edu.cn

Abstract. One approach to building an efficient XML query processor is to use RDBMSs to store and query XML documents. XML queries contain a number of features that are either hard to translate into SQLs or for which the resulting SQL is complex and inefficient. Among them, path expressions pose a new challenge for efficient XML query processing in RDBMSs. Building index structures for path expressions is necessary. Meanwhile, indexes occupy much disk space. There is a tradeoff between the consumption of disk space and the efficiency of query evaluation. In this paper, we present a cost model for the space consumption of indexes and their benefit to XML queries. Making use of the statistics of XML data and the characteristics of the target application, we adopt greedy algorithm to select some map indexes to be built. Our experimental study demonstrates that query performance get comparatively significant improvement over the case without indexes while only consuming disk space of modest size.

1

Introduction

Due to its flexibility, XML is rapidly emerging as the de facto standard for representing, exchanging and accessing data over the Internet. XML data is an instance of semi-structured data[1]. It comprises hierarchically nested collections of elements. Tags stored with elements in XML data describes the semantics of the data. Thus, XML data, like semi-structured data, is hierarchically structured and self-describing. As Web applications are processing an increasing amount of XML data, there is a growing interest in storing XML data in relational databases so that these applications can use a complete set of data management services (including concurrency control and scalability, etc.) and benefit from the highly optimized relational query processors. Recently storing XML data into relational databases has been extensively studied[2,3,4]. However, how to build an efficient XML query processor is still largely an open problem.

The work was supported by the Hi-Tech Research and Development Program of China under grant No. 2002AA413110. Shuigeng Zhou was also supported by the HiTech Research and Development Program of China under grant No. 2002AA135340 and partially supported by the Open Research Fund Program of State Key Lab of Software Engineering of China under grant No. SKL(4)003.

´ Pastor (Eds.): ER 2003 Workshops, LNCS 2814, pp. 261–272, 2003. M.A. Jeusfeld and O. c Springer-Verlag Berlin Heidelberg 2003

262

Z. Guo et al.

Various languages for querying semi-structured data or XML have been proposed[5,6,7,8]. One of the most important features of these languages is path expressions. XML query languages such as XPath[7] and XQuery[8] use path expressions to traverse XML data. Processing path expressions plays a key role for XML query evaluation. Naive evaluation of path expressions requires to exhaustively search the entire XML document, which is inefficient. Particularly, in relational databases, path traversal needs to be realized by joining relevant tables. It is well-known that the join operations are most costly, which substantially compromise the benefit of using a relational engine for processing XML data. An efficient way to process path expression queries, in order to reduce join costs, is to use indexes. Index technique has been extensively studied in various contexts. The join index was first proposed in relational databases[9]. Several data structures were also proposed to handle path queries in object-oriented databases, e.g., access support relations[10] and join index hierarchy[11]. However, different from relational databases and object-oriented databases, the structure of XML documents is highly flexible, and path expressions in XML query languages are highly expressive, so that the existing index structures are not directly applicable to XML query processing. In so-called native systems, several indexes were proposed: strong DataGuide[12] in Lore system[13], and the T-index proposed in [14]. They can be used to process simple path expressions. However, strong DataGuide and T-index may be very large in size while only offering a little improvement over the naive evaluation of path expressions. In [15], Zheng et al. proposed a new index structure, called structural map, for efficiently processing path expression queries on top of any existing database systems. The structural map consists of two parts—a guide and a set of maps. They present two kinds of maps, 1-way map for evaluating simple path expressions, and n-way map for evaluating regular path expressions. They create maps for all path expressions appearing in the query workload. Structural map can significantly speed up the evaluation of path expression queries. Meanwhile, these maps (referred to as map indexes below) will occupy some disk space. If more map indexes are built, some queries are likely to be processed more efficiently. However, the overhead of disk space consumption will be overwhelming. If only map indexes for part of path expressions are built, then the evaluation of some queries will be slowed down more or less. Thus there is a tradeoff between the consumption of disk space and the efficiency of query evaluation. A group of map indexes chosen to be built is called an index scheme. We know that variant XML data hold varied statistics, and different applications may present a variety of access patterns. For example, a Web site may perform a large volume of simple, localized lookup queries, whereas a catalog printing application may require large and complex queries with deeply nested results[16]. In this paper, we introduce a novel cost-based approach to index selection for efficiently processing XML path expressions. We describe the design and implementation of our approach, that automatically finds a near-optimal index scheme for a specified target application.

Index Selection for Efficient XML Path Expression Processing

263

Fig. 1. The tree structure of an example XML document

The design principles underlying our approach are cost-based search for a near-optimal index scheme and reuse of existing technologies. The first principle is to take the application into consideration. More precisely, a query workload contains a set of queries and an associated weight that reflects the importance of each query for the application. The second principle is to leverage existing relational database technologies whenever possible. We use relational optimizer to obtain cost estimates. Our work is based on that of [15]. We present a cost model for the space consumption of map indexes and their resulting benefit to XML queries by making use of the statistics of XML data and the characteristics of the workload. We adopt greedy algorithm to select a near-optimal set of map indexes to be created, i.e., a near-optimal index scheme. Under certain predefined constraint of disk space, this group of map indexes can bring the most benefit. Our experimental study demonstrates that query performance gets comparatively significant improvement over the case without indexes in the cost of modest overhead of disk space. To our best knowledge, we are the first one to take characteristics of applications and disk space constraint of indexes together into consideration.

2

Structural Map

First we briefly review XML storage model and the work of [15]. An XML document can be represented as a data graph. In the data graph, a node represents an element or attribute. Each node has a unique identifier and is labelled by its tag name. A leaf node is also attached with a string, which represents the text content of an element or attribute. An edge represents a parent-child relationship. For example, Fig. 1 shows a data graph for a fragment of an XML document specified in XML benchmark, XMach-1[17]. The XML storage model has been extensively studied recently[2,3,4]. In our study, for simplicity and without loss of generality, we assume a rather simple DTD-based storage model that maps an element type in DTD into a relation and handles the parent-child relationships as pairs of identifiers. For example, the XML fragment in Fig. 1 is stored in five relations: document, chapter, head, section and paragraph. Fig. 2 shows the relations for document, chapter, section and paragraph. The from and to field of each relation represent the identifiers

264

Z. Guo et al.

Fig. 2. The storage example for the XML document

of the parent and of the node itself which the tuple refers to, respectively. In a relation for an atomic element type, a text field is added to handle the content string. In structural map[15], there are two kinds of map indexes, 1-way-map and n-way-map. Given a label path l1 /l2 / . . . /ln , a map is a set of pairs of identifiers id1 , idn . More specifically, for each label pair (li , lj ) in the guide, if there is only a simple path li /li+1 / . . . /lj , then build a 1-way-map for it; if there are more than one path from li to lj , then build a n-way-map for regular path li //lj . For more detail on structural map, please refer to [15]. Although mapping XQuery statements to SQL statements is an important task of the query evaluation process, it is not the focus of this paper. Hence we omit any further discussion on this issue. We refer the interested readers to recently proposed mapping algorithms from XML query languages to SQL[18, 19]. Next, we examine how to evaluate an XML query with the assistance of map indexes. As an example, the XQuery statement is Qx for $x in /document/chapter[@id=‘c1’]//head return $x. If no map indexes have been built, then the corresponding SQL statement would be something like Q1 1 select h1.text from chapter c1, head h1 where c1.to=h1.from and c1.id=‘c1’ union select h2.text from chapter c2, section s2, head h2 where c2.to=s2.from and s2.to=h2.from and c2.id=‘c1’ union ... However, if the map index mi for the label path chapter//head have been built, the SQL statement would be like Q2 select h.text from mi, chapter c, head h where mi.from = c.to and mi.to = h.to and c.id=‘c1’. It is obvious that this SQL query is more efficient than the former tedious one. Of course, in order to build the map index mi, we have to commit the following SQL statement Qmat to the RDBMS: 1

To avoid cluttering the queries, we have omitted sorting and tagging here.


265

Table 1. Size of map indexes XML Data (M) Database (M) Map Indexes (M) 2 2.1 7.9 4 4.0 16.1 6 6.0 24.0 8 8.1 32.2 10 9.9 40.1

create table mi as select c1.to as from, h1.to as to from chapter c1, head h1 where c1.to=h1.from union select c2.to as from, h2.to as to from chapter c2, section s2, head h2 where c2.to=s2.from and s2.to=h2.from union ... In fact, the idea is much like that of materialization in query optimization. Since the “materialized” map indexes will occupy much disk spaces, we have to tradeoff the overhead of disk space consumption and their benefit to query evaluation. We only choose part of map indexes, then build them, rather than build map indexes for all label pairs. In order to have a more clear understanding of its justification, We next report the result of our preliminary experiments. We evaluated the detail of disk consumption if map indexes were all built. The experimental result is shown in Table 1. We varied the size of the original XML document from 2M to 10M, and measured the disk size required by database and map indexes, respectively. As shown in Table 1, the overhead brought by map indexes is overwhelming. In the case of the XML document of 10M, all of the map indexes require 40M disk space. That is hardly bearable. Therefore, we should carefully choose an optimal group of map indexes to be built.

3

Cost Model

Now we formally state the problem of map indexes selection. First, we model the query workload wkld of the specified application as wkld = {(Qi , wi ), i = 1, 2, · · · n},

(1)

where Qi is an XML query, wi is its weight, reflecting its relative importance within the workload wkld. A high weight means that the query is more frequently requested, or it should be executed with less cost (i.e., holds high priority). With different set of map indexes built, the cost of the workload would be different. The cost of the workload wkld against a particular index scheme S, is

266

Z. Guo et al.

the weighted sum of the cost of different queries in wkld, cost(wkld, S) =

n

wi ∗ cost(Qi , S).

(2)

i=1

Here S is the set of map indexes chosen. If no map index is chosen, S = ∅. The map indexes selection problem can then be defined as follows: Definition 1. Given an XML document D, space constraint Cons on disk size of map indexes, and a workload wkld, determine the optimal index scheme S for this document, so that cost(wkld, S) is minimal. 2 Unfortunately, this problem can be proven to be NP-hard. An index scheme consists of some map indexes. If there are n map indexes which can be built, then each of 2n sets of map indexes will be a candidate of the optimal index scheme. Namely, there are totally 2n states in the entire search space. Therefore, exhaustive search is not practical. Thus our search strategy is based on greedy heuristics. We adopt greedy algorithm to find a near-optimal set of map indexes so that it can bring the most benefit to the evaluation of the specified workload under certain predefined disk space constraint. Virtually, we pay little attention to the exact value of cost(wkld, S). Rather we are more interested in the difference between cost(wkld, S1 ) and cost(wkld, S2 ). If cost(wkld, S1 ) is less than cost(wkld, S2 ), then the index scheme S1 is more beneficial than S2 . The benefit from index scheme S is defined as follows: bf (S) = cost(wkld, ∅) − cost(wkld, S). (3) However, this definition of bf (S) is not conveniently operational. Therefore, we present its operational definition as bf (S) =

n

i wi ∗ bfind (S).

(4)

i=1 i Here, bfind (S) denotes the benefit resulted from S with respect to Qi , individually. It is defined as follows i bfind (S) = db cost(Qjmat ). (5) j Qmat is used by Qi

In this definition, Qmat is the select part of the SQL statements, which are used to build the map indexes. db cost(Q) denotes the cost for evaluating a SQL query Q returned by the RDBMS optimizer. (Most commercial DBMSs provide support for such statistics). Also, let db size(Q) denote the total size of the result of a query Q. DBMSs can also provide this kind of estimates. It is noteworthy to point out that the above definition is approximate. The rationale behind this approximation is that, if the map index corresponding to Qjmat has been built, then we can use them to rewrite Qi , and then during the evaluation process, the cost spent on evaluating the “map index” part could be reduced.


267

Procedure 1 MaxBenefit(G, S) 1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: 15: 16: 17: 18: 19:

for p = 1 to n do for q = 1 to n do if mipq ∈ S then mi A[p, q] = db cost(Qmatpq ); P [p, q] = {mipq } else A[p, q] = 0; P [p, q] = ∅ end if end for end for for r = 1 to n do for p = 1 to n do for q = 1 to n do if (A[p, r] + A[r, q] > A[p, q]) and (p = q) then A[p, q] = A[p, r] + A[r, q]; P [p, q] = P [p, r] + P [r, q] end if end for end for end for return A and P

Even if the set of map indexes has been fixed, how to make use of them to assist query evaluation is still a problem. With respect to a query, we maybe have many options. For example, regarding to path expression l1 /l2 /l3 /l4 /l5 , if map indexes for label path l1 /l2 /l3 and l2 /l3 /l4 have been built respectively, if we select the first one, then we cannot make any use of the latter, while vice versa. Thus it is not easy to make a proper decision between the two choices. In order to tackle this problem, we denote each label li as a vertex in a directed graph G. In the graph G, the edge connecting the vertexes li and lj is labelled ij by a distance dij = db cost(Qij mat ). Here, Qmat is the select part of the SQL statement to build the map index for label path li /lj or li //lj . In order to improve the query evaluation process as much as possible, we wish to find paths between each pair of vertexes of the graph G, which maximize the benefit between the two vertexes. Hence, we devise a procedure MaxBenefit, as shown in Procedure 1, that, given the graph G and the index scheme S, attempts to find these paths. The input of Procedure MaxBenefit consists of 2 arguments in which G is the corresponding directed graph, S is the index scheme which are being examined. Lines 1–6 initialize matrices A and P . A[p, q] represents the maximal benefit already computed to the label path lp / . . . /lq or lp //lq , while P [p, q] indicates the corresponding map indexes which are used in evaluating this label path. Lines 7–11 use dynamic programming technique to calculate the finally maximal benefit and the set of map indexes being used. ij After carrying out this algorithm, we obtain {Qij mat |Qmat is used by Qi } for i each i = 1, 2, · · · , n. Then we can get bfind (S) and bf (S). In order to exploit greedy algorithm, we should distinguish the benefit contributed by an individual map index mi from that by others. However, with respect to varied index

268

Z. Guo et al.

schemes, the benefit contributed by mi may be different. The reason for this phenomenon is that, though mi1 is used by Q1 in the case of an index scheme S1 , it may not be used by Q1 in the case of another index scheme S2 , since S2 contains another more beneficial map index mi2 . In order to measure the contributions of each individual map index mi, we propose the following definition, similar to that of [20]. Definition 2 (Marginal Benefit). mbf (mi, S) is a function for the calculation (measurement) of the marginal benefit contributed by mi, given that all map indexes in S − {mi} are already built, where mi ∈ S. More formally (In terms of mathematical formulae), bf (S) − bf (S − {mi}) if mi ∈ S mbf (mi, S) = (6) bf (S + {mi}) − bf (S) if mi ∈ /S Therefore, we can evaluate the benefit of building a map index, and then, in view of the cost model, develop the optimal index selection algorithm.

4

Search Strategies

Assuming that S is the current group of map indexes to be built, let us examine how to get another index scheme S which is better than S, according to whether the total size of S exceeds the disk space constraint or not. The principles which guide us to make choices are that (a)if the total size of S exceeds the space constraint, then we have to discard the map index which brings little benefit, while occupying much disk space among all of the currently used map indexes; and that (b)if the total size of S is within the predefined space constraint, then we can pick another map index from the remaining candidate ones which can bring much benefit with little space storage overhead; and that (c)each time we changed S, we should recompute the total size of S and the marginal benefit of each individual map index mi, then rearrange them in ascending order by mbf (mi, S)/size(mi). The entire procedure is an iterative one. We will iteratively add or discard map indexes, though only one at a time. Since this problem is NP-hard, our search strategies are based on greedy heuristics. Guided by a certain search strategy, we obtain a sequence of index schemes, S0 , S1 , . . ., and finally get a near-optimal index scheme Sf . In our approach, the selection of initial index scheme S0 is important. It influences significantly the final Sf . Therefore, we experiment with three different initial index schemes to examine its effects: empty, full and random. In the case of empty, the initial set S0 of map indexes is empty, that is to say, we continually insert more map indexes into this set in a greedy fashion until the disk space limit does not allow us to do so. In the second case full, we assume that in the beginning, all map indexes are chosen. Surely their total disk consumption will exceed the space constraint, so we have to discard some poor map indexes gradually. Here, “poor” map indexes means that they can only bring about little benefit while devouring much disk storage. As for the case random, we choose an initial index scheme S0 randomly.


269

If S0 ’s total size goes beyond the predefined space constraint, we discard poor map indexes only one at a time, while if its size is below the space constraint, we add some good map indexes gradually. It is noteworthy to point out that, in these three cases, the sequence of S0 , S1 , · · · , Sf is monotonic. Namely, either S0 ⊂ S1 ⊂ · · · ⊂ Sf or S0 ⊃ S1 ⊃ · · · ⊃ Sf holds. The set S of chosen map indexes grows larger and larger, or shrinks continually. We do not consider this unidirectional search will always yield best solutions. Therefore, we propose a bidirectional search strategy, called backforth strategy. Here, backforth means that we maybe add a map index in this iteration, but after we calculate again the marginal benefit function of map indexes, and rearrange them, probably we will remove a poor map index in the next iteration. In the following iteration, we maybe add or remove a map index depending on whether the total size of the currently examined index scheme S is below or beyond the predefined disk space constraint. However, some delicate problems arise from this back and forth strategy (it is the origin of the name of backforth strategy). Assuming that the current index scheme is Sc , we add a map index mic into Sc . After we rearrange the currently chosen map indexes in ascending order by mbf (mii , Sc )/size(mii ), or in other words, reorganize the heap of Sc , we maybe remove mic from Sc in the next iteration. If this case happens, we will fall into an endless iterative process. It is evident that this sort of adding or removing operations is fruitless at all. Thus we must take this delicate case carefully in our algorithms. It is noted that the selection of the initial index scheme S and the adoption of backforth strategy are orthogonal. That is to say, we can always employ backforth strategy no matter which initial index scheme (empty, full, or random) is being adopted. We design the algorithm Empty for the case of empty S0 , as shown in Procedure 2, which does not employ backforth strategy. Lines 5–10 are the main body of the algorithm Empty, which is based on greedy heuristics. mi is the best one among the remaining map indexes, hence, we pick it and add it into the current index scheme S. This iterative process continues until the disk space consumed by S exceeds the predefined disk space constraint Cons. Next in order to show how to combine different selection policy of initial index scheme with the backforth strategy, we present the algorithm RandomBF, in which the initial index scheme is constructed randomly, and the backforth strategy is being employed. Procedure 3 presents the pseudo code of the algorithm RandomBF. This algorithm naturally introduces some perturbation during the process of searching the optimal solution, which can overcome some weakness of greedy algorithm in a sense. Lines 6–7 show the operations of adding a beneficial map index into the current index scheme S when the disk space constraint allows. Lines 9–10 give those of discarding a poor map index when S’s size is larger than the disk space constraint. The mechanism for avoiding an infinite loop is shown in lines 11 and 12. If a certain index scheme S, which appeared in an earlier iteration, appears once again, the infinite loop should terminate. Otherwise, it will run ceaselessly.

270

Z. Guo et al.

Procedure 2 Empty() 1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13:

Cands ← (the set of all candidate map indexes) Cons ← (disk space constraint) S ← ∅ {S0 is empty} Recalculate(Cands, S) loop find mi ∈ (Cands − S) with maximal mbf (mi, S)/size(mi) if disk size(S + {mi}) ≤ Cons then S ← S + {mi}; Recalculate(Cands, S) else exit loop end if end loop return S

Procedure 3 RandomBF() 1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: 15: 16: 17:

5

... S ← (a random subset of Cands) {S0 is randomly constructed} Recalculate(Cands, S) loop if disk size(S) ≤ Cons then find mi ∈ (Cands − S) with maximal mbf (mi, S)/size(mi) S ← S + {mi} else find mi ∈ S with minimal mbf (mi, S)/size(mi) S ← S − {mi} end if if S already appeared once then exit loop {avoid an infinite loop} end if Recalculate(Cands, S) end loop return S

Performance Analysis

In this section, we present preliminary results on the performance of our index selection algorithms. We have implemented these algorithm in an XML-RDBMS system, VXMLR[21]. And we also conducted experiments to evaluate the effectiveness of our approach. As emphasized in previous sections, one of distinguished aspects of our approach is that we tradeoff disk space overhead of map indexes and their benefit to query processing. The results of our experiments presented in this section support our argument, and demonstrate that judiciously building part of map indexes can still significantly improve query evaluation process. In our experiments, the data sets and the workload used are from the XML benchmark, XMach-1[17]. The data sets are generated following the description of XMach-1, which size ranges from 20M to 100M. The XML data is stored in


271

Fig. 3. Benefits of the optimal map indexes

IBM DB2 UDB7.1, which acts as the back-end repository. The workload consists of eight queries, which cover a wide range of processing features on XML data. Without loss of generality, the weight of each query is set to the same value, e.g., all are 1. Our experiments were conducted on a 1.4G Pentium IV machine with 256M of main memory, running Windows 2000 Server. Fig. 3 depicts the average evaluation time of the eight queries in the workload as a function of the size of the XML data, which ranges from 20M to 100M. The evaluation time without any map index and with the optimal index scheme obtained by our index selection algorithm, is shown in Fig. 3. In this experiment, the disk space constraint of all map indexes is set to 10% of the original size of the databases. As indicated by Fig. 3, although we only build part of map indexes, the query performance still gets considerable improvement over the case without map indexes while these map indexes do not consume too much storage space.

6

Conclusions

In this paper, we have proposed an index selection algorithm for choosing an optimal set of map indexes, given the XML data and the workload. We present a cost model to approximate the benefit of an index scheme, and marginal benefit of each individual map index. Based on greedy heuristics, we tradeoff the overhead from disk space consumption of map indexes and their benefit to the specified workload. Preliminary performance study shows that our approach is effective. Even though we do not build map indexes for all of label paths, query performance still get considerable improvement. Acknowledgments. We would like to thank Shihui Zheng of Fudan University and Wei Wang of Hong Kong University of Science and Technology for so many helpful discussions.

272

Z. Guo et al.

References 1. S. Abiteboul. Querying Semi-Structured Data. In Proc. of ICDT ’97, pages 1–18. 2. D. Florescu and D. Kossmann. A Performance Evaluation of Alternative Mapping Schemes for Storing XML Data in a Relational Database. Technical Report 3680, INRIA, 1999. 3. J. Shanmugasundaram, K. Tufte, C. Zhang, et al. Relational Databases for Querying XML Documents: Limitations and Opportunities. In Proc. of VLDB’99, pages 302–314. 4. F. Tian, D. J. DeWitt, J. Chen, et al. The Design and Performance Evaluation of Alternative XML Storage Strategies. SIGMOD Record Special Issue on Data Management Issues in E-commerce, March 2002. 5. S. Abiteboul, D. Quass, J. Mchugh, et al. The Lore Query Language for Semistructured Data. International Journal on Digital Libraries, 1(1):68–88, April 1997. 6. A. Deutsch, M. Fernandez, D. Florescu, et al. XML-QL: A Query Language for XML. W3C Note, 1998. http://www.w3.org/TR/1998/NOTE-xml-ql-19980819. 7. J. Clark and S. DeRose. XML Path Language (XPath). W3C Recommendation, 1999. http://www.w3.org/TR/xpath. 8. S. Boag, D. Chamberlin, M. F. Fernandez, et al. XQuery 1.0: An XML Query Language. W3C Working Draft, 2002. http://www.w3.org/TR/xquery. 9. P. Valduriez. Join Indices. TODS, 12(2):218–246, 1987. 10. A. Kemper and G. Moerkotte. Access Support in Object Bases. In Proc. of SIGMOD’90, pages 364–374. 11. J. Han, Z. Xie, and Y. Fu. Join Index Hierarchy: An Indexing Structure for Efficient Navigation in Object-Oriented Databases. TKDE, 11(2):321–337, 1999. 12. R. Goldman and J. Widom. DataGuides: Enabling Query Formulation and Optimization in Semistructured Databases. In Proc. of VLDB, pages 436–445, 1997. 13. J. McHugh, S. Abiteboul, R. Goldman, et al. Lore: A Database Management System for Semistructured Data. SIGMOD Record, 26(3):54–66, 1997. 14. T. Milo and D. Suciu. Index Structures for Path Expressions. In Proc. of ICDT, pages 277–295, 1999. 15. S. Zheng, A. Zhou, J. X. Yu, et al. Structural Map: A New Index for Efficient XML Path Expression Processing. In Proc. of WAIM, 2002. 16. P. Bohannon, J. Freire, P. Roy, et al. From XML Schema to Relations: A CostBased Approach to XML Storage. In Proc. of ICDE’02. 17. T. Bohme and E. Rahm. XMach-1: A Benchmark for XML Data Management. In Proc. of GDC, 2001. 18. M. J. Carey, J. Kiernan, J. Shanmugasundaram, et al. XPERANTO: Middleware for publishing object-relational data as XML documents. In Proc. of VLDB, pages 646–648, 2000. 19. M. F. Fernandez, W. C. Tan, and D. Suciu. SilkRoute: Trading between Relations and XML. WWW9/Computer Networks, 33(1-6):723–745, 2000. 20. C. Y. Chang and M. S. Chen. Exploring Aggregate Effect with Weighted Transcoding Graphs for Efficient Cache Replacement in Transcoding Proxies. In Proc. of ICDE, 2002. 21. A. Zhou, H. Lu, S. Zheng, et al. VXMLR: A Visual XML-Relational Database System. In Proc. of VLDB, 2001.

CX-DIFF: A Change Detection Algorithm for XML Content and Change Presentation Issues for WebVigiL* Jyoti Jacob, Alpa Sachde, and Sharma Chakravarthy Computer Science Computer Science and Engineering Department The University of Texas at Arlington, Arlington, TX 76019 {jacob,sachde,sharma}@cse.uta.edu

Abstract. The exponential increase of information on the web has affected the manner in which the information is accessed, disseminated and delivered. The emphasis has shifted from mere viewing of information to efficient retrieval and monitoring of selective changes to information content. Hence, an effective monitoring system for change detection and notification based on user-profile is needed. WebVigiL is a general-purpose, active capability-based information monitoring and notification system, which handles specification, management, and propagation of customized changes as requested by a user. The emphasis of change detection in WebVigiL is to detect customized changes on the content of the document, based on user intent. As XML is an ordered semi-structured language, detecting customized changes to part of the value of the text nodes and even portion of the content spanning multiple text nodes of an ordered XML tree is difficult. In this paper, we propose an algorithm to handle customized change detection to content of XML documents based on userintent. An optimization to the algorithm is presented that has a better performance for XML pages with certain characteristics. We also discuss various change presentation schemes to display the changes computed. We highlight the change detection in the context of WebVigiL and briefly describe the rest of the system.

1 Introduction The Internet is evolving as a repository of information, and the user’s interest has expanded from querying information to monitoring evolution of or changes to pages. The emphasis is on selective change detection, as users are typically interested in changes to a particular portion or section and not the entire page. The need to monitor changes to documents of interest is not only true for the Internet but also for other large heterogeneous repositories. WebVigiL [1-3] provides a powerful way to disseminate information efficiently without sending unnecessary or irrelevant information. The emphasis in WebVigiL is on detecting changes to web pages and notifying the users based on their given profiles. eXtensible Markup Language (XML) is rapidly gaining popularity as the data transportation and exchange * This work was supported, in part, by the Office of Naval Research, the SPAWAR System Center-San Diego & by the Rome Laboratory (grant F30602-01-2-0543), and by NSF (grantsIIS-0123730 and IIS-0097517). M.A. Jeusfeld and Ó. Pastor (Eds.): ER 2003 Workshops, LNCS 2814, pp. 273–284, 2003. © Springer-Verlag Berlin Heidelberg 2003

274

J. Jacob, A. Sachde, and S. Chakravarthy

Fig. 1. WebVigiL Architecture

language. The emphasis in this paper is on selective monitoring of XML content. The self-descriptive characteristic of XML gives us some useful information on the semantics of the document and enables us to detect changes at a finer granularity (e.g., an element or text level) rather than at the document level. But the current user is interested in detecting changes at a finer granularity especially at the text level (e.g. keywords, phrases). Hence, a mechanism is needed, which will monitor customized changes on portions of the XML content. The main contribution of this paper is CXDiff, an approach for detecting customized changes to the content of ordered labeled XML documents. Fig. 1 shows the overall system architecture of WebVigiL. A web based user interface is provided to the user to submit his/her profiles termed as sentinels, indicating the pages to monitor, when to monitor, the types of changes and the methods for presentation and notification. Sentinels are validated for syntactic and semantic correctness and populated in Knowledgebase. Once the sentinels are validated semantically, the change detector module generates the ECA rules for the run time management of that sentinel. The fetch module fetches pages for all active (or enabled) sentinels, forwards them to the version management module for adding them to the page repository and notifies the change detection module. Based on the type of the documents, either the HTML change detection [4] or the XML change detection mechanism is called. The presentation module takes these changes and presents it in a user-friendly manner. The remainder of the paper is organized as follows. Section 2 discusses various tools developed for detecting changes to web pages. Section 3 gives the problem overview. In section 4, we have discussed the change operations and algorithm proposed for change detection to XML documents. Section 5 discusses the various presentation modules and section 6 provides current status and conclusions.

CX-DIFF: A Change Detection Algorithm for XML Content

275

2 Related Work Work in change detection to flat-files [5] and detecting differences between two strings [6, 7] in terms of inserts and deletes has been well established. WordPerfect has a “mark changes” facility that can detect changes based on how documents are compared (on either a word, phrase, sentence, or paragraph basis). Due to the semi-structured nature of XML, it can be conveniently represented in a tree structure. Many algorithms have been proposed for tree-tree comparison taking some tree features into consideration [8-10]. Chawathe et al [10] proposed an algorithm for hierarchical structured data wherein minimum costs edit script is generated which transforms tree T1 to T2. This algorithm works for semi-structured documents such as latex. But the assumptions made for latex do not hold for XML documents as they contain duplicate nodes and sub-trees. X-diff [11] detects changes on parsed unordered labeled tree of XML. X-diff finds the equivalent second level sub-trees and compares the nodes using the structural information denoted as signature. In order to detect move operations i.e., if a node is moved from position i in the old tree to position j in the new tree, an unordered tree cannot be considered. In [12], the authors formulated a change detection algorithm called Xydiff to detect changes between given two ordered XML trees T1 and T2. XMLTreeDiff [13], a tool developed by IBM, is a set of JavaBeans and does ordered tree to tree comparison to detect changes between XML documents. DeltaXML [14, 15] developed by Mosnell provides a plug-in solution for detecting and displaying changes to content between two versions of an XML document. They represent changes in a merged delta file by adding additional attributes such as insert/delete to the original XML document. The Diff and Merge tool [16] provided by IBM compares two XML files based on node identification. It represents the differences between the base and the modified XML files using a tree display of the combination in the left-hand, Merged View pane with symbols and colors to highlight the differences using the XPath syntax. DOMMITT [17] is a UNIX diff utility tool that enables the users to view differences between the DOM [18] representations of two XML documents. The Diff algorithms on these DOM representations produces edit scripts, which are merged into the first document to produce an XML document in which the edit operations are represented as insert/delete tags for a user interactive display of differences. Most of the algorithms in the literature detect changes to structure as well as content. WebVigiL handles change management only on the content of XML documents. Detecting changes to the structure in addition to the context would be an overhead. In addition, most of the change detection tools for XML do not support customized changes to the nodes (i.e., change to part of a node or spanning multiple nodes) and hence these algorithms cannot be mapped directly to satisfy the monitoring requirements of WebVigiL.

276

J. Jacob, A. Sachde, and S. Chakravarthy

Fig. 2. XML Document

Fig. 3. Ordered Labeled XML tree

3 Customized Change Detection Issues As XML was defined for semi-structured document containing ordered elements [19, 20], such documents can be mapped into an ordered labeled tree. The ordered tree for the XML document in Fig. 2 is shown in Fig. 3. We consider the tree representation of the XML document similar to the Document Object Model (DOM) [18] representation. In this paper, the nodes of the XML tree will be referenced using the defined label of a DOM Node. In XML, the context (elements nodes) defines the content of the document. In the tree structure for an XML document, a leaf node represents the content of the page for a particular context. Hence, changes are detected to the text nodes and attributes nodes, which constitute the leaf nodes. Change detection for semi-structured, ordered XML tree is complex because of the following issues: 1. WebVigiL supports customized change detection to the contents, such as phrase and keyword change. Keywords and phrases can be part of the node or can span multiple nodes. Hence the algorithm should be capable of extracting the required content of interest and detect changes. 2. Change detection for semi-structured, ordered XML tree is complex on account of duplicate nodes and sub-trees. By duplicate nodes, we mean similar leaf nodes containing the same context. As shown in Fig. 3, the node ‘J K Rowling’ appears twice in the tree for the same context (path) i.e. ‘Books-Section-Book-Author’. Duplicate sub-trees defined for the same context are also possible in XML. Order becomes very critical for such duplicate nodes as a node n, existing at position pi in th the old tree should be compared to the node existing in the equivalent i position in the new tree with respect to their siblings. 3. For an XML tree T1 rooted at R with children pi to pm, a node along with its structural information can be moved from j where i ≤ j ≤ m in T1 to position k in T2 where j k, when considered with respect to the siblings. The change mechanism developed should be capable of detecting such move operations. An algorithm CX-Diff is proposed, taking into consideration an ordered, labeled XML tree and the position of occurrence of the node with respect to its sibling.

CX-DIFF: A Change Detection Algorithm for XML Content

277

Fig. 4. Change Operations on trees T1 and T2

4 CX-Diff: Customized Change Detection for Ordered Documents Given two ordered XML trees T1 and T2, consider the change operations from the set E = {insert, delete, move} which when applied to T1 transforms it into a new tree T2. To detect the change operations, the structure is also taken into consideration. The content of a leaf node is defined as its value and is denoted as v(x) where x is a leaf node. The operations can be defined as follows: th Insert: Insertion of a new leaf node at the i position is denoted as insert. Insert of a th keyword is defined as the appearance of a keyword k in the i leaf node x of the tree T1. Insert of a phrase is defined as the appearance of a complete phrase at position i in tree T1, denoted by (p, i). As structure defines the context for the content in XML, a node of the same value but different ancestral elements is considered inserted. Delete: Given two ordered XML trees T1 and T2, T1 will be same as T2 except that it will not contain leaf node x. Delete of a keyword is defined as the disappearance of th the keyword k in the i leaf node x of the tree T1. Phrase delete is defined as th disappearance of a phrase p at i position in the tree T1, denoted by (p,i). Move: For the tree T1, containing leaf nodes from n1 to nm, a leaf node x containing signature s is shifted from position j in T1 to position k in the new tree T2 where 1
Fig. 5. XML query-processing system for $p in document("paper.xml")//paper return <paper> <papertitle>$p/papertitle { for $a in $p/authors/author return $a/name } for $b in $p/reference/book return $b/booktitle { for $ba in $b/author return

$ba/name } for $pp in $p/reference/paper return <paper> <papertitle>$pp/papertitle { for $ppa in $pp/authors/author return $ppa/name }

Fig. 6. A query written in Xquery

292

B.-J. Shin and M. Jin

0 1 2 3 4 5 6 7 8 FOR <paper> <papertitle> Data(1,papertitle) FOR 9 10 11 12 13 14 15 16 17 Data(2,name) ENDFOR(6) FOR : 41 42 43 44 45 46 47 48 49 ENDFOR(37) FOREND(31) FOREND(0)

Fig. 7. Structure information for publishing XML documents

4.1.2 Structure Information for Publishing XML Documents The XQuery Parser extracts the structure information as well as the relational schema information of the desired XML document. Here, we focus on publishing the desired XML data written in FLWOR expressions of XQuery. The structure information is passed to the XML Generator to be used in publishing the desired XML document. The structure information extracted from the XQuery query given in Figure 6 is shown in Figure 7. 4.2 SQL Generator To extract the desired XML data written in XQuery queries from relational databases, XQuery queries should be translated into the corresponding SQL statements. SQL Generator generates SQL queries for creating views using the extracted schema information. First, the schema information such as the tables, columns, and hierarchical structure of tables should be drawn from the path expressions. The following SQL queries get the schema information. The drawn schema information is shown in Figure 8. The hierarchical structure of tables is also described in the figure. The hierarchical structure of tables is represented through the parentID and parentCode attributes. ParentCode is introduced to accommodate multiple parent tables and recursive structures. SELECT table, column, parentCode FROM Path WHERE pathExp LIKE “#%/paper#/papertitle”

(1)

SELECT table, column, parentCode FROM Path WHERE pathExp LIKE “#%/paper#/authors#/author#/name”

(2)

Now, SQL Generator generates SQL queries for extracting the desired XML data using the drawn schema information. The following are some SQL queries necessary for extracting the desired XML data of the query in Figure 6. CREATE VIEW Paper_View( paperID, papertitle ) AS SELECT paperID, papertitle FROM Paper

(3)

CREATE VIEW Author_View( paperID, authorID, name ) AS SELECT P.paperID, A.authorID, A.name FROM Paper_View P, Author A WHERE P.paperID = A.parentID AND A.parentCode = 1

(4)

Storing and Querying XML Documents

293

Note that each view has the identifier of the tuple and its parents if they exist. We use the Node Outer Union technique[7] that has been known to be one of the most efficient strategies for materializing relational data. Figure 9 shows the query execution plan for the Node Outer Union technique for our running example. /HYHO

3DSHU

3DUHQW&RGH

3DUHQW&RGH

3DSHU,G 3DUHQW,G 3DUHQW&RGH 2UGHU 3DSHU7LWOH 'RF,G

$XWKRU

3DSHU

$XWKRU,G 3DUHQW,G 3DUHQW&RGH 2UGHU 1DPH &RXQWU\ 8QLYHUVLW\ 'RF,G

/HYHO

3DSHU,G 3DUHQW,G 3DUHQW&RGH 2UGHU 3DSHU7LWOH 'RF,G

%RRN %RRN,G 3DUHQW,G 2UGHU %RRN7LWOH 'RF,G

$XWKRU 3DUHQW&RGH

/HYHO

$XWKRU,G 3DUHQW,G 3DUHQW&RGH 2UGHU 1DPH &RXQWU\ 8QLYHUVLW\ 'RF,G

3DUHQW&RGH

Fig. 8. Extracted schema information

7\SH3DSHU,'3DSHU7LWOH$XWKRU,'1DPH%RRN,'%RRN7LWOH3DSHUB3DSHU,'

2XWHU8QLRQ

3DSHU,'3DSHUB3DSHU,'$XWKRU,'1DPH

3DSHU,'%RRN,'$XWKRU,'1DPH

3393DSHUB3DSHU' $3DUHQW,'$1'$3DUHQW&RGH

$3DUHQW,' %9%RRN,'$1'$3DUHQW&RGH

3DSHUB3DSHUB$XWKRUB9LHZ -RLQ

%RRNB$XWKRUB9LHZ -RLQ

3DSHU,'3DSHUB3DSHU,'3DSHU7LWOH

3DSHU,'$XWKRU,'1DPH

3DSHU,'%RRN,'%RRN7LWOH

33DUHQW,' 393DSHU,'$1'33DUHQW&RGH

393DSHU,' $3DUHQW,'$1'$3DUHQW&RGH

393DSHU,' %3DUHQW,'

3DSHUB3DSHUB9LHZ339 -RLQ

$XWKRUB9LHZ -RLQ

%RRNB9LHZ%9 -RLQ

3DSHU,'3DSHU7LWOH

3DSHUB9LHZ39 $XWKRU$

3DSHU3

%RRN%

Fig. 9. SQL query execution for extracting the desired XML data

4.3 XML Constructor The desired XML data is extracted by executing SQL queries generated by SQL Generator in the relational database. The result data is represented in flat structures. Hence, we have to put the result in a hierarchical structure in order to publish it as an XML document. We use the Sorted Outer Union technique[7]. In the Sorted Outer

294

B.-J. Shin and M. Jin

Union technique, the key to structuring the relational data is to order it the way that it needs to appear in the resulting XML document. Thus, in our running example, sorting the result of node outer union by the sort sequence (paperID, paper_paperID, bookID, authorID) will ensure that the final result is in the desired document order. Figure 10 shows the result tuples of executing the corresponding SQL queries in the Sorted Outer Union technique. The type column in Figure 10 is added to the result of the Sorted Outer Union to indicate the corresponding view that was defined by the SQL Generator. This is useful in tagging to be performed by the XML Generator. type 1 2 2 3 6 4 5 1 2

paperID papertitle authorID name bookID booktitle paper_paperID 1 XML Query Proc... NULL NULL NULL NULL NULL 1 NULL 1 B.Shin NULL NULL NULL 1 NULL 2 M.Jin NULL NULL NULL 1 NULL NULL NULL 1 Professional XML… NULL 1 NULL 4 K.Williams 1 NULL NULL 1 Efficiently pubis… NULL NULL NULL NULL 2 1 NULL 3 J.Shanmugasundaram NULL NULL 2 2 Efficiently Publis… NULL NULL NULL NULL NULL 2 NULL 3 J.Shanmugasundaram NULL NULL NULL

Fig. 10. Resulting tuples XMLGenerator( list_pointer XQuery, recordset record_set ) { boolen inner_for = false; list_morpheme w; int front = -1, rear = -1; for( i = 0; i < MAX_MORPHEME; i++ ) { w = XQuery[i]; if( w->*morpheme == "FOR" || w->*morpheme == "LET" ) { if( inner_for = true && front != rear ) queue_output( &front, rear ); else inner_for = true; } else if( w->*morpheme == "DATA" ) { if( w->tuple_type == record_set("type") ) { if( front != rear ) queue_output( &front, rear ); printf("%s", record_set(*att_name)); } else { record_set.movenext; if( w->tuple_type == record_set("type") ) { if( front != rear ) queue_output( &front, rear ); printf("%s", record_set(*att_name)); } else { i = break_for( i ); front = rear;

} } } else if( w->morpheme == "FOREND" || w->morpheme == "LETEND" ) i = w->start_for - 1; else { if( inner_for = false ) printf("%s", w>morpheme); else { rear = rear + 1; queue[rear] = *morpheme; } } } } void queue_output( int *front, int rear ) { while( *front == rear ) { *front = *front + 1; printf("%s", queue[*front]; } } int break_for( int i ) { while( XQuery[i]->morpheme != "FOREND" && XQuery[i]->morpheme != "LETEND" ) i++; return i; }

Fig. 11. Algorithm to generate the final XML document

4.4 XML Generator XML Generator generates the final XML document by using the structure information and the resulting tuples. Figure 11 shows the algorithm to tag and generate the final XML document. As shown in the figure, the algorithm takes the extracted structure

Storing and Querying XML Documents

295

information and the resulting tuples as inputs, and publishes the final XML document. It generates a tagged XML document traversing the extracted structure information that is like the one shown in Figure 7. While traversing the structure information, appropriate values from the resulting tuples are chosen and inserted in the final XML document.

5 Conclusion In this paper, we have proposed a method for storing and querying XML data using relational databases. We proposed an association lining that extends shared inlining and hybrid inlining to reduce relational fragments and excessive joins. We aim to cope with the problems originated due to the discrepancy between the hierarchical structure of XML and the flat structure of relational databases. Additionally, we stored the structure information of XML data in the Path table, which will be used in publishing the desired XML documents. We developed a technique to translate XML queries written in XQuery into SQL statements by exploiting extracted schema information drawn from the Path table. The information on the structure of XML data and resulting tuples produced by executing corresponding SQL queries is exploited in generating the desired XML documents represented in queries written in XQuery FLWOR expressions. Thus, the desired XML documents are published simply. The efficiency of our association inlining technique is to be further verified in terms of relational fragments and the number of joins. The simplicity of our technique for generating XML documents using schema information based on path expressions and structure information is to be evaluated against reasonable datasets.

References 1. 2. 3. 4. 5. 6. 7.

Carey, D., Florescu, D., Ives, Z., Lu, Y., Shanmugasundaram, J., Shekita, E., Subramanion, S.: XPREANTO: Publishing Object-Relational Data as XML. Informal Proceedings of the International Workshop on the Web and Databases (2000) 105–110 Deutsch, A., Fernandez, M., Suciu, D.: Storing Semi-Structured Data with STORED. Proceedings of ACM SIGMOD Conference on Management of Data (1999) 431–442 Fernandez, M., Tan, W., Suciu, D.: SilkRoute: Trading Between Relations and XML. th Proceedings of the 9 W3C Conference (2000) 723–745 Florescu, D., Kossmann, D.: Storing and Querying XML Data Using an RDBMS. IEEE Data Engineering Bulletin, Vol. 22, No. 3. (1999) 27–34 Funderburk, J.E., Kiernan, G., Shanmugasundaram, J., Shekita, E., Wei, C.: XTABLES: Bridging Relational Technology and XML. IBM Systems Journal (2002) 616–641 Shanmugasundaram, J., Kiernan, J., Shekita, E., Fan, C., Funderburk, J.: Querying XML th Views of Relational Data. Proceedings of the 27 VLDB Conference (2001) 261–270 Shanmugasundaram, J., Shekita, E., Barr, R., Carey, M., Lindsay, B., Pirahesh, H., Reinwald, B.: Efficiently Publishing Relational Data as XML Documents. Proceedings of th the 26 VLDB Conference (2000) 65–76

296 8. 9. 10. 11. 12. 13.

B.-J. Shin and M. Jin Shanmugasundaram, J., Shekita, E., Kiernan, J., Krishnamurthy, R., Viglas, E., Naughton, J., Tatarinov, I.: A General Technique for Querying XML Documents Using a Relational Database System. SIGMOD Record, Vol. 30, No. 3. (2001) 20–26 Shanmugasundaram, J., Tufte, K., He, G., Zhang, C., Dewitt, D., Naughton, J.: Relational Databases for Querying XML Documents: Limitations and Opportunities. Proceedings of th the 25 VLDB Conference (1999) 302–314 Williams, M., Brundage, M., Dengler, P., Gabriel, J., Hoskinson, A., Kay, M., Maxwell, T., Ochoa, M., Papa, J., Vanmane, M.: Professional XML Databases. Wrox Press (2000) Yoshikawa, M., Amagasa, T.: XRel: A Path-Based Approach to Storage and Retrieval of XML Documents Using Relational Databases. ACM Transactions on Internet Technology, Vol. 1, No. 1. (2001) 110–141 W3C Recommendation. XML Path Language (XPath) Version 1.0. In http://www.w3c.org/TR/xpath/ (1999) W3C Recommendation. XQuery 1.0: An XML Query Language. In http://www.w3c.org/TR/xquery/ (2002)

Improving Query Performance Using Materialized XML Views: A Learning-Based Approach Ashish Shah and Rada Chirkova Department of Computer Science North Carolina State University Campus Box 7535, Raleigh NC 27695-7535 {anshah,rychirko}@ncsu.edu

Abstract. We consider the problem of improving the efficiency of query processing on an XML interface of a relational database, for predefined query workloads. The main contribution of this paper is to show that selective materialization of data as XML views reduces query-execution costs in relatively static databases. Our learning-based approach precomputes and stores (materializes) parts of the answers to the workload queries as clustered XML views. In addition, the data in the materialized XML clusters are periodically incrementally refreshed and rearranged, to respond to the changes in the query workload. Our experiments show that the approach can significantly reduce processing costs for frequent and important queries on relational databases with XML interfaces.

1 Introduction The extended markup language (XML) [18] is a simple and flexible format that is playing an increasingly important role in publishing and querying data in the World Wide Web. As XML has become a de facto standard for business data exchange, it is imperative for businesses to make their existing data available in XML for their partners. At the same time, most business data are still stored in relational databases. A general way to publish XML data in relational databases is to provide XML interfaces over the stored relations and to enable querying the interfaces using XML query languages. In response to the demand for such frameworks, database systems with XML interfaces over non-XML data are increasingly available, notably relational systems from Oracle, IBM, and Microsoft. In this paper we consider the problem of improving the efficiency of evaluating XML queries on relational databases with XML interfaces. When querying a data source using its XML interface, an application issues a query in an XML query language and expects an answer in XML. If the data source is a relational database, this way of interacting with the database adds new dimensions to the old problem of efficiently evaluating queries on relational data. In the standard scheme for evaluating queries on an XML interface of a relational database, the relational query-processing engine computes a relation that is an answer to the query on the stored relational data; see [9] for an overview. On top of this process, the query-processing engine has to (1) translate the query from an XML query language into SQL (the resulting query is then M.A. Jeusfeld and Ó. Pastor (Eds.): ER 2003 Workshops, LNCS 2814, pp. 297–310, 2003. © Springer-Verlag Berlin Heidelberg 2003

298

A. Shah and R. Chirkova

posed on the relational data), and (2) translate the answer into XML. To efficiently process a query on an XML interface of a relational database, the query-processing engine has to efficiently perform all three tasks. We propose an approach to reducing the amount of time the query-processing engine spends on answering queries on XML interfaces of relational databases. The idea of our approach is to circumvent the standard query-answering scheme described above, by precomputing and storing, or materializing, some of the relational data as XML views. If the DBMS has chosen the “right” data to materialize, it can use these XML views to answer some or most of the frequent and important queries on the data source without accessing the relational data. We show that our approach can significantly reduce the time to process frequent and important queries on relational databases with XML interfaces. Our approach is not the first view-based approach to the problem of efficiently computing XML data on relational databases. To clarify how our approach differs from previous work, we use the terms (1) view definitions, which are data specifications given in terms of stored data (or possibly in terms of other views), and (2) view answers, which are the data that satisfy the definition of a view on the database. In past work, researchers have looked into the problem of efficiently evaluating XML queries over XML view definitions of relational data (e.g., SilkRoute [8] or XPERANTO [16]). We build on the past work by adding a new component to this framework: We incrementally materialize XML view answers to frequent and important XML queries on a relational database, using a learning approach. To the best of our knowledge, we are the first to propose this approach. The following are the contributions of this paper: • We develop a learning-based approach to materializing relational data in XML. • We propose a system architecture that takes advantage of the materialized XML to reduce the total query-execution times for incoming query workloads. • We show how to transform a purely relational database system to accommodate materialized XML and our system architecture. Using our approach may result in significant efficiency gains on relatively static databases. Moreover, it is possible to combine our solution with the orthogonal approaches described in [8,16], thus achieving the combined advantages of the two solutions. The remainder of the paper is organized as follows. Section 1.1 discusses related work. In Section 2 we formalize the problem and outline our approach. In Sections 3 and 4, we describe the system architecture and the learning algorithm. Section 5 describes experimental results. We discuss the approach in Section 6, and conclude with Section 7. 1.1 Related Work The problem of XML query answering has recently received a lot of attention. [11, 13] propose a logical foundation for the XML data model. [3] describes a system for data-model management, with tools to map schemas between XML and relations. [6]

Improving Query Performance Using Materialized XML Views

299

looks into developing XML documents in a normal form that guarantees some desirable properties of the document format. [7] proposes an approach to efficiently representing and querying semistructured Web data. [10] proposes an XML data model and a formal process, to map Web information sources into commonly perceived logical models; the approach provides for easy and efficient information extraction from the World-Wide Web. [14] describes an approach to XML data integration, based on an object-oriented data model. [15] proposes an XML data-management system that integrates relational DBMS, Java and XSLT. [20] reports on a system that manages XML data based on a flexible mapping strategy; given XML data, the system stores data in relations, for efficient querying and manipulation. XCache [2] describes a web-based XMLquerying system that supports semantic caching; ACE-XQ [4] is a caching system for queries in XQuery; the system uses sophisticated cache-management mechanisms in the XML context. SilkRoute [8] is a framework for publishing relational data using XML view definitions. The approach incorporates an algorithm for translating queries from XQuery into SQL and an optimization algorithm for selecting an efficient evaluation plan for the SQL queries. XPERANTO [16] is an XML-centric middleware layer that lets users query and structure the contents of a relational database as XML data and thus allows them to ignore the underlying relations. Using the XPERANTO query facility and the default XML view definition of the underlying database, it is possible to specify custom XML view definitions that better suit the needs of the applications. The motivation for using views in query processing comes from informationintegration applications; one approach, called data warehousing [17], uses materialized views. [1,5,19,21] propose a unified approach to the problem of view maintenance in data warehouses. In our work, we use a learning method called concept, or rule, learning [12].

2 Problem Specification and Outline of the Proposed Approach In this section we specify the problem of improving the efficiency of answering queries on XML interfaces of relational databases, and outline our solution. An XMLrelational data source (“data source”) comprises a relational database system and an XML interface. For a query in an XML query language, to evaluate the query on a data source means to obtain an XML answer to the query via the XML interface of the source. Suppose there is a finite set of important queries, with associated relative weights, that users or applications frequently pose on the data source. We call these queries a query workload. In our cost model, the cost of evaluating a query on a data source is the total time elapsed between posing the query on the source and obtaining an answer to the query in XML. The total cost of evaluating a query workload on a data source is the weighted sum of the costs of evaluating all workload queries, using their relative weights. We consider the problem of improving the efficiency of evaluating a query workload on a data source; the goal here is to reduce the total cost of evaluating a given query workload on a given data source. To improve the efficiency of evaluating a query workload on a data source, we propose an approach based on incrementally materializing XML views of workload-

300


relevant data. To materialize a view is to compute and store the answer to the view on the database. We materialize views in XML rather than in relations, to reduce or eliminate the time required to translate (1) the workload queries from an XML query language into SQL, and (2) the relational answers to the queries into XML. In the proposed system architecture, when answering a query, the query-processing engine first searches the materialized XML views, rather than the relational tables; if the query can be answered using the views, there is no need to access the underlying relations. Using this approach may result in significant efficiency gains when the underlying relational data do not change very often. In our approach, we need to decide which data to materialize in XML. We use a learning-based approach to materialize only the data that is needed to answer the workload queries on the data source. In database systems, it is common to maintain statistics on the stored data, for the purposes of query optimization [9]. We maintain similar statistics on access rates to the data in the stored relations, and materialize the most frequently accessed tuples in XML. We use learning techniques combined with the access-rate statistics to decide when and how to change, incrementally, the set of records materialized in XML. We manage the materialized data using the concept of clustering. In our approach, clustering means combining related XML records into a single materialized XML structure. These XML structures are stored in a special relation and can be queried using the data source’s XML query language. (In the remainder of the paper we assume that XQuery is the language of choice.) Storing the most frequently accessed tuples in materialized XML clusters increases the probability that future workload queries will be satisfied by the clusters. To answer those queries that are not satisfied by the XML clusters, we use the relational query-processing engine.

3 The System Architecture We now discuss the architecture of the system. We describe the query-processing subsystem, the required changes to the schema of the originally relational data source, and the process of generating workload-related XML data from the stored relations. 3.1 The Query-Processing Subsystem In this section we describe a typical query path taken by an input query; see Fig. 1. The solid lines in Fig. 1 show the primary query path, which is taken for all queries on the data. If a workload query can be answered by the materialized XML clusters, then only the primary path is taken. Otherwise, the query next follows the secondary query path, shown in dotted lines in Fig. 1; here, the input query is pushed down to the relational level and is answered using the stored relations, rather than the materialized XML. The XML clusters are stored as values of an attribute in a special relation. The system queries the relation in SQL to find the most relevant cluster, and then poses the XQuery query on the cluster. The schema for the clusters is specified by the database administrator.


301

Fig. 1. The Query-Processing subsystem

3.2

Setting Up Materialized XML Clusters

In this section we describe how to set up materialized XML clusters, by transforming the relational-database schema to accommodate XML. For simplicity, we use a schema with just two relations, R(A1,…, An) and S(B1,…, Bm). A1 is the primary key of the relation R. 3.2.1 Modifying the Given Relational Schemas In our approach, for tuples of certain relations we keep track of how many times each tuple is accessed in answering the workload queries. To enable these access counts, we change the schema of the relational data source, by adding an extra attribute to the schema of one or more of the stored relations. The most likely candidates for this schema change are the relations of interest, which are relations that have high access rates, primarily large relations that are involved in expensive joins. For instance, suppose we have a query that involves a join of the relations R and S. If the relation R is large, the query would be expensive to evaluate, hence we consider R as a suitable candidate for the schema change. (Alternatively, the database administrator can make the choice of the schema to modify.) Suppose we decide to add an attribute A(n+1) to the schema of the relation R; we will store access counts for the tuples in relation R as values of this attribute. R(A1,…,An, A(n+1)) is the schema of the modified relation. Initially, the value of A(n+1) is NULL in all tuples. 3.2.2 Creating the Relations for the Materialized XML Clusters We now define the schema of the relation T that will store the materialized XML clusters, as T(A1, C). Recall that A1 is the primary key of the relation R; using this attribute in the relation T helps us index the materialized XML clusters in the same way as the relation R. The attribute C is used to store the materialized XML clusters in text format. To summarize, we set up materialized XML clusters by doing the following: 1. Select a relation of interest (R in the example) to modify.

302


2. Add an access-count attribute to the schema of the selected relation. 3. Create a new relation (T in the example), to hold the materialized XML version of the data in the selected relation of interest (R in the example).

4 The Learning Algorithm In this section we describe a learning algorithm that populates and incrementally maintains the XML clusters. We first describe how to select relational tuples for materialization, and then explain our clustering strategy for building an XML tree of “interesting records.” Our general approach is as follows. When answering queries, we first pose each query on the materialized XML clusters in the relation T that we have added to the original stored relations. Whenever a query cannot be answered using the materialized XML clusters (or at system startup, see next paragraph), the query is translated into SQL and pushed down to the stored relations. Each time this process is activated, the system increments access counts for all tuples that contribute to the answer to the SQL query. At system startup, the relation T that holds the materialized XML is empty. As a result, all incoming queries have to be translated into SQL and pushed to the relational query-processing engine. The materialization phase starts when the access counts in the relations of interest exceed an empirically determined threshold value, see Section 4.3; all tuples whose access counts are greater than the threshold value are materialized into XML. The schema for the materialized XML is specified by the input XQuery workload. (Alternatively, it can be specified by the database administrator.) As the learning algorithm executes over an extended time period, the most frequently accessed tuples in the relations of interest are materialized into XML and stored in the relation T. 4.1 Learning I: Discovering Access Patterns in the Relations of Interest To incrementally materialize and maintain XML clusters of workload-relevant data, the system periodically runs a learning process that translates frequently accessed relational tuples into XML and reorders the resulting records in a hierarchy of clusters. We now describe the first stage of the learning process, where the system discovers access patterns in the relations of interest by using the access-count attribute. Once the access pattern is established, the system translates the most frequently accessed tuples into XML. To obtain the current access pattern, the system needs to execute the following steps. 1. (This step is executed during the system startup.) Input an expected query stream and set up the desired output XML schema. 2. Pose the incoming workload queries on the stored relations; in answering the queries, increment the access counts for those tuples in the relations of interest that contribute to the answers to the queries. During the system startup we use an expected, rather than real, query stream to determine access patterns in the relations of interest. For example, if each workload


303

query may use one of the given 250K keywords with given frequencies, then for our expected query stream we select the 1000 most-frequent keywords. 4.2 Learning II: Materializing XML and Forming Clusters Once the first stage of the learning process has discovered the access patterns in the relations of interest, the system performs, in several iterations, the following steps: 1. To generate the materialized XML records, retrieve from the relations of interest all tuples whose access counts are greater than the predefined threshold value. 2. Translate the data into XML and store in the materialized XML relation. 3. Form clusters (also see section 4.3): a. Find all relational tuples that are related to the materialized XML, w.r.t. the workload queries. b. Select those of the tuples whose access counts exceed the threshold value, and translate them into XML. c. Cluster the tuples and materialized XML into a single XML tree. 4.3 The Clustering Phase In our selective materialization, we use clustering to increase the scope of materialized XML beyond the relations of interest, by incrementally adding to the XML records “interesting records” from other relations. The criterion for adding these interesting records is the same as the criterion for materializing relational tuples in XML. More precisely, the relations with the most frequently accessed records are selected in the descending order of access frequency. For example, if there are three relations R1, R2, R3, in descending order of tuple-access frequencies, then we can form clusters, starting with R1 and R2, then R2 and R3, and so on. The relation T now contains a single XML structure, which holds related records with high access rates. In each cluster, the records are sorted in the order of their access counts. In the current implementation, the schema for the cluster is provided as an external input (see Fig. 1). Choosing cluster schemas automatically is a direction of future work. We now explain on an example how to form hierarchies of clusters. Consider a database with four relations, R1-R4, in the descending order of tuple-access frequency. We first modify the relation R1, to store the XML clusters generated from the tuples retrieved from a join of R1 and R2 on some attribute. Similarly, we modify R2 to store a join of R2 and R3, and so on. With every join of Rn and Rn+1, we form the mostfrequently accessed clusters; the clusters form a hierarchy w.r.t. their access rates: For example, the cluster formed from R1 and R2 will have higher access rates than the cluster for R2 and R3. In our experiments, we have explored the first level of clustering for simple queries; see Section 5. We are working on implementing multiple levels of clustering for more complex queries. In our approach we determine the threshold value empirically: At system startup time, we repeat the learning process several times to arrive at a suitable value. The choice of the threshold value is a tradeoff between larger materialized views and better query-execution times: A lower threshold value means more tuples will be

304


materialized as XML; thus more queries will get satisfied in the XML views. A higher threshold value prevents most of the relational data from being selected for materialization, which limits the number of queries that can be answered using the views. The key is to strike a balance between the point at which the system materializes tuples and the proportion of records to be materialized. In our future work, we intend to make the choice of this threshold value dynamic.

5 Experimental Setup and Results 5.1 The Setup The CDDB collection [22] is a database that stores information about CDs and CD tracks. The CDDB schema comprises two relations, Disc(cd_id,cd_title,genre,num_of_tracks) and Tracks(cd_id,track_title). (For simplicity, we omit other attributes of the relations in CDDB.) The Disc relation has 250K tuples. Each CD has an average of 10 tracks, stored in the Tracks relation. Fig. 2 shows some tuples in the two relations in CDDB. In our experiments, we used Oracle 9.2 on a Dell Server P4600 with Intel Xeon CPU at 2GHz and 2GB of memory running on Microsoft Windows 2000. We implemented the middleware interface in Java using Sun JDK 1.4, and ran it on an Intel Pentium II 333MHz machine with 128MB of memory on Red Hat Linux 7.3. We conducted a significant number of runs to ensure that the effect of network delays on our experiments is minimal.

Fig. 2. Some tuples in relations Disc and Track in the CDDB database

Fig. 3. Data in the Disc relation with the modified schema

Fig. 4. Schema for the relation that holds the materialized XML and an example of a simple cluster


305

To determine access patterns for the Disc relation, we added a new attribute, count, to the schema; this attribute holds an access count for each CD record. The rest of the database schema is unchanged. (Section 3.2.1 explains how to choose relations for the schema change.) Fig. 3 shows the tuples in the Disc relation with the modified schema. Fig. 4 shows the table XmlDiscTrack. This new relation holds materialized XML as text data in record format. The process of defining this materialized table is explained in Section 3.2.2. In the XmlDiscTrack relation that we create in the CDDB database, attributes cd_id and count are the same as in the Disc relation. The value of the count attribute in XmlDiscTrack equals the value of count in the corresponding tuple in the Disc relation, at the point in time where that tuple was materialized as XML. The XML attribute in XMLDiscTrack holds the materialized XML. For example, the value of the XML attribute, in the tuple for the ‘Air Supply’ CD in XMLDiscTrack, is shown in Fig. 4. Workload queries: The workload queries in our experiments use CD titles as keywords. In our architecture, the query-processing engine first tries to answer each workload query by searching the XML clusters in the relation XMLDiscTrack; if it fails to find an answer there, the engine then searches the Disc table using SQL. The two query paths are shown in Fig. 1 in Section 3. In learning stage I, whenever the system answers an input query using the original stored relations in the CDDB database, it increments the access count for each answer tuple in the Disc table. For learning stage II to be invoked, the access counts have to reach the threshold value; see Section 4.2. In the second stage of learning, we materialize in XML all the tuples in the Disc relation whose access counts exceed the threshold value. The generation of materialized XML is explained in Section 4. Fig. 4 shows the materialized XML for the CD “Air Supply” generated from Disc and Track. Cluster formation: This phase is invoked for every tuple in the relation XMLDiscTrack that holds materialized XML. The XML shown in Fig. 4 is only suitable to answer those queries that ask for tracks in that CD; this restriction limits the scope of the approach. Hence, we form clusters. Clusters are formed by identifying related records. The algorithm for selecting these related records is explained formally in Section 4. The tuples in the Disc relation that match ‘Air Supply’ and that have their access counts above the threshold value are chosen to form the clusters. These tuples are converted to XML and merged into the original structure. An example for the merged structure is shown in Fig. 5. Once the clustering phase is completed, the XML shown in Fig. 6 replaces the XML in Fig. 4. 5.2

Experimental Results

In this section we show the results of our experiments on the feasibility of our learning-based materialization approach. Comparing the efficiency of querying materialized XML to the efficiency of getting answers to SQL queries on the stored relational data. The objective of this experiment was to analyze whether XQuery-based querying is effective on materialized XML views, as compared to using SQL on the stored relations.

306


Fig. 5. An example of materialized clustered XML

Fig. 6. Comparison between using random queries on materialized XML and on relational data

Fig. 7. Average query times for a random set of 1000 repeated queries on relational data and an analysis of the time required to convert this relational data to XML


307

Fig. 7 shows query-execution times for 5000 XML records, for the query SELECT * FROM Disc, Track WHERE Disc.cd_id = Track.cd_id AND Disc.cd_title = ‘%Eddie Murphy%’. Interesting tuples in the join of Disc and Track are stored in XML. The relational tables hold 2.5 million tuples (250K CDs times 10 tracks). The cluster records are similar to the XML shown in Fig. 5. The graph is a plot of queryexecution times for XQuery queries based on the attribute cd_title of the Disc relation. The experiment shows that processing a query on an XML view is faster than using SQL on the relations and then converting the answer to XML. Fig. 6 shows that executing SQL queries is more time-consuming than executing their XQuery counterparts on materialized XML. In pushing XQuery queries to the relational data, converting the answers into XML is a major overhead. We analyze the overhead in Fig. 7, which shows that the process of converting answer tuples into XML is the most expensive part of answering queries. Hence, it would be beneficial if such data were to be materialized.

Analyzing the maximum time spent in converting query answers to XML. The objective of this experiment was to analyze the time spent on translating relational query answers into XML. The graph in Fig. 7 is a plot of query-execution times for SQL queries based on the attribute cd_title of the Disc relation. The graph shows, as a solid line, the mean execution times for relational queries plus the times to convert the answers into XML. We see that of the total time of around 190 ms, converting relational data into XML takes around 60ms (the dotted line). While the relational query takes 190 ms – 60 ms = 130 ms to execute, there is an overhead of 60 ms in converting the relational data to XML. These results are the motivation for using materialization techniques.

Simulation runs to show the decrease in total query-execution times when querying the materialized XML alongside the stored relational data.

Fig. 8. Average query-execution times for a randomized set of 1000 repeated queries on a combination of relational and materialized data

Fig. 8 shows 30 simulation runs for a query workload of 1000 randomly selected CD titles. The X-axis shows the query ID, while the Y-axis shows the queryexecution times. The vertical dotted lines show the points at which XML materialization took place. (Recall that the system periodically runs the learning

308


algorithm.) It can be seen in Fig.8 that after every learning stage, the slope of the curve falls. Intuitively, after new learning has taken place, the XML clusters can satisfy a higher number of queries, with higher efficiency.

6 Discussion The proposed approach is to store materialized XML views in a relational database using learning. One extreme of the approach is to materialize the entire relational database as XML and then use a native XML engine to answer queries. This way, we would be able to avoid the overhead of translating all possible queries on the data source into SQL, and of translating the relational answers to the queries into XML. However, query performance might degrade considerably, as XML query-answering techniques are slower than their relational counterparts. In addition, the system would have to incur a significant overhead of keeping the XML consistent with the underlying relations. In Section 4 we described the process of grouping together related records in XML clusters. This approach allows a database system to incrementally find an optimal proportion of XML records that can be accessed faster than the relational tables. This optimal proportion can be arrived at by varying the size of the clustered XML and the threshold value. Additional improvements can be made when user applications maintain local caches: It may be beneficial to prefetch the XML data in the application’s cache, so that future queries from the application have a higher chance of being satisfied locally. In our approach, the added counters and flags in the relations have to be updated frequently and thus create an overhead. In our future work, we plan to reduce the overhead by updating tuple-access counts offline or during periods of lower queryloads. We materialize only frequently-accessed tuples; thus, only a fraction of the database is materialized as XML at any given time. (The clusters are recomputed from scratch every time the learning phase is invoked.) The advantage of the learning approach is to balance the proportion of data in relations and XML, by materializing the tuples that are in the answers to multiple queries. As the materialized XML is generated based on the access count of relational tuples, there may be queries that need to access both materialized XML and relational database. We plan to explore how to handle such queries in our future work.

7 Conclusions and Future Work We have described a view- and learning-based solution to the problem of reducing total query-execution times in relational data sources with XML interfaces. Our approach combines learning techniques with selective materialization; our experiments show that it can prove beneficial in improving query-execution speeds in relatively static databases. This paper describes an implementation that is external to the database engine. We are currently working on incorporating our approach inside a relational database-


309

management system. We are looking into automating schema definition for materialized XML clusters, by using the information about past query workloads and the relations accessed by these workloads. We are working on developing a learning approach to selecting “interesting records” for XML clusters. We plan to implement a dynamic approach to selecting the threshold value in XML materialization. We plan to devise better strategies for (1) prioritizing XML records within clusters, and (2) automatically dematerializing obsolete XML data. Finally, we plan to automate the choice of the relations of interest, given a query workload.

References 1. 2. 3.

4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15.

J. Chen, S. Chen, and E.A. Rundensteiner. A transactional model for data warehouse maintenance. In Proc. of the 21st Int’l Conference on Conceptual Modeling (ER), 2002. L. Chen, E.A. Rundensteiner, and S. Wang. XCache: A semantic caching system for XML queries. In Proc. 2002 ACM SIGMOD International Conference on Management of Data, 2002. K.T. Claypool, E.A. Rundensteiner, X. Zhang, H. Su, H.A. Kuno, W.C. Lee, and G. Mitchell. Gangam — a solution to support multiple data models, their mappings and maintenance. In Proc. 2001 ACM SIGMOD International Conference on Management of Data, 2001. L. Chen, S. Wang, E. Cash, B. Ryder, I. Hobbs, and E.A. Rundensteiner. A fine-grained replacement strategy for XML query cache. In Proc. Fourth ACM CIKM International Workshop on Web Information and Data Management (WIDM 2002), pages 76–83, 2002. J. Chen, X. Zhang, S. Chen, A. Koeller, and E.A. Rundensteiner. DyDa: Data warehouse maintenance in fully concurrent environments. In Proc. ACM SIGMOD, 2001. D.W. Embley and W.Y. Mok. Developing XML Documents with Guaranteed “Good” th Properties. In Proc. 20 International Conference on Conceptual Modeling (ER), pages (426–441), 2001. I.M.R.E. Filha, A.S. da Silva, A.H.F. Laender, and D.W. Embley. Using nested tables for representing and querying semistructured web data. In Proceedings of the Advanced Information Systems Engineering, 14th International Conference (CAiSE 2002), 2002. M. Fernandez, Y. Kadiyska, D. Suciu, A. Morishima, and W.C. Tan. SilkRoute: A framework for publishing relational data in XML. ACM Trans. Database Systems, 27(4):438–493, 2002. Yannis E. Ioannidis. Query optimization. In Allen B. Tucker, editor, The Computer Science and Engineering Handbook, pages 1038–1057. CRC Press, 1997. Z. Liu, F. Li, and W.K. Ng. Wiccap data model: Mapping physical websites to logical views. In Proc. 21st International Conference on Conceptual Modeling (ER), 2002. Liu Mengchi. A logical foundation for XML. In Proc. Advanced Information Systems Engineering, 14th International Conference (CAiSE 2002), pages 568–583, 2002. Tom M. Mitchell. Generalization as search. Artificial Intelligence, 18:203–226, 1982. Liu Mengchi and Tok Wang Ling. Towards declarative XML querying. In Proc. 3rd International Conference on Web Information Systems Engineering (WISE 2002), pages 127–138, 2002. K. Passi, L. Lane, S.K.Madria, B.C. Sakamuri, M.K. Mohania, and S.S. Bhowmick. A model for XML schema integration. In Proc. 3rd Int’l Conf. E-Commerce and Web Technologies, 2002. Giuseppe Psaila. ERX: An experience in integrating entity-relationship models, relational databases, and XML technologies. In Proc. XML-Based Data Management and Multimedia Engineering EDBT workshop, 2002.

310


16. J. Shanmugasundaram, J. Kiernan, E. J. Shekita, C. Fan, and J. Funderburk. Querying XML views of relational data. In Proc. 27th Int’l Conference on Very Large Data Bases, 2001. 17. Jennifer Widom. Research problems in data warehousing. In Proc. Fourth International Conference on Information and Knowledge Management, pages 25–30, 1995. 18. Extensible Markup Language (XML) http://www.w3.org/XML. 19. X. Zhang, L. Ding, and E.A. Rundensteiner. Parallel multi-source view maintenance. VLDB Journal: Very Large DataBases, 2003. (To appear). 20. X. Zhang, M. Mulchandani, S. Christ, B. Murphy, and E.A. Rundensteiner. Rainbow: mapping-driven XQuery processing system. In Proc. ACM SIGMOD, 2002. 21. Xin Zhang and Elke A. Rundensteiner. Integrating the maintenance and synchronization of data warehouses using a cooperative framework. Information Systems, 27:219–243, 2002. 22. The CDDB database. http://www.freedb.org.

A Framework for Management of Concurrent XML Markup Alex Dekhtyar and Ionut E. Iacob Department of Computer Science University of Kentucky Lexington, KY 40506 {dekhtyar,eiaco0}@cs.uky.edu Abstract. The problem of concurrent markup hierarchies in XML encodings of works of literature has attracted attention of a number of humanities researchers in recent years. The key problem with using concurrent hierarchies to encode documents is that markup in one hierarchy is not necessarily well-formed with respect to the markup in another hierarchy. The proposed solutions to this problem rely on the XML expertise of the editors and their ability to maintain correct DTDs for complex markup languages. In this paper, we approach the problem of maintenance of concurrent XML markup from the Computer Science perspective. We propose a framework that allows the editors to concentrate on the semantic aspects of the encoding, while leaving the burden of maintaining XML documents to the software. The paper describes the formal notion of the concurrent markup languages and the algorithms for automatic maintenance of XML documents with concurrent markup.

1

Introduction

The problem of concurrent markup hierarchies has recently attracted the attention of a number of humanities researchers [13,6,15]. This problem typically manifests itself when a researcher must encode in XML a large document (book, manuscript, printed edition) with a wide variety of features. A concurrent hierarchy is formed by a subset of the elements of the markup language used to encode the document. The elements within a hierarchy have a clear nested structure. When more than one such hierarchy is present in the markup language, the hierarchies are called concurrent. A typical example of concurrent hierarchies is the XML markup used to encode the physical location of text in a printed edition: book, page, physical line, vs. the markup used to encode linguistic information about the text: sentence, phrase, word, letter. The key problem with using concurrent hierarchies to encode documents is that markup in one hierarchy is not necessarily well-formed with respect to the markup in another hierarchy.

This work has been supported in part by NSF ITR grant 0219924. In addition, the work of the second author has been supported in part by NEH grant RZ-20887-02. The manuscript image [1] appearing in this paper was digitized for the Electronic Boethius project by David French and Kevin Kiernan and is used with permission of the British Library Board.


312

A. Dekhtyar and I.E. Iacob

The study of concurrent XML hierarchies for encoding documents is related to the problem of manipulation and integration of XML documents. However, most of the research on XML data integration addresses the problem of integrating heterogeneous, mostly data-centric XML provided by various applications ([4,9,10,8]). In our case, the data to be integrated has a common denominator: the document content, and the XML encodings are document-centric. Also, the features of the document to be marked up are not (in most cases) heterogeneous, but they might be conflicting in some instances. Management of concurrent markup has been approached in a few different ways. The Text Encoding Initiative (TEI) Guidelines [13] suggest a number of solutions based on the use of milestone elements (empty XML elements) or fragmentation of the XML encoding. Durusau and O’Donnell [6] propose a different approach. They construct an explicit DTD for each hierarchy present in the markup. Then they determine the ”least common denominator” in the markup — the units of content inside which no overlap occurs, in their case, words. They associate attributes indicating the XPath expression leading to the content of each word element for each hierarchy. Other scholars have proposed the use of non-XML markup languages that allow concurrent hierarchies [7]. In their attempts to resolve the problem of concurrent hierarchies, both [13] and [6] rely on the human editor to (i) introduce the appropriate solution to the XML DTD/XSchema, and (ii) follow it in the process of manual encoding of the documents. At the same time, [6] emphasizes the lack of software support for the maintenance of the concurrent markup, which makes, for example, adhering to some of the TEI solutions a strenuous task. While some recent attempts have been made to deal with the problem of concurrent markup from a computer science perspective [14,15], a comprehensive solution has yet to be proposed. This paper attempts to bridge the gap between the apparent necessity for concurrent markup and the lack of software support for it by proposing a framework for the creation, maintenance and querying the concurrent XML markup. This framework relies on the following: – Separate DTDs for hierarchies; – Use of a variant of fragmentation with virtual join suggested by TEI Guidelines [13] to represent full markup; – Automatic maintenance of markup; – Use of a database as XML repository. The ultimate goal of the proposed framework is to free the human editor from the effort of dealing with the validity and well-formedness issues of document encoding and to allow him or her to concentrate on the meaning of the encoding. This goal is achieved in the following way. Durusau and O’Donnell [6] note the simplicity and clarity of DTDs for individual concurrent hierarchies, as opposed to a unified DTD that incorporates all markup elements. Our approach allows the editor to describe a collection of such simple DTDs without having to worry about the need to build and maintain a ”master” DTD. At the same time, existence of concurrent DTDs introduces the need for specialized software to support the editorial process drive it by the semantics of the markup. This

A Framework for Management of Concurrent XML Markup

313

software must allow the editor to indicate the positions in the text where the markup is to be inserted, select the desired markup, and take record the results. In this paper we introduce the foundation for such software support. In Section 2 we present a motivating example based on our current project. Section 3 formally defines the notion of a collection of concurrent markup languages. In Section 4 we present three key algorithms for the manipulation of concurrent XML markup. The Merge algorithm builds a single master XML document from several XML encodings of the same text in concurrent markup. The Filter algorithm outputs an XML encoding of the text for an individual markup hierarchy, given the master XML document. The Update algorithm incrementally updates the master XML document given an atomic change in the markup. This paper describes the work in progress. A major issue not addressed in here is the database support for multiple concurrent hierarchies in our framework. This problem is the subject of ongoing research.

2

Motivating Example

Over the past few years researchers in the humanities have used XML extensively to create readable and searchable electronic editions of a wide variety of literary works [11,12,6]. The work described in this paper originated as an attempt to deal with the problem of concurrent markup in one such endeavor, The ARCHWay Project, a collaborative effort between Humanities scholars and Computer Scientists at the University of Kentucky. This project is designed to produce electronic editions of Old English manuscripts. In this section, we illustrate how concurrent markup occurs in ARCHWay. Building electronic editions of manuscripts. Electronic editions of Old English manuscripts [11,12]combine the text from a manuscript (both the transcript and the emerging edition), encoded in XML using an expressive array of features (XML elements), and a collection of images of the surviving folios of the manuscript. The physical location of text on the surviving folios, linguistic information, condition of the manuscript, visibility of individual characters, paleographic information, and editorial emendations are just some of the features that need to be encoded to produce a comprehensive description of the manuscript. Specific XML elements are associated with each feature of the manuscript. Concurrent hierarchies and conflicts. Most of the features have explicit scopes: the textual content (of the manuscript) that the feature relates to, be it the text of a physical line, or a line of verse or prose, or manuscript text that is missing due to a damage in the folio. Unfortunately, the scopes of different features often overlap, resulting in non-well-formed encoding (we call such a situation a conflict). Consider a fragment of folio 38 verso of British Library Cotton Otho A vi [1] (King Alfred’s Boethius manuscript) shown in Fig.1. The text of the three lines depicted on this fragment is shown in the box marked (0) in Fig.1. The remaining

314


Fig. 1. A fragment of King Alfred’s Boethius manuscript [1] and different XML encodings

boxes in Fig.1 show the following markup for this fragment: (i) information about physical breakdown of the text into lines ( element); (ii) information about the structure of the text (<w> element encodes words), (iii) information about the damage and text obscured by the damage ( and tags)1 . Some of the encodings of this fragment are in conflict. The solid boxes over parts of the image indicate the scope of the elements and the dotted boxes indicate the scope of the elements. In addition, we indicate the positions of some of the <w> tags. Damage and restoration markup overlaps words in some places: the damaged text includes the end of one word and the beginning of the next word. In addition to that, some words start on one physical line and continue on another. Resolving markup conflicts. The TEI Guidelines [13] suggest a number of possible ways to resolve conflicts. These methods revolve around the use of empty 1

The encodings are simplified. We have removed some attribute values from the markup to highlight the structure of each encoding.


315

<w>hu <w>iu <w>me <w>hæfst <w>afrefredne <w>ægier <w>ge <w>mid

(a) Milestone elements. ..... <w>æg <w>ier .....

(b) Fragmentation. ..... <w id=”1”>æg <w id =”1”>ier .....

(c) Fragmentation with virtual join (variant with “glue” attribute). Fig. 2. Resolving markup conflicts

milestone tags and the fragmentation of markup. We illustrate the proposed suggestions in Fig.2 on the example of the markup conflict between the <w> and elements at the end of line 22. The first suggested way (Fig.2.(a)) uses milestone (empty) elements. In this case the editor determines the pairs of tags that may be in conflict, and for each such pair declares at least one tag as empty in the DTD/XSchema. The other two ways (Fig.2.(b),(c)) are variants of the fragmentatation technique: one of the conflicting elements is split into two parts by the other one (in Fig.2 we choose to split <w> element). Simple fragmentation, however, may be confusing: encoding in Fig.2.(b) creates the impression that “æg” and “ier” are two separate words. To alleviate this problem, a variety of conventions based on the use of attributes can be proposed to indicate that a specific element encodes a fragment. Fig.2.(c) shows one such convention that uses a “glue” attribute Id. This implied attribute will get the same value for all fragments of the same encoding. Key drawback. The answer lies not only in alleviating the markup conflict problem: a more general problem of maintenance of markup in situations where conflicts are a frequent occurrence must be addressed. Up to this point, such maintenance resided in the hands of human editors who were responsible for specific encoding decisions to prevent markup conflicts. This tended to generate a variety of gimmick solutions in the markup language, such as introduction of tags whose sole purpose was to overcome a specific type of conflict, but which, in the process made the DTD/XSchema of the markup language complex and hard to maintain. Our approach, described in the remainder of this paper allows the software to take over the tasks of markup maintenance, simplifying the work of editors.

3

Concurrent XML Hierarchies

In this section we formally define the notion of the collection of concurrent markup hierarchies. Given a DTD D, we let elements(D) denote the set of all

316


markup elements defined in D. Similarly, we let elements(d), where d is an XML document, denote the set of all element tags contained in document d. Definition 1. A concurrent markup hierarchy CM H is a tuple CM H =< S, r, {D1 , D2 , ..., Dk } > where: • S is a string representing the document content; • r is an XML element called the root of the hierarchy; • Di , i = 1, k are DTDs such that: (i) r is definedin each Di , 1 ≤ i ≤ k, and ∀1 ≤ i, j ≤ k, i = j elements(Di ) elements(Dj ) = {r}; (ii) ∀1 ≤ i ≤ k, ∀t ∈ elements(Di ) r is an ancestor of t in Di . In other words, the collection of concurrent markup hierarchies is composed of textual content and a set of DTDs sharing the same root element and no other elements. Definition 2. Let CM H =< S, r, {D1 , D2 , ..., Dk } > be a concurrent markup hierarchy. A distributed XML document dd over CM H is a collection of XML documents: dd =< d1 , d2 , ..., dk > where (∀1 ≤ i ≤ k) di is valid w.r.t. Di and content(d1 ) = content(d2 ) = ... = content(dk ) = S 2 . The notion of a distributed XML document allows us to separate conflicting markup into separate documents. However, dd is not an XML document itself, rather it is a virtual union of the markup contained in d1 ,. . . ,dk . Our goal now is to define XML documents that incorporate in their markup exactly the information contained in a distributed XML document. We start by defining a notion of a path to a specific character in content. Definition 3. Let d be an XML document and let content(d) = S. Let S = c1 c2 . . . cM . The path to ith character in d denoted path(d, i) or path(d, ci ) is the sequence of XML elements forming the path from the root of the DOM tree of d to the content element that contains ci . Let D be a DTD and let elements(D) ∩ elements(d) = ∅, and let the root of d be a root element in D. Then, the path to ith character in d w.r.t. D, denoted path(d, i, D) or path(d, ci , D) is the subsequence of all elements of path(d, i) that belong to D. Following XPath notation, we will write path(d, i) and path(d, i, D) in a form a1/a2/ . . . /as . We notice that path(d, i, D) defines the projection of the path to ith character in d onto a specific DTD. For example, if path(d, i) = col/f ol/pline/line/w/dmg and D contains only elements , and <w>, then path(d, i, D) = col/pline/w. We can now use paths to content characters to define “correct” single-document representations of the distributed XML documents. Definition 4. Let d∗ be an XML document and let D be a DTD, such that elements(d∗ )∩elements(D) = ∅ and the root of d∗ is a root element in D. Then, 2

content(doc) denotes the text content of the XML document doc.


317

the set of filters of d∗ onto D, denoted F ilters(d∗ , D) is defined as follows: F ilters(d∗ , D) = {d|content(d) = content(d∗ ), elements(d) = elements(d∗ ) ∩ elements(D) and (∀1 ≤ i ≤ |content(d)|)path(d∗ , i, D) = path(d, i)} Basically, a filter of d∗ on D is any document that contains only elements from D that preserves the paths to each content character w.r.t. D. If we are to combine the encodings of all di s of a distributed document dd in a single document d∗ we must make sure that we can “extract” every individual document di from d∗ . Definition 5. Let dd =< d1 , d2 , . . . dk > be a distributed XML document over the collection of markup hierarchies CM H =< S, r, {D1 , . . . , Dk } >. A set of mergers of dd denoted M ergers(dd) is defined as M ergers(dd) = {d∗ |elements(d∗ ) ⊆

k

elements(Dj )

j=1

and (∀1 ≤ i ≤ k)di ∈ F ilters(d∗ , Di )} Given a distributed XML document dd, we can represent its encoding by constructing a single XML document d∗ from the set M ergers(dd). d∗ incorporates the markup from all documents d1 , . . . , dk in a way that (theoretically) allows the restoration of each individual document from d∗ . A document d∗ ∈ M ergers(dd) is called a minimal merger of dd iff for each content character cj , path(d∗ , cj ) consists exactly of the elements from all path(di , cj ), 1 ≤ i ≤ k.

4

Algorithms

Section 3 specifies the properties that the “right” representations of distributed XML documents (i.e., XML markup in concurrent hierarchies within a single XML document) must have. In this section we provide the algorithms for building such XML documents. In particular, we address the following three problems: – Merge: given a distributed XML document dd, construct a minimal merger d∗ of dd. We will refer to the document constructed by our Merge algorithm as the master XML document for dd. – Filter: given a master XML document for some distributed document dd and one of the concurrent hierarchies Di , construct the document di . – Update: given a distributed XML document dd, its master XML document d∗ and a simple update of the component di of dd, that changes it to di , construct (incrementally) the master XML document d for the distributed document dd =< d1 , . . . , di , . . . , dk >. Fig.3 illustrates the tasks addressed in this section and the relationship between them and the encoding work of editors. In the proposed framework, the

318


Fig. 3. The framework solution.

editors are responsible for defining the set {D1 , . . . , Dk } of the concurrent hierarchies and for specifying the markup for each component of the distributed document dd. The MERGE algorithm then automatically constructs a single master XML document d∗ , which represents the information encoded in all components of dd. The master XML document can then be used for archival or transfer purposes. When an editor wants to obtain an XML encoding of the content in a specific hierarchy, the Filter algorithm is used to extract the encoding from the master XML document. Finally, we note that MERGE is a global algorithm that builds the master XML document from scratch. If a master XML document has already been constructed, the Update algorithm can be used while the editorial process continues to update incrementally the master XML document given a simple (atomic) change in one of the components of the distributed XML document. Each algorithm is discussed in more detail below. Note that the theorems in this section are given without proofs. The proofs can be found in [5]. 4.1

MERGE Algorithm

The MERGE algorithm takes as input tokenized versions of the component documents d1 , . . . , dk of the distributed document dd and produces as output a single XML document that incorporates all the markup of d1 , . . . , dk . The algorithm resolves the overlap conflicts using the fragmentation with a ”glue” attribute approach described in Section 2. A special attribute link is added to all markup elements that are being split, and the value of this attribute is kept the same for all markup fragments. The algorithm uses the Simple API for XML (SAX)[3] for generating tokens. SAX callbacks return three different types of token strings: (i) start tag token string (ST), (ii) content token string (CT), (iii) end tag token string (ET). If token is the token returned by the SAX parser, then we use type(token) to denote its type (ST, CT, ET) as described above and tag(token) to denote the tag returned by SAX (for ST and ET tokens). The MERGE algorithm works in two passes. On the first pass, the input documents are parsed in parallel and an ordered list is built of ST and ET tokens for the creation of the master XML document. The second pass of the


319

algorithm scans the token list data structure built during the first pass and outputs the text of the master XML document. The main data structure in the MERGE algorithm is tokenListSet, which is designed to store all necessary markup information for the master XML document. Generally speaking, tokenListSet is an array of token lists. Each array position corresponds to a position in the content string of the input XML documents. In reality, only the positions at which at least one input document has ST or ET tokens have to be instantiated. For each position i, tokenListSet[i] denotes the ordered list of markup entries at this position. At the end of the first pass of the MERGE algorithm, for each i, tokenListSet[i] will contain the markup elements to be inserted in front of ith character of the content string in the master XML document exactly in the order they are to be inserted. The second pass of the MERGE algorithm is a straightforward traversal of tokenListSet, which for each position outputs all the tokens and then the content character. Fig.4 contains the pseudocode for the MERGE algorithm. The algorithm iterates through the positions in the content string of the input documents. For each position i, the algorithm first collects all ET and ST tokens found at this position. It then determines the correct order in which the tokens must be inserted in the master XML document, and resolves any overlaps by inserting appropriate end tag and start tag tokens at position i and adding the link attribute to the start tag tokens. In the algorithm push(Token,List) and append(Token,List) add Token at the beginning and at the end of List respectively. Theorem 1. Let dd =< d1 , . . . , dk > be a distributed XML document. Let d∗ be the output of MERGE(d1 , . . . , dk ). Then d∗ is a minimal merger of dd. 4.2

FILTER Algorithm

The FILTER algorithm takes as input an XML document d∗ produced by the MERGE algorithm and a DTD D, filters out all markup elements in d∗ that are not in D and merges the fragmented markup. In one pass the algorithm analysis the ordered sequence of tokens provided by a SAX parser and performs the following operations: – removes all ST and ET tokens of markup elements not in D; – from a sequence ST, [CT], ET, [CT], ..., ST, [CT], ET of tokens for a fragmented element in D, removes the ”glue” attributes and outputs the first ST token, all possible intermediate CT tokens and the last ET token in the sequence; – all other tokens are output without change in the same order they are received from the SAX parser. The pseudo-code for FILTER appears in Fig.5. The following theorem states that FILTER correctly reverses the work of the MERGE algorithm. Theorem 2. Let dd =< d1 , . . . , dk > be a distributed XML document, and d∗ be the output of MERGE(dd). Then (∀1 ≤ i ≤ k), FILTER(d∗ , Di ) = di .

320


Fig. 4. The MERGE algorithm

Fig. 5. The FILTER and UPDATE algorithms


321

LCA AFROM

ATO

FROM TO from to

Fig. 6. The XML document tree model used in UPDATE algorithm

4.3

UPDATE Algorithm

The UPDATE algorithm updates the master XML document (see Fig.3) with the new markup element. It takes as the input two integers, f rom and to, the starting and ending positions for the markup in the content string and the new markup element, T AG. Due to possible need to fragment the new markup this process requires some care. The goal of the algorithm is to introduce the new markup into the master XML document in a way that minimizes the number of new fragments. The algorithm uses the DOM model [2] for the XML document and performs the insertion of the node in the XML document tree model. In this model, for an element with mixed content, the text is always a leaf. Then f rom and to will be positions in some leaves of the document tree. Let F ROM and T O be the parent nodes of the text leaves containing positions f rom and to respectively. We denote by LCA the lowest common ancestor of nodes F ROM and T O. Let AF ROM be child of LCA that is the ancestor of F ROM , and let AT O be the child of LCA that is the ancestor of T O (see Fig.6). The UPDATE algorithm traverses the path F ROM → . . . → AF ROM → LCA → AT O → . . . → T O and inserts T AG nodes with glue attributes as needed. The pseudo-code description of the algorithm is shown in Fig.5. The following theorem says that the result of UPDATE allows for correct recovery of components of the distributed document. Theorem 3. Let dd =< d1 , . . . , dk > be a distributed XML document and d∗ be the output of MERGE(dd). Let T AG ∈ elements(Di ), (f rom, to, T AG) be an update request and di be a well-formed result of marking up the content between f rom and to positions. Then, FILTER(UPDATE(d∗ , (f rom, to, T AG)), Di ) = di .

5

Future Work

This paper introduces the general framework for managing concurrent XML markup hierarchies. There are three directions in which we are continuing this research. First, we are working on providing the database support for the maintenance of concurrent hierarchies. Second, we are studying the properties of the proposed algorithms w.r.t. the size of the markup generated, optimality of the

322


markup and computational complexity, and efficient implementation of the algorithms. Finally, we are planning a comprehensive comparison study of a variety of methods for support of concurrent hierarchies.

References 1. British Library MS Cotton Otho A. vi, fol. 38v. 2. Document Object Model (DOM) Level 2 Core Specification. http://www.w3.org/TR/2000/REC-DOM-Level-2-Core-20001113/, Nov 2000. W3C Recommendation. 3. Simple API for XML (SAX) 2.0.1. http://www.saxproject.org, Jan 2002. SourceForge project. 4. Serge Abiteboul, Jason McHugh, Michael Rys, Vasilis Vassalos, and Janet L. Wiener. Incremental maintenance for materialized views over semistructured data. In Proc. 24th Int. Conf. Very Large Data Bases, VLDB, pages 38–49, 24–27 1998. 5. Alex Dekhtyar and Ionut E. Iacob. A framework for management of concurrent XML markup. Technical Report TR 374-03, University of Kentucky, Department of Computer Science, June 2003. http://www.cs.uky.edu/∼dekhtyar/publications/TR374-03.concurrent.ps. 6. P. Durusau and M. B. O’Donnell. Concurrent Markup for XML Documents. In Proc. XML Europe, May 2002. 7. C. Huitfeldt and C. M. Sperberg-McQueen. TexMECS: An experimental markup meta-language for complex documents. http://www.hit.uib.no/claus/mlcd/papers/texmecs.html, February 2001. 8. Ioana Manolescu, Daniela Florescu, and Donald Kossmann Kossmann. Answering XML queries over heterogeneous data sources. pages 241–250. 9. Wolfgang May. Integration of XML data in XPathLog. In DIWeb, pages 2–16, 2001. 10. Wolfgang May. Lopix: A system for XML data integration and manipulation. In The VLDB Journal, pages 707–708, 2001. 11. W.B. Seales, J. Griffioen, K. Kiernan, C. J. Yuan, and L. Cantara. The Digital Atheneum: New Technologies for Restoring and Preserving Old Documents. Computers in Libraries, 20(2):26–30, February 2000. 12. E. Solopova. Encoding a transcript of the beowulf manuscript in sgml. In Proc. ACH/ALCC, 1999. 13. C. M. Sperberg-McQueen and L. Burnard(Eds.). Guidelines for Text Encoding and Interchange (P4). http://www.tei-c.org/P4X/index.html, 2001. The TEI Consortium. 14. C. M. Sperberg-McQueen and C. Huitfeldt. GODDAG: A Data Structure for Overlapping Hierarchies, Sept. 2000. Early draft presented at the ACH-ALLC Conference in Charlottesville, June 1999. 15. A. Witt. Meaning and interpretation of concurrent markup. In Proc., Joint Conference of the ALLC and ACH, pages 145–147, 2002.

Object Oriented XML Query by Example Kathy Bohrer, Xuan Liu, Sean McLaughlin, Edith Schonberg, and Moninder Singh {bohrer,xuanliu,ediths,moninder}@us.ibm.com [email protected]

Abstract. This paper describes an XML query language called XML-QBE, which can be used to both query and update XML documents and databases. The language itself has a simple XML form, and uses a query by example paradigm. This language was designed as a middleware layer between UML data models and backend database schemas, as part of a solution to the distributed, heterogeneous data-base problem and legacy database problem. Because the XML layer is derived from UML, XML-QBE is object-oriented. Queries and updates have a very similar form, and the form itself is XML. Therefore this language is also easy to process and analyze. We describe the language, the rationale, and our solution architecture.

1 Introduction The use of XML is now pervasive. At the application level, XML has become the common medium for information exchange. At the middleware level, XML is being incorporated into standard protocols such as SOAP. At the system level, the convenient, self-describing format of XML makes it ideal for the persistent storage of both structured and unstructured data. Consequently, XML query languages are being designed and standardized for managing the growing bodies of XML documents and repositories. In this paper, we present an XML query language, which we call XML-QBE. XML-QBE is itself an XML language, in addition to being a query language for XML. It uses a “query by example” paradigm, which means that the queries look like the data. Typically, query by example languages are very intuitive and easy to use. We designed XML-QBE as part of a solution to integrate distributed, heterogeneous, and legacy databases. In our solution architecture, a unifying XML data model, which is an XML Schema, is defined to integrate the data models across multiple backend databases, thus hiding the details of the potentially heterogeneous and inconsistent backends (see Figure 1). A database backend can be an XML database, a repository of XML documents, a traditional relational or LDAP database, etc. The XML schema itself is derived automatically from a UML data model, using an XMI-to-XML Schema translation tool, written in XSL. For each backend, we provide a mapping table, which specifies how the XML schema is mapped to the target backend schema. Each mapping table is used to translate XML-QBE queries into the appropriate target query language, such as SQL for relational backends, LDAP query language, or XQuery for XML document repositories. M.A. Jeusfeld and Ó. Pastor (Eds.): ER 2003 Workshops, LNCS 2814, pp. 323–329, 2003. © Springer-Verlag Berlin Heidelberg 2003

324

K. Bohrer et al.

The design goals for XML-QBE were the following:

y Expressiveness - we are able to express complex queries which retrieve sets of related objects, which are either nested or related through object reference. y Easy to analyze – queries themselves are XML data. y Semantics reflect the UML data model, not the target - the user is not aware of an underlying relational implementation, for example, or semantics of SQL. y Simple and declarative syntax – the query-by-example paradigm was chosen for this reason. y Uniform syntax for all operations – operations include query, create, delete, and modify. y Well-defined syntax and semantics – syntax is expressible by an XML Schema. y Generality -- applicable to all UML data models. y Support for multiple targets – queries can be mapped to a variety of backends. U M L D ata M odel U M L to X M L S chem a translation tool

XM L S chem a

Ma ppin g Tab les X M L S ch em a to B ac ken d Targe t (S Q L , XQ u ery, LD AP ...)

B ackend D atabase Schem a Backend D atabase S chem a

Backend D atabase S chem a

Fig. 1. 1: Solution Overview Figure So lution O verview

2 XML-QBE Features and Examples XML-QBE queries are operations over a collection of objects represented by XML elements. The XML-QBE data models are object-oriented, and they correspond to UML data models (see Figure 1). Objects have properties which are specified in the corresponding UML model. Similarly, objects are related to each other according to the associations defined in the UML model. A sample XML representation for a set of objects describing a personal profile is shown in the Appendix. The conventions used by our XML object representation include the following: 1. All objects have an “oid” property, which is the primary key of the object class. 2. Objects may be nested. For example, the person object with oid PERS1. includes the personDemographics object PDEM1, the occupation object OCCU1, the hobby object HOBB1, and the nationality object NAT1 nested within it.

Object Oriented XML Query by Example

325

3. A property whose name ends with “Id” is a reference to another object. For example, the property defaultNameId in person references the personName object PNAM1. (In the corresponding UML data model, there is an association defaultName between the Person class and the PersonName class). 4. Multi-valued properties are always grouped together under an element with a name ending in Group. For example, partyActivityGroup in the person object is the parent of all activities (including hobbies and occupations) of the person. The remaining subsections give an overview of the XML-QBE language. 2.1 Single Object Queries The simplest queries request objects of the same class. To indicate which objects to return, the values of properties are specified. To indicate which object properties to return, property names are specified without any values. If more than one property is specified, then any result objects returned must match all properties specified in the query. The following query returns properties of all personName objects with lastName “Bingh”. Specifically, the use, fullName, firstName, lastName, and title are returned. (The oid of the object is also always returned.) <use/> Bingh

The result of this query when applied to the document specified in the Appendix is: PNAM1 <use>LEGAL Cherry Bingh Cherry Bingh Mrs.

An element in a query may use the attribute “return” to specify more precisely which properties to return. Attribute “return” can have the value “all” (return all properties), “none” (do not return this object), or “specified” (return only those properties which are present in the query). If the “return” attribute is not used, then the default is “specified”. The “return” attribute can apply to property elements within an object. If an object has return value “all”, and a property element of the object has return value “none”, then this property is not returned, while all of the other properties of the object are returned. However, the converse is not true. If an object has return value “none”, and a property element of the object has return value “all”, then no property of the object is returned.

326

K. Bohrer et al.

2.2 Nested Object Queries Objects in our data model can be nested. We describe how the rules defined in section 2.1 extend to nested objects. To indicate which objects to return, the values of properties are specified. For nested objects, values may be specified for properties at any level of nesting. Specifying the value of an inner property means return all objects which contain an embedded object with the specified property value. To indicate which object properties to return, property element names are specified without any values. Similarly, embedded element names are specified without any values to indicate which embedded objects to return. In this case, all properties and embedded objects of the nested object are returned. Embedded element names also can have a return attribute, with values “all”, “none”, or “specified”. The following query returns the person objects with birthdate 3/1/1965. It also returns the personDemographics and the nationalityGroup objects embedded in these person objects. Note that the default return value for person and personDemographics is “specified”, since this element has embedded property elements. The default return value for nationalityGroup is “all”, since there are no properties specified for this nested element. 1965-03-01

2.3 Related Object Queries Often, it is necessary to request an object and the other objects which are related to it. For example, consider the query for retrieving all person and personName objects for all persons with defaultName “Cherry Bingh”. One way to write this query is shown below. It uses the oid value PNAM1 of the personName object for Cherry Bingh. <defaultNameId>PNAM1 PNAM1

However, usually the oid of an object requested is not known. Therefore, for querying related objects, we provide attributes for symbolically naming elements and referencing related elements using these symbolic names. The attribute “link” is used to symbolically name any property element. The attribute “linkref” can be used with any property element to reference another symbolically named property element. If the link attribute value of a property and the linkref attribute value of another property are equal, then their property values must be equal in the query result. The value of a link attribute must be unique within a query. The following query returns the birthDate of all persons with fullName “Cherry Bingh”. Since birthDate and fullName are properties of different objects, the query


327

requests both objects. The objects in the query are linked using the link attribute in the oid property of person and linkref attribute in realPartyId attribute of personName. Cherry Bingh

The result of this query replaces the symbolic link “pers” with the real oid PERS1: PERS1 PDEM1 1965-03-01 PNAM1 Cherry Bingh PERS1

The examples in this section illustrate the simplicity and expressiveness of the XML-QBE language. The queries are simple templates, which are able to specify selection and linking across objects related by nesting and association. Other query features of XML-QBE not described here include the ability to select objects based on property expressions (operators: or, and, lt, le, ge, ge, eq, ne, exists, like), the ability to query multiple unrelated objects, and more complex queries on related object, which require exists, disjunction and union operations.

3 Related Work and Conclusion XQuery [7, 8] from the W3C consortium is a functional, expression-based language for querying XML data and documents as well as other kinds of structured data for which XML views can be defined, such as relational data. XQuery is a very rich language that can be used to create arbitrarily complex queries with complete generality. Two systems, XPERANTO [6] and SilkRoute [3] leverage such query languages to provide efficient, full-purpose and general XML query capabilities over relational database systems. While XPERANTO is based on the newer XQuery language, SilkRoute is based on one of its precursors, namely XML-QL [9]. In both cases, however, the focus is on providing frameworks for efficient processing of arbitrarily complex queries on XML views of relational data that are specified using the query language of the system, and not on developing a scheme for efficient

328

K. Bohrer et al.

querying by example. XQuery is a procedural language, and the syntax of XQuery is not XML. XQuery can be a target of XML-QBE. Several commercial systems, such as IBM DB2 XML Extender [4] and Oracle XML DB [5], provide the ability to store, manage and access XML as well as relational data by allowing the composition/decomposition of XML documents from/to a relational database system. While these systems are feature-rich and provide a lot of functionality in handling transformations between XML and relational data, their query functionality is extremely limited. This is primarily due to the fact that the focus of these systems is to allow the composition and decomposition of XML documents to/from relational data; not to query relational data per-se. For example, the IBM DB2 Extender allows only the query that is defined via the mapping file (data access definition file) which only allows the entire view to be recovered, not just any part that is the result of a query in question. The only way to allow such queries would be to dynamically update the mapping file with new SQL statements for each query that was to be executed, a cumbersome task requiring SQL knowledge. Zhang et al. [10] describe methods for doing queries-by-example in the context of performing simple queries for data in XML documents; they do not address the issue of performing complex queries over XML views of relational data. We have presented a new XML query language, called XML-QBE, which we believe is useful for retrieving information from both XML documents and backend databases. Its simple query by example form makes it easy to use as well as to process and analyze. The underlying data model is object-oriented, derived from UML. Our implementation is table-driven, based on a description of a backend database schema. Thus we have begun to address the problem of heterogeneous and legacy backends. More work needs to be done in this direction in order to better handle distributed databases with possibly inconsistent schemas.

References 1.

Bohrer, Liu, Kesdogan, Schonberg, Singh, Spraragen, “Personal Information Management and Distribution”, 4th International Conference on Electronic Commerce Research, Nov. 2001. 2. Bohrer, Kesdogan, Liu, Podlaseck, Schonberg, Singh, Spraragen. “How to go Window th Shopping on the World Wide Web without Violating the User’s Privacy”, 4 International Conference on Electronic Commerce Research, Nov. 2001. 3. M. Fernandez. W. Tan, D. Suciu (2000). “SilkRoute: Trading Between Relations and XML”. Proceedings of the 9th International World Wide Web Conference. 4. “IBM DB2 XML Extender”. http://www-3.ibm.com/software/data/db2/extenders/xmlext/ 5. “Oracle XML DB”. http://www.oracle.com/ip/index.html?xmldbcm_intro.html 6. J. Shanmugasundaram, J. Kiernan, E. Shekita, C. Fan and J. Funderburk (2001). “Querying XML Views of Relational Data”. Proceedings of the 27th VLDB conference. 7. “XQuery 1.0: An XML Query Language”. W3C Working Draft November 2002. http://www.w3.org/TR/xquery/. 8. “XML Query”. http://www.w3.org/XML/Query. 9. “XML-QL: A Query Language for XML”. W3C Submission, August 1998. http://www.w3.org/TR/NOTE-xml-ql/. 10. S. Zhang, J. Wang and K. Herbert (2002). “XML Query by Example”. International Journal of Computational Intelligence and applications. 2(3), 329–337.


Appendix

329

Automatic Generation of XML from Relations: The Nested Relation Approach Antonio Badia Computer Engineering and Computer Science department University of Louisville [email protected]

Abstract. We propose a method to generate XML documents from relational databases by using nested relational algebra. Starting with a specification of the structure of the XML document, and a description of the database schema, we give an algorithm to build automatically a nested algebra query that will generate a nested view from the relational database; this view is isomorphic to the desired XML document, that can easily be generated from it. We discuss limitations of (and extensions to) the framework.

1

Introduction

As the relational and object-oriented technologies are extended to capture XML documents, structured and semistructured data are found side by side in database applications [1]. Hence, a natural line of research is translating between the two environments, that is, producing relational data out of XML documents, and XML documents out of relational data. There has been considerable research on both aspects of the translation process [5,6,7,8,10]. Proposals to create database schemas from XML documents have received special attention, since such procedures can be used to store XML data in relational databases [2,4]. Proposals to create XML documents from relational data, on the other hand, have been less researched [8,10]. Such proposals usually include two components: a specification of the target XML document, and a description of the source relational data, in the form of a view or a SQL query over the relational database. In this paper, we show how it is possible, from the description of the target XML document, and information about the relational database schema (including integrity constraints), to generate the required view or query automatically. However, since XML data is hierarchical and relational data is flat, there is a gap that must be closed somehow. We propose the use of a nested relational algebra as a natural bridge between the two data models (the use of nested algebra, with different purposes, is explored in [4,8]). We give a procedure for generating a Nest-Project-Join (NPJ) algebra expression that generates an XML document with a given structure from a relational database. We investigate some issues involved in the process, like the loss of data from non-matched tuples in a join. ´ Pastor (Eds.): ER 2003 Workshops, LNCS 2814, pp. 330–341, 2003. M.A. Jeusfeld and O. c Springer-Verlag Berlin Heidelberg 2003

Automatic Generation of XML from Relations

331

Section 2 gives some background in XML and nested relational algebra (to make the paper self-contained). For lack of space, we do not mention related research except that which is directly related to approach. Section 3 explains the translation proposed, giving the algorithm (in Subsection 3.1) and some examples (in Subsection 3.2). Section 4 lists some issues that come up with this approach and provides solutions to them. Finally, Section 5 gives some conclusions and directions for further research.

2

Background

Because of space constraints, we assume the reader familiar with the basics of XML and of the (flat) relational algebra and concentrate, in this section, in introducing the ideas of the nested relational data model and algebra. The nested relational model is obtained by getting rid of the first normal form assumption, that states that tuple elements are atomic (without parts) values. By allowing set-based values, we can have relations as members of a tuple. Formally, let U = {A1 , ..., An } be a finite set of attributes. A schema over U ( and its depth) is defined recursively as follows (the following is taken from [11]): 1. If A1 , ..., An are atomic attributes from U , then R = (A1 , ..., An ) is a (flat) schema over U with the name R. depth(R) = 0. 2. If A1 , ..., An are atomic attributes from U , R1 , ..., Rm are distinct names of schema with the sets of attributes (denoted by attr(R1 ), ..., attr(Rm )) such that {A1 , ..., An } and {attr(R1 ), ..., attr(Rm )} are pairwise disjoint, then R = (A1 , ..., An , R1 , ..., Rm ) is a (nested) schema with the name R and R1 , ...Rm are called subschemas. depth(R) = 1 + maxm i=1 depth(Ri ). Clearly, we can see a nested schema as a tree, with the depth giving us the height of the tree. If the depth of a nested schema is 1, all of the subschemas in R are flat schemas, i.e., we have the traditional (flat) relational model. Let R denote a schema over a finite set U of attributes. The domain of R, denoted by DOM (R), is defined recursively as follows1 : 1. If R = (A1 , ..., An ), where Ai (1 ≤ i ≤ n) are atomic attributes, then DOM (R)=DOM (A1 )×...×DOM (An ), where ”×” denotes Cartesian product. 2. If R = (A1 , ...An , R1 , ..., Rm ), where Ai (1 ≤ i ≤ n) are atomic attributes and Rj (1 ≤ j ≤ m) are subschemas nested into R, then DOM (R)=DOM (A1 )× ... × DOM (An ) × 2DOM (R1 ) × ... × 2DOM (Rm ) , where ”×” denotes Cartesian product and 2DOM (Rj ) denotes the power set of the set DOM (Rj )(1 ≤ j ≤ m). 1

As is customary, we assume that any atomic attribute A has associated a non empty set DOM (A).

332

A. Badia

A nested tuple over R is an element of DOM (R). A nested relation r over R is a finite set of nested tuples over R. We say that sch(r) = R. The nested relational algebra, like the model, can be seen as an extension of (flat) relational algebra. Thus, the operations of selection, projection, Cartesian product and (for schemacompatible relations) union and difference can be defined in a similar manner to the flat case. On top of that, the algebra has two additional operators: nest and unnest. The nest operator ν is usually defined as νL (r), where L ⊆ sch(r) is a list of attributes; the intended semantics of the operator is to change the structure of the input relation so that all tuples that coincide in their L values constitute a single tuple. Formally, let N1 and N2 be subsets of sch(r). Then the nest of r with respect to N1 , νN1 (r), is defined as follows: νN1 (r) := {t | (∃w ∈ r)t[N1 ] = w[N1 ] ∧ t[N2 ] = {u[N2 ] | u ∈ r ∧ u[N1 ] = t[N1 ]} We will say that we nest on N2 and group by N1 . We point out that it is customary to write the nesting operator using N2 as the explicit argument; i.e., νN2 (r). We mention the grouping attributes explicitly because it will simplify the expressions that we need to write. The unnest operator µ can be seen as the opposite of the nest operator (i.e., it has the property that, for any relation r, L ⊆ sch(r), µL (νL (r)) = r). We will not further specify this operator since we do not use it here.

3

The Method

In [10], a method is proposed to create XML documents starting with relational data sources. In this method, a mapping from relational to XML is specified in the XML/SQL language, which specifies the structure of the target document in XML and adds a part specifying how to extract the needed information from the relational database; this element is an SQL query. Unfortunately, because the relational model is flat and XML is hierarchical, a certain impedance mismatch must be saved. The authors of [10] attempt to save it by stipulating, in the SQL query, how the data must be massaged to be transformed. For instance, for groupings, an ORDER BY clause is used, since one can iterate then over the resulting evaluation and produce the desired nested result. However, this mixes implementation details with specification details. In this paper, we propose a method to automate the SQL extraction from the relational database by generating a query in relational algebra from the XML specification of the target. To save the data model mismatch, we use nested relational algebra instead of the traditional (flat) algebra. Nested algebra provides a formal framework in which to state and investigate the problem, making possible to consider questions like expressivity and optimization without committing to a particular system or implementation. The idea of using nested relations to represent and manipulate XML data is already proposed in [4,8]. The work of [4] extends the nested data model with structural variants, that allow for irregular semistructured data to be represented


333

<SEQUENCE tagname=’’Department’’ > <SEQUENCE tagname=’’Course’’> Fig. 1. CONSTRUCTOR Clause

within a single relation. [4] also defines QSByE (Querying Semistructured Data by Example), which extends a version of nested algebra to their data model. However, [4] does not discuss using this paradigm to extract XML data from relational data. The work of [8] is to find any nesting that satisfies some general requirement; it proposes to generate all possible nesting from a given (flat) relation, and choose some of them based on heuristics (like avoiding any flattening by the keys, since that leaves a relation unchanged); there is no target XML used. Here, we use nested algebra to generate a restructured relation for a particular target. There are other approaches to generating XML from relational data; the SilkRoute approach ([3]) uses a language called RXL to define an XML view over the relational database; to process queries, XML-QL is used as input, and SilkRoute translates such queries to SQL over the relational database. In [9], a system is described that publishes relational data in XML format by evaluating queries in XQuery. We point out that these approaches take queries as input, while we take a description of the structure of the desired XML document. Our solution is conceived as an extension of the research in [10], and we will use the same examples as [10], to contrast our approach with the one established there. A relational database with schema Professor(IdProf,ProfName,IdDept), Department(IdDept,DeptName,Acr), Course(IdCourse,IdDept,CourseName), and ProfCourse(IdProf,IdCourse) is used as the source of data. The target is an XML document, which is declared in the XML part of XML/SQL in a CONSTRUCTOR clause; Figure 1 is an example of such clause. This clause, applied to the source database described, generates a target document that looks like the one in Figure 2. Several things are to be noted about the CONSTRUCTOR clause. First, the clause uses some particular tags, CONSTRUCT, LIST and ATOM, to indicate the structure of the resulting document. The names of the tags to be used in the XML document are indicated in the tagname attribute. Thus, this is not an XML Schema, but a meta-description from which to generate the document. Second, besides indicating the structure desired, the clause indicates how to obtain it, by indicating how data needs to be nested (with attribute nestby, that indicates an attribute in the source database), and how atoms (simple attributes) relate

334

A. Badia

1 Computer Science Database Systems Compilers 2 Philosophy Philosophy I Fig. 2. Example of XML Document <SQL idsql=’’v2’’> SELECT d.iddept, d.deptname, c.idcourse, c.coursename FROM Department d, Course c WHERE d.iddept = c.iddept ORDER BY d.iddept Fig. 3. SQL query

to the source data (with attribute source, that also indicates an attribute in the relational database). More importantly, the above clause needs a view, defined in SQL, from where to extract the information. Note that the attributes required come from more than one relation and need to be nested. The query used in the SQL part of XML/SQL for the example is shown in Figure 3. The query specifically joins the tables needed to provide the information, and projects out the required attributes. The ORDER BY clause is used to prepare the data for the nesting needed by the XML data; obviously the intention is for the data to be scanned once and the nesting be obtained by using the fact that the data is physically sorted. It is not difficult to see that this approach mixes implementation with logical details; for instance, having the data sorted is a way to prepare for the nesting (obviously sorting is one algorithm to implement the nest operator; another one would be hashing). Thus, one would like to separate logical from physical description (so that, for instance, a large collection could be nested by using hashing instead of sorting). However, it is possible to do better than that: one could get rid of the SQL specification (view description) completely, and generate one automatically from the XML target description and the database schema. Our goal in this paper is to develop a procedure for doing that.


3.1

335

The Algorithm

We take as input a relational database schema and a target specification in XML. In order to make the program more useful, we will allow regular XML Schema descriptions (i.e., describing the target directly, instead of giving a metadescription using keywords CONSTRUCT, LIST and ATOM). However, in order to connect the target with the source, we still require a source attribute in all simple elements in the XML description2 . Note, though, that we do not ask for any extra information (like explicit declarations of nesting attributes), since this information can be inferred from the XML description. Our method can be described in a few steps: first, we will parse the XML Schema description to extract the attributes involved and the nesting of the schema (if any). The nesting of the schema is obtained by annotating each attribute with an integer called the attribute’s level. Intuitively, if we look at the XML data as a tree, the level indicates the attribute’s depth in the tree. Second, we will generate a nested relational algebra expression by using the information yielded by the parsing. Finally, this expression, when applied to the database, yields a (possibly nested) table which is then used to generate the desired XML document. There are a few nuisances to be dealt with using this method, which are explained after the method has been detailed, next. We assume that the XML Schema target definition contains complex elements (with internal structure of type SEQUENCE) and simple elements. Every simple element has an attribute source, the value of which is an attribute name in the source database. The definition corresponds to an XML document, with a single root node, and a tree structure. There are two ways of writing an XML Schema declaration: the first style is an inlined style, in which information about every element (including its attributes and internal structure) is declared when the element is introduced. In the second style, elements are associated with references (names), and later on elements are described, using the reference to associate the description with the described element. Here we assume the first (inlined) style of description, which is more concise and easier to parse (however, it is easy to modify our algorithm to be used with schema declarations in the second style). Finally, we assume that we have the following information about the source relational database: for each relation, the schema (attribute names), and all integrity constraints (i.e., the primary key, any foreign keys, and their associations). We make the assumption that all relations in the database are connected to some other relation through exactly one foreign-key/primary key link (more on this assumption later). In the first step, we go over the XML definition and get a list of attributes and, for each attribute, its level. The level indicates the depth of the attribute in the tree, with the root being 0 and each node having a level of one more than the parent node. Intuitively, attributes at level 1 are flat (i.e., correspond exactly to those in a relational database), while attributes at a level more than 1 have been nested. The information is collected in a Target List, a sequence 2

Technically speaking, since simple elements cannot have attributes in XML, source should be considered a facet ([12]).

336

A. Badia

INPUT: an XML Schema definition. OUTPUT: Target List T PRECOND: the Schema definition is nested, with all complex type definitions inline. Each tag with simple name has a source attribute that names an attribute in the relational database T = empty; LIFO structure = empty; level = 0; While not end of Schema do if (open tag for complex type $t$ is found) level = level +1; add $t$ to LIFO; else if (open tag for simple type $t$ is found) T = add (source name of $t$, level) to T; add $t$ to LIFO; /* these are the only two choices */ else if (closing tag is found) pop corresponding matching tag off of LIFO; /* else the XML Schema is not well formed */ End While If (LIFO is empty) return T /* else the XML Schema is not well formed */ Fig. 4. Algorithm to Generate a Target List

{(a1 , l1 ), . . . , (an , ln )}, where ai (1 ≤ i ≤ n) is an attribute and li is a number > 0 called the level of attribute ai . Let T be a target list; the At(T ) (called the attributes of T ) is defined as {ai | (ai , li ) ∈ T } and LV (T ) (called the levels of T ) is defined as {li | (ai , li ) ∈ T } . Finally, AL(T, i) (called the attributes of T at level i) is defined as {a | (a, i) ∈ T } Using the information in T , we create a Nest-Project-Join (NPJ) expression as follows: we iterate over T , starting at the deepest level, and for each level i and the next level i − 1 an NPJ expression with exactly one nest operator is built. The process is repeated until we reach the uppermost level in T . Intuitively, when there are several levels of depth in the XML document, we must create a nesting for each level; simply joining all relations at once and nesting once over them would not give the required output. We give two examples to clarify the process. Formally, for database schema DB = (R1 , . . . , Rm ), let Rel(DB, T, i) = {R ∈ DB | ∃A ∈ sch(R) ∩ At(T, i)}. These are the relations with attributes appearing at level i. As stated above, we assume that all relations in Rel(DB, T, i) can be joined among themselves in exactly one way, for any i. Given a set of relations R, R will denote the appropriate join of all relations in R; hence our assumption implies that there is exactly one expression Rel(DB, T, i) (more on this assumption later). Moreover, we also assume that there is also a way to join relations in Rel(DB, T, i) and Rel(DB, T, i − 1), perhaps through some other relations in the database. We call such relations Conn(i, i − 1), and the attributes involved in the joins Att(Conn(i, i − 1)). As an example, assume T such that Rel(DB, T, 1) is {Department, P rof essor};


337

INPUT: Target List T, schema DB OUTPUT: an NPJ expression Let i = max(LV(T)); Let TEMP = REL(DB, T, i); if (i = 1) return πAL(T,i) (T EM P ) /* this case implies no nesting is needed */ while (i = 1) do { if (i > 2) Attrs = att(Conn(i-2,i-1)); else Attrs = ∅; TEMP = νAL(t,i)∪Attrs (πAL(T,i)∪Attrs∪(Al(T,i−1)) (LIN K(REL(DB, T, i − 1), T EM P ))) i = i -1; } /* end while */ return TEMP; Fig. 5. Algorithm to produce an NPJ expression

Rel(DB, T, 2) is {Course}; and Rel(DB, T, 3) is {P rof essor}. Then Conn(2, 3) is {P rof Course}; att(Conn(2, 3)) is {IdDept}, and conn(1, 2) is ∅. Finally, the expression LIN K(Rel(DB, T, i), Rel(DB, T, i − 1)) is to denote the join of: all relations in Rel(DB, T, i), joined among themselves; all relations in Rel(DB, T, i − 1)), joined among themselves; and the join of all relations in Conn(i, i − 1), joined among themselves. For instance, following our previous example, LIN K(Rel(DB, T, 2), REL(DB, T, 3)) = P rof Course IdP rof P rof essor; and Course IdCourse LIN K(Rel(DB, T, 1), REL(DB, T, 2)) = Department IdDept P rof essor IdP rof P rof Course IdCourse Course. The algorithm in Figure 5 uses these expressions to iterate over the target list. If the maximum depth in the target list is 1, there is no nesting, and hence we simply return a Project-Join expression. If the maximum depth ≥ 2, there is some nesting needed. We iterate starting at maximum depth i and the next depth i − 1 to create and NPJ expression for these two levels; this expression is reused in creating another NPJ expression for levels i − 1 and i − 2, and the process is repeated until we reach level 1. The variable Attrs is used to see which attributes are needed for the join at the next level (so, when looking at levels i and i − 1, we check the join between level i − 1 and i − 2), since it is necessary to have those attributes present in the NPJ expression for levels i and i − 1. 3.2

Examples

The algorithm outlined above seems more complicated than the real strategy being used. Let us illustrate the approach with a couple of examples from [10]. The first one is the example that we have used in the previous section; its target list is {(iddept1)(deptname1)(coursename2)} This example has maximum depth of 2 and hence will iterate over the loop only once. It is therefore the simplest example using nesting; it requires only one application of the nest operator. Once the relations involved are known

338

A. Badia

(in this case Rel(DB, T, 1) = {Department} and Rel(DB, T, 2) = {Course}, while Conn(1, 2) = ∅), joins based on the foreign-key/primary key relationship are established. Finally, looking at the attribute levels indicates the arguments needed by the nest operator: in this case, we nest by coursename and group by iddept and deptname. The corresponding NPJ expression is νiddept,deptname (πiddept,deptname,coursename (Department iddept Course) Note the correspondence between this expression and the SQL query used in [10]. A more complex example is example 2.3 of [10], which calls for a document made up of a list of department entries, each one containing the department name, a list of professors (names) in the department, and a list of courses offered by the department, each course element containing a list of the professors that teach the course for the department3 . The solution of [10] is to give two SQL views, one grouping the information by department and the other one grouping the information by department and course (the idsql attribute is used to identify the view from which a source should be obtained). Note, however, that this essentially involves taking the Cartesian product of both views. In our approach, taking the join of all relations and then nesting would not work, even if two copies of relation Professor were used. Rather, the translation proceeds in layers, creating an NPJ expression for each pair of adjacent levels. First, the target list is obtained, yielding {(DeptN ame1)(P rof N ame2)(CourseN ame2)(P rof N ame3)} Observe that ProfName appears twice, with different levels. The translation proceeds bottom-up, by first creating a NPJ expression for levels 2 and 3. However, in doing so it is necessary to take into account that the resulting relation will be used as input in another NPJ expression; the attributes needed for joining and nesting at the outer level are pushed down into this expression, resulting in νCourseN ame,IdDept (πCourseN ame,IdDept,P rof N ame (Course P rof essor P rof Course)) where IdDept is the attribute pushed because it is needed at the next level (note also that this join involved going through a many-to-many table). This is the second value of T EM P (after one iteration of the loop; the first one, previous to entering the loop, was simply ProfCourse). Then the process is repeated for the rest of the target list. The outer NPJ expression, then, is νDeptN ame (πDeptN ame,P rof N ame,T emp (Department P rof essor T emp) The resulting nested relation corresponds exactly to that of example 2.3 of [10], but is obtained as a single query. Once the expression is obtained, generating the XML document consists of two simple steps. First, the expression is applied to the database to yield a single (possibly nested) table. This table is then transformed into a list (sequence) of XML documents, each one of them with the structure given by the target XML Schema. One document is obtained from each row in the table. A simple 3

Obviously, there is redundancy on such a document, but a user may request the information in such a format!


339

attribute in the row corresponds to a first level, simple element in the document (specified in the XML attribute source), while a complex attribute in the row corresponds to a complex element in the document. The complex element contents are obtained by matching simple elements inside complex attributes in the relation to simple attributes in the XML element (again, using the source specification), and recursively constructing complex elements in XML from complex attributes in the relation. Given the way the nested relational algebra expression was constructed, there should be a 1-1 correspondence (an isomorphism, really) between each row in the relation (seen as a tree) and the XML document specification (seen as a tree).

4

Extensions

We have made a series of syntactic simplifications in our algorithm which are not problematic. For instance, a (somewhat) significant simplification is to assume that the join of two relations is always based on a foreign key/primary key relationship, and that such a relationship is unique between two relations. Certainly, if one wants to get rid of the assumption it would become necessary to indicate, somehow, how the underlying relations are to be joined. Three cases are possible: the relations are not joined in any way (in which case one has to use a Cartesian product), or the relations are joined by attributes other than the primary key and foreign key; or the relations are joined by a primary key/foreign key relationship, but there exists more than one such relationship between the tables. The first case is very easy to address; our algorithm can be extended so that when no connection is found between two tables, a Cartesian product is used in the NPJ expression, instead of a join. For the other two cases, there seems to be no good solution without further information about the database (like metadata of some sort); hence the user should provide such information in the XML document description. We do not address such an extension here, since we consider this situation extremely infrequent. A more substantive issue is the inadequacy of joins in some situations. Sometimes, a value in a primary key may have no matching value in a foreign key. In the example above, a department may not have any related courses in the database. In that case, the question is what to do with the unmatched element: should we show a department with no courses attached as part of the answer (with an empty sequence of courses), or should we not include such a department in the answer at all? XML Schema allows the user to specify what the intended result was by using minOccur in the specification of the target, by using a value of “0” or “1”. Intuitively, the user can distinguish between a ’*’ and a ’+’ situation (using DTD vocabulary) to define the intended answer. If minOccur is set to 1, then only departments offering courses should be shown. This is exactly the semantics of the procedure outlined above, since the join operator will only produce tuples from one relation (in this case, Department that have matches in the other relation, Course)4 . If minOccur is set to 0, then all departments 4

Note that 1 is the default value for minOccurs in XML Schema.

340

A. Badia

must be shown, whether they offer any courses or not (if they don’t, an empty sequence should be shown). The problem is how to specify this in the (nested) relational algebra, since the join operator will drop any tuple in a relation that has no match in the relation it is being joined to. The most straightforward answer is to use the outerjoin operator. In the previous example, a left outerjoin between Department (the left input) and Course (the right input) will create a result where departments with no courses associated will be preserved in a tuple padded with nulls on the Course attributes. The presence of nulls, however, creates a problem for the nesting operator: if we try to nest by an attribute with nulls, what should the semantics of the operation be? An easy solution to this problem is to observe that, when outerjoin is used to preserve some information, we usually want to nest by the null attributes and group by the non-null ones (as it is the case in our example, since we nest by coursename, and group by iddept, deptname). Thus, one can stipulate that, in such a case, the nesting should produce a tuple with iddept, deptname values and an empty relation (i.e., an empty set) associated with them. From this, the XML document should produce a element with values for elements , <deptname> and no value for complex element . There are other several issues that need to be addressed to overcome the limitations of our method. We are currently working on some of them, which can only briefly mention here for lack of space. Obviously, by using only NPJ expressions we are limited in the views that we can create. Adding selections would be an obvious extension; the main issue here how to add selection conditions to the XML Schema specification of the output. On the other hand, our XML documents do not use the choice (|) or optional (“?”) constructors (the original examples in [10] did not use them either). One problem with using such operators is that it is difficult to make them correspond to anything in the relational algebra; proposed translations from XML to relations have difficulties dealing with both ([5]). One possible approach is based on the presence of null values. Finally, the assumption that each simple component in the XML document must come from one and only one source attribute can be relaxed at the cost of some complexity in the XML specification.

5

Conclusions and Further Research

In this paper we have proposed a method to generate XML documents from a (flat) relational database using only a description of the target in XML Schema. The nested relational model is used as the bridge between flat relations and hierarchical XML data. Our algorithm parses the target description in XML to obtain all needed information without the user having to explicitly declare it. One of the advantages of this approach is that it automates the somewhat tedious task of creating a view (or SQL query) for each XML document that we want to define; the system takes care of such task automatically, letting the user concentrate on the specification of the desired output document.


341

The approach focuses on nest-project-join expressions; clearly, there are several possible extensions, which we are currently exploring, including dealing with choice and optional operators in the XML description, adding selections to the algebra expression, and allowing more complex matches between attributes in the relational tables and elements in XML. Acknowledgments. The author wishes to thank the anonymous reviewers for their helpful feedback.

References 1. S. Abiteboul, P. Buneman and D. Suciu Data on the Web: From Relations to Semistructured Data and XML, Morgan Kaufman, 1999. 2. A. Deutsch, M. Fernandez, and D. Suciu Storing Semistructured Data with STORED, in Proceedings of the ACM SIGMOD Conference, 1999. 3. M. Fernandez, A. Morishima, D. Suciu and W. Tan, Publishing Relational Data in XML: the SilkRoute Approach, in IEEE Data Engineering Bulletin, no. 24(2), 2001. 4. A. S. da Silva, Irna Evangelista Filha, Alberto H. F. Laender and David W. Embley Representing and Querying Semistructured Web Data Using Nested Tables with Structural Variants, in Proceedings of ER 2002. 5. M. Mani, D. Lee, and R. R. Muntz Semantic Data Modeling Using XML Schemas, in Proceedings of ER 2001. 6. D. Lee and W. Chu, CPI: Constraint-Preserving Inlining Algorithm for Mapping XML DTD to Relational Schema, Data and Knowledge Engineering journal, volume 39, 2001. 7. D. Lee and W. Chu Constraints-Preserving Transformation from XML Document Type Definition to Relational Schema, in Proceedings of ER 2000. 8. D. Lee, M. Mani, F. Chiu, and W. Chu Nesting-Based Relational-to-XML Translation, in Int’l Workshop on Web and Databases (WebDB), 2001. 9. J. Shanmugasundaram, J. Kiernan, E. J. Shekita, C. Fan and J. Funderburk Querying XML Views of Relational Data, in Proceedings of VLDB 2001. 10. C. Vittori, C. Dorneles and C. Heuser Creating XML Documents from Relational Data Sources, in Proceedings of EC-Web, 2001. 11. Vossen, G. Data Models, Database Language and Database Management Systems, Addison-Wesley, 1991. 12. Extensible Markup Language (XML) 1.0, Bray, T. and Paoli, J. and SperbergMcQueen, C.M. (eds), W3C Recommendation, http://www.w3.org/TR/REC-xml-20001006, edition 2.

Toward the Automatic Derivation of XML Transformations Martin Erwig Oregon State University School of EECS [email protected]

Abstract. Existing solutions to data and schema integration require user interaction/input to generate a data transformation between two different schemas. These approaches are not appropriate in situations where many data transformations are needed or where data transformations have to be generated frequently. We describe an approach to an automatic XML-transformation generator that is based on a theory of information-preserving and -approximating XML operations. Our approach builds on a formal semantics for XML operations and their associated DTD transformation and on an axiomatic theory of information preservation and approximation. This combination enables the inference of a sequence of XML transformations by a search algorithm based on the operations’ DTD transformations.

1

Introduction

XML is rapidly developing into the standard format for data exchange on the Internet, however, the combination of an ever growing number of XML data resources on the one hand, and a constantly expanding number of XML applications on the other hand, is not without problems. Of particular concern is the danger of isolated data and application “islands” that can lead users to perceive a prodigious supply of data that is often inaccessible to them through their current applications. This issue has been observed and extensively addressed in previous work in data integration, for example, [8,14,6,7,19,13] and more recently in schema integration and query discovery [21,24,15,16]. So far, however, all the proposed solutions require user input to build a translation program or query. Even more troubling, since each different data source requires a separate transformation, the programming effort grows linearly with the number of data sources. In many cases this effort is prohibitive. Consider the following scenario. An application to evaluate the publication activities of researchers accepts XML input data, but requires the data to be of the form “publications clustered by authors”. A user of this system finds a large repository of bibliographic data, which is given in the format according to the DTD shown in Figure 1 on the left. In the following, we will refer to the corresponding XML data as bib. The application cannot use these data because ´ Pastor (Eds.): ER 2003 Workshops, LNCS 2814, pp. 342–354, 2003. M.A. Jeusfeld and O. c Springer-Verlag Berlin Heidelberg 2003

Toward the Automatic Derivation of XML Transformations

343

byAuthor author*> author (name,(book|article)*)> book title> article (title,journal)> name (#PCDATA)> title (#PCDATA)> journal (#PCDATA)>

Fig. 1. DTD of available data and DTD of required data.

bibliographic entries are not grouped by authors. What is needed is a tool that can transform bib into a list of author elements, each containing a sublist of their publications. Such a format is shown in Figure 1 on the right. Although tools are available that support the transformation, they sometimes require non-trivial programming skills. In almost all cases they require some form of user interaction. In any case, users might not be willing to invest their time in generating one-time conversion tools. Moreover, if the integration of several different data sources should be required to create several different transformations, the programming or specification effort quickly becomes untenable. An intrinsic requirement is that these transformations be “as information preserving as possible”. In the best case the generated transformation preserves the information content completely, but in many instances transformations that lose information are also sufficient. For example, if an application requires only books with their titles, a transformation that “forgets” the author information of an XML document works well. Our solution to the described problem can be summarized as follows: First, identify an algebra of information-preserving and information-approximating XML transformations. In particular, these operations have a precisely defined type, that is, an associated schema transformation for DTDs. By induction it then follows that if we transform a DTD d into a DTD d by a sequence of these elementary XML transformations, the same sequence of operations transforms an XML value of DTD d lossless or approximating into an XML value of DTD d . The second step is then to define a search algorithm that constructs a search space of DTDs by applying algebra operations and find a path from a source DTD d to the required target DTD d . The path represents the sequence of operations that realize the sought transformation. There might be, of course, cases in which the automatic inference does not work well. The situation is comparable to that of search engines like Google that do not always find good matches due to a lack of semantics or structure associated with the query keywords. Nevertheless, search engines are among the most valuable and most frequently used tools of the Internet since they provide satisfactory results in practice. For the same reasons, automatic integration tools, although not complete, might be valuable and useful tools in practice. This paper presents the proposed approach through examples. Due to space limitations we have to restrict ourselves to the description of a small number of elementary XML operations that can be employed in generated transformations

344

M. Erwig

and also a subset of axioms for information approximation. Nevertheless, we will be able to demonstrate the automatic generation of an XML transformation within this restricted setting. The rest of this paper is structured as follows. In Section 2 we will discuss related work. In Section 3 we will formally define the problem of XMLtransformation inference. In Section 4 we axiomatize the notions of information preservation and approximation. In Section 5 we define what it means for an XML transformation to be DTD correct. In Section 6 we introduce basic XML transformations that will be used as building blocks in Section 7 in the inference of complex XML transformations. Finally, Section 8 presents some conclusions.

2

Related Work

Related work has been performed in two areas: (i) schema matching and query discovery and (ii) data semantics and information content. Schema Matching and Query Discovery. Approaches for matching between different data models and languages are described in [19,2,3]. Data integration from an application point of view is also discussed, for example, in [8,6,14,13]. We will not review all the work on data integration here because data integration is traditionally mainly concerned with integrating a set of schemas into a unified representation [22], which poses different challenges than translating between two generally unrelated schemas. A more specific goal of schema matching is to identify relationships between (elements) of a source and a target schema. Such a mapping can then be used to deduce a transformation query for data. The Cupid system [15] focuses exclusively on schema matching and does not deal with the related task of creating a corresponding data transformation/query. The described approach combines different methods used in earlier systems, such as MOMIS [4] or DIKE [20]. The Clio system [9] is an interactive, semi-automated tool for computing schema matchings. It was introduced for the relational model in [16] and was based on so-called value correspondences, which have to be provided by the user. In [24] the system has been extended by using instances to refine schema matchings. Refinements can be obtained by inferring schema matchings from operations applied to example data, which is done by the user who manipulates the data interactively. User interaction is also needed in [21] where a two-phase approach for schema matching is proposed. The second phase, called semantic translation, is centered around generating transformations that preserve given constraints on the schema. However, if few or even no constraints are available, the approach does not work well. It has been argued in [16] that the computation of schema matchings cannot be fully automated since a syntactic approach is not able to exploit the semantics of different data sources. While this is probably true for arbitrarily complex matches, it is also true that heuristic and linguistic tools for identifying renamings can go a long way [12,5]. Certainly, quality and sophistication of


345

transformations can be increased by more semantic input. However, there is no research that could quantify the increase/cost ratio. So it is not really known how much improvement is obtained by gathering semantics input. The approach presented in this paper explores the extreme case where users cannot or are not willing to provide input, which means to provide fully automatic support for data transformation. Information Content. A guiding criterion for the discovery of transformations is the preservation (or approximation) of the data sources to which the transformations will be eventually applied. Early research on that subject was performed within relational database theory [10,11] and was centered around the notion of information capacity of database schemas, which roughly means the set of all possible instances that a schema can have. The use of information capacity equivalence as a correctness criterion for schema transformations has been investigated in [17,18]. In particular, this work provides guidelines as to which variation of the information capacity concept should be applied in different applications of schema translation. One important result that is relevant to our work is that absolute information capacity equivalence is too strong a criterion for the scenario “querying data under views”, which is similar in its requirements to data integration. In other words, those findings formally support the use of information approximation in transformation inference.

3

Formalization of Transformation Inference

In the following discussion we make use of the following notational conventions. denote Symbols x, x , y, z XML elements (also called XML values) lists of XML elements , d, d DTDs tags t, u t XML elements with tag t t[x1 . . . xk ], t[] XML elements with tag t and subelements x1 . . . xk (or ) Sometimes we want to refer to a subelement without caring about the exact position of that element. To this end we employ a notation for XML contexts: Cx stands for an XML element that contains somewhere a subelement x. Similarly, C represents an XML element that contains a list of subelements. This notation is particularly helpful for expressing changes in contexts. To simplify the discussion, we do not consider attributes or mixed content of elements in the following. Now we can describe the problem of XML-transformation inference precisely as follows. We are given an XML data source x that conforms to a DTD d (which is written as x : d), but we need the data in the format described by the DTD d . Therefore, we are looking for an XML transformation f that, when applied to x, yields an XML value x that conforms to the DTD d (that is, f (x) : d ) and contains otherwise as much as possible the same information as x. This last

346

M. Erwig

condition can be expressed by defining a partial order on XML values ≺ that formalizes the notion of having less information content. A slight generalization of the problem is to find transformations f with the described property without knowing x. We can express the problem mathematically as follows. P (d, d ) = {f | ∀x.x : d =⇒ f (x) : d ∧ f .f (x) : d ∧ f (x) ≺ f (x)} P defines the set of all transformations f that map an XML value conforming to d to a value conforming to d and also have the property that there is no other transformation f with that property that preserves more information content. The generalized definition reflects the application when the DTD d of the XML data source is known, but the (possibly very large) XML document x has not been loaded (yet). In the following we consider this second case since it subsumes the previous one.

4

Information Preservation and Information Approximation

We formalize the concepts of information preservation and approximation by defining corresponding relations on XML trees. These relations are induced by operations on XML values. We consider here the renamings of tags and regrouping as an information-preserving operation and the deletion of elements as an information-approximating operation. This limitation is not really a problem since the whole theory is generic in the axiomatization of information preservation/approximation, which means that the set of chosen operations does not affect the overall approach. Formally, two elements that have non-matching tags, such as x = a and x = a, are considered to be different. However, if we rename the tag t in x to u, both elements become identical. We write {t → u} for a renaming of t to u and {t → u}(x) for the application of the renaming to the element x. It happens quite frequently that the same data are named differently by different people. For example, we might find bibliographic data sources that wrap the author information by tags , , , and so on. With regard to the information contained in the XML value, the actual choice of individual tag names does not really matter. Therefore, we can consider a broader kind of equality “up to a tag renaming r”, written as ≡r . For example, under the renaming {t → u} the elements x and x are equal, which we could express, for example, by x ≡{t→u} x . This is because {t → u}(x) = x . We must be careful not to rename with a tag that is already in use in the element to be renamed. For example, if we renamed to , the meaning of the bibliographic data from Section 1 would change. In general, a renaming r can consist of a set of tag renamings, which means that r is a function from old tags to new tags. These two sets can be extracted from a renaming by dom(r) and rng(r), respectively. We can formalize the equivalence of DTDs modulo renamings by a rule like ren≡ shown in Figure 2. In this and the rules to follow, r denotes an arbitrary


ren≡

rng(r) ∩ tags(x) = ∅ x ≡ r x

r(x) = x

cong≡

x1 ≡r y1

...

347

xk ≡r yk

t[x1 . . . xk ] ≡r t[y1 . . . yk ]

grp≡

del

Ct[1 ] . . . t[k ] ≡r Ct[1 ] . . . Ct[k ] x1 r y1 ... xk r yk cong Cx r C t[x1 . . . xk ] r t[y1 . . . yk ]

Fig. 2. Axiomatic definition of information content and approximation

(set of) renaming(s). The first premise of the rule prevents name clashes by requiring fresh tags in renamings. The function tags computes the set of all tags contained in an XML element. We also have to address the fact that some renamings are more reasonable than others, for example, {name → aname} is more likely to lead to equivalent schemas than, say {name → price}. In the described model, any two structurally identical DTDs can be regarded as equivalent under some renaming. This leads to equivalence classes that are generally too large. In other words, schemas that would not be considered equivalent by humans are treated as equivalent by the model. This will be particularly evident when the tags used in the source and target DTD are completely or mostly different. This problem can be addressed by defining an ordering on renamings that is based on the number and quality of renamings. A cost or penalty can be assigned to each renaming based on its likeliness. For example, names that are “similar” should be assigned a relatively low cost. Measures for similarity can be obtained from simple textual comparisons (for example, one name is the prefix of another), or by consulting a thesaurus or taxonomy like WordNet [1]. Synonyms identified in this way should also have a low penalty. In contrast, any renaming that has no support, such as {name → price}, receives a maximum penalty. With this extension we can measure any equivalence d ≡r d by a number, which is given by the sum of the penalties of all renamings in r. Later, we can use this measure to select the “cheapest” among the different possible transformations by favoring a few, well-matching renamings. Renaming is the simplest form of extending verbatim equality to a form of semantic equivalence. As another example, consider a structural equivalence condition that is obtained from the observation that an element x with tag u containing k repeated subelements with tag t is a grouped or factored representation of the association of each t-element with the rest of x. Therefore, it represents the same information as the corresponding “de-factored” or “ungrouped” representation as k u-elements each containing just one t-element. For instance, the following element on the left represents (in a factored way) the same information as the two elements shown on the right.

348

M. Erwig Principia Math. Russel Whitehead

Principia Math. Russel Principia Math. Whitehead

In general, an element Ct[1 ] . . . t[k ] contains the same information as the list of elements Ct[1 ] . . . Ct[k ]. This idea can be captured by the axiom grp≡ shown in Figure 2. Finally, we also need congruence rules to formalize the idea that if elements x and x contain the same information, then so do, for example, the elements t[x] and t[x ]. This is achieved by the rule cong≡ shown in Figure 2. This approach for formalizing the notion of information equivalence by a set of axioms and rules provides a sound basis for judging the correctness of inferred transformations. In a similar way, we can axiomatize the notion of information approximation. For instance, deleting a subelement from an element x yields a new element x that contains fewer information than x but agrees otherwise with x. This idea can be expressed by the axiom del shown in Figure 2 where we also give a congruence rule cong for information approximation. Since the definition of approximation is an extension of equivalence, we also have to account for renamings in the predicate r .

5

DTD Correctness of XML Transformations

DTDs can be formally defined by extended context-free grammars. Non-recursive DTDs can be represented simply by trees, that is, they can be represented essentially in the same way as XML values. This tree representation simplifies the description of DTD transformations. Note that in this representation * and | occur as tags. For example, the DTD for bib can be represented by the following tree. bib[*[|[book[title, *[author]], article[title, *[author], journal]]]] Representing DTDs as trees means that we can re-use the tree operations we have already defined for XML values. The complexity of the resulting notation can be simplified by abbreviating *[e] by e∗ and |[e, e ] by (e|e ) so that we can recover most of the original DTD notation: bib[(book[title, author∗ ] | article[title, author∗ , journal])∗ ] A DTD transformation is given by a function that maps a DTD d to another DTD d . For each XML transformation f , we can consider its corresponding DTD transformation, for which we write f . Depending on the language in which


349

f is defined and on the formalism that is used to describe DTDs and DTD transformations, there might exist zero, one, or more possible DTD transformations for f . The DTD transformation f that corresponds to an XML transformation can also be considered as f ’s type, which is expressed by writing f : d → d if f (d) = d . Formally relating DTD transformations to the transformations of the underlying XML values is achieved by the notion of DTD correctness, that is, an XML operation f : d → d is defined to be DTD correct if f applies to x

=⇒

∀x : d.f (x) : d

In other words, DTD correctness means that the DTD transformation f that is associated with an operation f is semantically meaningful, that is, it reflects correctly the DTD transformation for each underlying XML value. (We can write the condition also as: ∀x : d.f (x) : f (d).)

6

Basic XML Transformations

The feasibility of the automatic XML-transformation inference hinges to a large part on the ability to express complex XML transformations as compositions of a small set of simple operations, which we call basic operations. The design of these basic operations is guided by the following criteria. All basic operations must (a) be information preserving or information approximating, (b) have a clearly specified DTD transformation, and (c) be DTD correct. Why do we require these properties? Item (a) ensures that inferred transformations do not change the information contained in XML data or at most lose information, but never introduce new information. Properties (b) and (c) will ensure that the inference, which is directed by DTDs, yields transformations of XML values that conform to these DTDs. The notion of DTD transformations and correctness will be explained below. Next we consider three basic XML transformations that have been designed guided the just mentioned criteria: renaming, product, and deletion. Renaming. The rename operation α takes a renaming r = {t1 → u1 , . . . , tk → uk } with ui = ti for 1 ≤ i ≤ k and applies it to all tags in an XML element x. We require that the new tags ui do not occur in x. r(x) if rng(r) ∩ tags(x) = ∅ αr (x) = x otherwise Let us check the design constraints for this operation. For information preservation we require that the XML value obtained by the operation in question is equivalent to the original XML value. In the case of renaming we therefore require αr (x) ≡r x, which follows directly from the axiom ren≡ shown in Figure 2. The DTD transformation that corresponds to renaming can be described by: αr : d → r(d)

350

M. Erwig

which means that α transforms an XML value conforming to a DTD d into a value whose DTD is obtained by renaming tags according to r. The proof of DTD correctness can be performed by induction over the syntactic structure of the DTD transformation. Product. Another basic operation is the operation π for de-factoring XML elements. We also call this operation product since it essentially computes a combination of an element with a list of its subelements. The tag t of the subelement to be considered is a parameter of π. πt (u[Ct[1 ] . . . t[k ]]) = u[Ct[1 ] . . . Ct[k ]] The additional root tag u is needed in the definition to force the repetition to apply below the root element. We assume implicitly in this and all other definitions that operations leave all XML values unchanged that do not match the pattern of the definition. In the case of π this means that for any element x that does not contain repeated t-subelements we have πt (x) = x. Again we can check the properties of the operation π. First, information preservation follows from the axiom grp≡ and the congruence rule cong≡ shown in Figure 2. The type of π is: π t : u[Ct∗ ] → u[Ct ∗ ] DTD correctness can again be shown by induction. Deletion. As an example for an information-approximating operation, consider the XML transformation δt that deletes a sequence of t-subelements (on one level) from an XML element. It can be defined as follows. δt (Ct[1 ] . . . t[k ]) = C Obviously, δ is not information preserving, but it is information approximating, which can be proved using the axiom del from Figure 2. The type of δ can be described succinctly by re-using the context notation for XML trees. δ t : Ct∗ |t → C As for the other XML transformations, DTD correctness can be proved by induction. To summarize, for all the basic operation ω defined, we have the following property. ∀x.x : d =⇒ ω(x) : ω(d) ∧ (∃r.x ≡r ω(x) ∨ ω(x) r x) That is, each basic operation ω is: (1) DTD correct and (2a) information preserving or (2b) information approximating (recall that ω denotes the DTD transformation of ω).


7

351

Transformation Inference

A very simple, although effective, initial approach is to build a search space of DTDs starting from the DTD of the source document, say d, by repeatedly applying all matching operations until the target DTD, say d , is reached. By “matching operations” we mean basic operations whose argument type have d as an instance. In the search we always favor following paths along information-preserving operations over information-approximating operations. Whenever we apply α we take tags(d ) as a pool from which to draw new names. We also have to ensure not to repeatedly apply inverse renamings to prevent running into infinite search paths. Once we have reached d by this procedure, the path from d to d in this search space corresponds to a sequence of basic XML transformations ω1 , . . . , ωk whose composition f = ωk · . . . · ω1 is the sought transformation of type d → d . This is because we are using only DTD-correct transformations. If all basic operations ωi are information preserving, then so is the transformation f . If at least one ωi is information approximating, then so is f . If we are not able to generate d , the algorithm stops with an error. To illustrate the transformation inference by an example, consider the task of creating a list of title/author pairs for books from the bib element. This means to find a transformation from the DTD d for bib bib[(book[title, author∗ ] | article[title, author∗ , journal])∗ ] into the following DTD d . bookAuthors[book[title, author]∗ ] First, since the tag bookAuthors is not contained in the source DTD d, we know that we have to apply αr with r = {bib → bookAuthors}. Next, we can apply δarticle because its type matches with the context C1 = bookAuthors[(book[title, author∗ ] | )∗ ] However, we might also apply πauthor by choosing, for example, the following context (note that u = bookAuthors). C2 = (book[title, author∗ ] | article[title, , journal])∗ (Alternatively, we could also match author∗ in the book element.) Nevertheless, we choose to apply δ because it is simpler, which is somehow indicated by the smaller context C1 . We could also try to apply δbook to delete the book element, which, however, does not seem to make any sense because we then “lose” a tag of the target DTD. After having applied δarticle , we have reached the DTD described by the context C1 . Now it makes sense to apply πauthor . Before we do this, however, we simplify C1 according to a rule d| = d to remove the now

352

M. Erwig

unnecessary | constructor. So the context for the application of πauthor is (with u = bookAuthors): C3 = book[title, ∗ ]∗ The resulting DTD after the application of πauthor is bookAuthors[(book[title, author]∗ )∗ ] A final simplification through the rule (d∗ )∗ = d∗ [23] yields the target DTD. The inference process has therefore generated the transformation f = πauthor · δarticle · α{bib→bookAuthors} The description is a bit simplified, because in order to apply the operations in f to some XML value, we need all the contexts that were determined during the inference process. Treating these contexts here like implicit parameters, we can now apply f to bib and obtain the desired XML value. With two additional operations for lifting elements upward in XML trees and grouping elements according to common subelements, we can describe the XML transformation that is required for the example given in Section 1. Designing these operations so that they are DTD correct and information preserving/approximating and making transformation inference powerful enough to discover them is part of future work.

8

Conclusions

The fast growing number of Web applications and available information sources carries the danger of creating isolated data and application islands because the distributed nature of the Internet does not enforce the use of common schemas or data dictionaries. Our approach aims at avoiding these data islands and to promote the free flow and integration of differently structured data by developing a system for the automatic generation of XML transformations. Our approach differs from previous efforts since we aim at a fully automated transformation discovery tool where user interaction is not required a priori. It will not, however, rule out any additional input the user is willing to provide. As one example, user-defined renamings can be easily integrated into our approach by setting penalties for these renamings to zero. In other words, users can interact if they want to, but are not required to do so.

References 1. WordNet: A Lexical Database for the English Language. http://www.cogsci.princeton.edu/˜wn/. 2. S. Abiteboul, S. Cluet, and T. Milo. Correspondence and Translation for Heterogeneous Data. In 6th Int. Conf. on Database Theory, LNCS 1186, pages 351–363, 1997.


353

3. P. Atzeni and R. Torlone. Schema Translation between Heterogeneous Data Models in a Lattice Framework. In 6h IFIP TC-2 Working Conf. on Data Semantics, pages 345–364, 1995. 4. S. Bergamaschi, S. Castano, and M. Vincini. Semantic Integration of Semistructured and Structured Data Sources. SIGMOD Record, 28(1):54–59, 1999. 5. M. W. Bright, A. R. Hurson, and S. Pakzad. Automated Resolution of Semantic Heterogeneity in Multidatabases. ACM Transactions on Database Systems, 19(2):212–253, 1994. 6. V. Christophides, S. Cluet, and J. Simèon. On Wrapping Query Languages and Efficient XML Integration. In ACM SIGMOD Conf. on Management of Data, pages 141–152, 2000. 7. S. Cluet, C. Delobel, J. Siméon, and K. Smaga. Your Mediators Need Data Conversion! In ACM SIGMOD Conf. on Management of Data, pages 177–188, 1998. 8. A. Eyal and T. Milo. Integrating and Customizing Heterogeneous E-Commerce Applications. VLDB Journal, 10(1):16–38, 2001. 9. L. M. Haas, R. J. Miller, B. Niswonger, M. T. Roth, P. M. Schwarz, and E. L. Wimmers. Transforming Heterogeneous Data with Database Middleware: Beyond Integration. Bulletin of the IEEE Computer Society Technical Committee on Data Engineering, 22(1):31–36, 1999. 10. R. Hull. Relative Information Capacity of Simple Relational Database Schemata. SIAM Journal of Computing, 15(3):856–886, 1986. 11. T. Imielinski and N. Spyratos. On Lossless Transformation of Database Schemes not Necessarily Satisfying Universal Instance Assumption. In 3rd ACM SIGACTSIGMOD-SIGART Symp. on Principles of Database Systems, pages 258–265, 1984. 12. P. Johannesson. Linguistic support for Analysing and Comparing Conceptual Schemas. IEEE Transactions on Knowledge and Data Engineering, 21(2):165–182, 1997. 13. A. Y. Levy, A. Rajaraman, and J. J. Ordille. Querying Heterogeneous Information Sources Using Source Descriptions. In 22nd Int. Conf. on Very Large Databases, pages 251–262, 1996. 14. B. Lud¨ ascher, Y. Papakonstantinou, and P. Velikhov. Navigation-Driven Evaluation of Virtual Mediated Views. In 7th Int. Conf. on Extending Database TechnologyEuropean, LNCS 1777, pages 150–165, 2000. 15. J. Madhavan, P. A. Bernstein, and E. Rahm. Generic Schema Matching with Cupid. In 27th Int. Conf. on Very Large Databases, pages 49–58, 2001. 16. R. J. Miller, L. M. Haas, and M. A. Hern` andez. Schema Mapping as Query Discovery. In 26th Int. Conf. on Very Large Databases, pages 77–88, 2000. 17. R. J. Miller, Y. Ioannidis, and R. Ramakrishnan. The Use of Information Capacity in Schema Integration and Translation. In 19th Int. Conf. on Very Large Databases, pages 120–133, 1993. 18. R. J. Miller, Y. Ioannidis, and R. Ramakrishnan. Schema Equivalence in Heterogeneous Systems: Bridging Theory and Practice. Information Systems, 19(1):3–31, 1994. 19. T. Milo and S. Zohar. Using Schema Matching to Simplify Heterogeneous Data Translation. In 24th Int. Conf. on Very Large Databases, pages 122–133, 1998. 20. L. Palopoli, G. Terracina, and D. Ursino. Towards the Semi-Automatic Synthesis of Cooperative Information Systems and Data Warehouses. In ADBIS-DASFAA Symp. on Advances in Databases and Information Systems, pages 108–117, 2000. 21. L. Popa, Y. Velegrakis, R. J. Miller, M. A. Hern` andez, and R. Fagin. Translating Web Data. In 28th Int. Conf. on Very Large Databases, 2002.

354

M. Erwig

22. S. Ram and V. Ramesh. Schema Integration: Past, Current and Future. In A. Elmagarmid, M. Rusinkiewicz, and A. Sheth, editors, Management of Heterogeneous and Autonomous Database Systems, pages 119–155. Morgan Kaufman, 1999. 23. J. Shanmugasundaram, K. Tufte, G. He, C. Zhang, D. DeWitt, and J. Naughton. Relational Databases for Querying XML Documents: Limitations and Opportunities. In 25th Int. Conf. on Very Large Databases, pages 302–314, 1999. 24. L. L. Yan, R. J. Miller, L. M. Haas, and R. Fagin. Data-Driven Understanding and Refinement of Schema Mappings. In ACM SIGMOD Conf. on Management of Data, 2001.

VACXENE: A User-Friendly Visual Synthetic XML Generator 1

1

Khoo Boon Tian , Sourav S Bhowmick , and Sanjay Madria

2

1

School of Computer Engineering, Nanyang Technological University, Singapore [email protected] 2 Department of Computer Science, University of Missouri-Rolla, Rolla, MO 65401 [email protected]

Abstract. Synthetic collections of valid XML documents are useful in many XML applications. However, creating test cases for XML applications by manually from DTDs can be tedious and may not cover all possible or required instances. In this paper, we present VACXENE (VisuAl synthetiC Xml gENErator), a Java-based tool that creates test cases for XML applications by generating random instances of valid XML documents from a single DTD. XML generator provides a user-friendly GUI that allows us to control the appearance of the XML output by imposing user-defined constraints. The paper presents an overview of the various features supported by VACXENE and reports some preliminary results regarding performance.

1 Introduction XML has emerged as the dominant standard for representing and exchanging data over the Internet. When compared with other mark-up languages such as HTML, the main advantage of XML is that each XML document can have a Document Type Definition (DTD) associated with it. A DTD serves as an implicit semantic schema for the XML document and makes it possible to define much more powerful queries than what is possible with simple, keyword-based text retrievals. Also, XML’s nested, self-describing structure provides a simple yet flexible means for applications to model and exchange data. For example, a business can easily model complex structures such as purchase orders in XML form. As another example, all of Shakespeare’s plays can be marked up and stored as XML documents. Overall, XML can serve at least two roles [6]. First, as a new markup language, a web browser can browse an XML file in the same way as an HTML file. Second, XML can serve as a standard way of storing semi-structured data sets. XML makes it possible for users to ask very powerful queries against the web. Consequently, a great deal of research recently has focused on storing, indexing, and querying XML data [6]. One of the critical steps in XML research is the performance evaluation of new techniques for storing, indexing and querying XML data. It is imperative to have access to large XML data sets with widely varying characteristics for gaining insights into the performance of proposed techniques on different kinds of XML data. However, using only real XML data can be very limiting for three reasons [2]. First, there is not much publicly available XML data at this time. Second, all the real XML M.A. Jeusfeld and Ó. Pastor (Eds.): ER 2003 Workshops, LNCS 2814, pp. 355–365, 2003. © Springer-Verlag Berlin Heidelberg 2003

356

K.B. Tian, S.S Bhowmick, and S. Madria

data that we have encountered has relatively simple structure. Using more complex XML data can provide better insights, even if this data is synthetic. Third, like all real data, we have very little control over the characteristics of real XML data. Synthetically generated data has always been important for evaluating and understanding new ideas in database research. Synthetic data generators allow users to generate large volumes of data with well-understood characteristics. One can easily vary the characteristics of the generated data by varying the input parameters of the data generator. This allows users to systematically cover much more of the space of possible data sets than relying solely on real data over which users have little or no control. As such, using synthetic data for evaluating research ideas and testing the performance of database systems can provide users with deeper insights and stronger conclusions than relying solely on real data. Of course, while experimenting with synthetic data is an ideal way to explore the behavior of different solutions on data with different characteristics, an additional validation step may be necessary to ensure that the conclusions drawn from synthetic data extend to real world applications. In this paper, we describe a data generator for generating synthetic XML data called VACXENE1 (VisuAl synthetiC Xml gENErator) that allows a high level of control over the properties of the generated XML data using a small number of parameters. The simple and intuitive nature of the data generation parameters means that the characteristics of the generated XML data will be easy to understand, even though this data may not necessarily resemble any available real data. This data generator is certainly not the ultimate solution to the problem of generating synthetic XML data, but we have found it very useful in our research on XML data management, and we believe that it can also be useful to other researchers. Given a DTD, VACXENE generates tree-structured XML documents of arbitrary complexity. It uses the information provided by the user to generate one or more XML documents with variety of characteristics, and it also generates values for the elements and attributes in these documents. It can generate both data-centric and document-centric XML data. The data generator has a user-friendly GUI for inputting various input parameters and visualizing the generated synthetic data set.

2 Related Work Recently, there has been several works in the area of generating synthetic XML data. In [5], synthetic XML data is used to evaluate different strategies for storing XML in relational database systems. The XML data used is extremely simple in characteristics and consists of elements at one level with no nesting. The elements are randomly connected in a graph structure using IDREF attributes. This graph-structured view of XML data is useful in some contexts, but XML data is by nature tree structured, and it may often be useful to have a tree-structured view of this data. Furthermore, the data generation process of [5] has very few opportunities for varying the structure and distribution of the generated data. In [4] and [7], two benchmarks are proposed for evaluating the performance of XML data management systems. Both benchmarks use synthetic XML data that 1

The name also implies that our data generator is a “vaccine” for the “cure” of lack of diverse characteristics in real XML data.

VACXENE: A User-Friendly Visual Synthetic XML Generator

357

models data from high-level applications: a database of structured text documents and a directory of these documents in [4], and data about on-line auctions in [7]. The structure of the data in both cases is fixed and simple, and there is very little opportunity for varying it. This kind of data may be adequate for a benchmark that serves as a standard yardstick for comparing the performance of XML data management systems. However, if we wish to evaluate a particular XML data management system and gain insights into its performance, then using XML data with widely varying structure over which we have more control can be more useful. IBM provides a data generator that generates XML data that conforms to an input DTD [1]. Like the previous approaches, the IBM data generator is limited in the control it provides over the data generation process. For example, we cannot control the number of words nested in an element. This is important, as document-centric XML as opposed to data-centric XML have order and may contain few sentences in an element and one may wish to generate such documents. A general purpose synthetic XML documents generator is presented in [2]. The data generator can generate XML documents of arbitrary complexity. It generates XML elements and values within these elements, but it does not currently handle the generation of attributes. The data generator starts by generating a tree called the path tree that represents the structure of the XML data. The data generator assigns tag names to the nodes of this tree, and specifies the frequency distribution of the XML elements represented by these nodes. It uses the information in this tree to generate one or more XML documents, and it also generates values for the elements in these documents. It does not use a DTD or XML-schema to generate the documents and hence may not reflect meaningful XML documents always. ToXgene [3] is a template-based generator for large, consistent collections of synthetic XML documents, developed as part of the ToX (the Toronto XML Server) project. It was designed to be declarative, and produces fairly complex XML content. The ToXgene Template Specification Language (TSL) is a subset of the XML Schema notation augmented with annotations for specifying certain properties of the intended data, such as value distributions, the vocabulary for CDATA content, etc. It also allows different elements (or attributes) to share CDATA literals, thus allowing the generation of references among elements in the same (or in different) documents. This enables the generation of collections of correlated documents (i.e., documents that can be joined by value). ToXgene also allows the specification of most common integrity constraints (e.g., uniqueness) over the data in such lists; thus, one can generate consistent ID, IDREF and IDREFS attributes. In contrast to the proposals for generating synthetic XML data in [4, 7], our data generator can generate much more complex data, and it provides much more control over the characteristics of the generated data. Unlike in [2], we use a DTD to generate synthetic data and hence we can generate meaningful elements and attributes. VACXENE also allows us to choose element and attribute values from specific dictionary or domain knowledge. This enables us to generate XML data that contains keywords from a particular domain. Compared to [3], we provide mechanism to create data-centric as well as document-centric XML. We also provide a user-friendly GUI so that novice users can easily specify input parameters and generate synthetic documents and visualize them effectively. Nevertheless, it may be possible to use ideas from these proposals to extend our data generator. For example, IDREF attributes may be used to connect the elements of the generated documents as in [5].

358


Also, different probability distributions and element sharing can be supported as in [3]. Next, we describe the different steps of generating synthetic XML data, and we point out the input parameters that control each step.

3 Specifying Input Parameters In this section, we discuss the various input parameters specified in VACXENE for generating synthetic XML documents. The XML Generator has many input features that help us to make it powerful but yet easy to use. These features provide the user with a wide range of functionality. We now elaborate on these functionalities. 3.1 Specifying the DTD The first step for creating synthetic valid XML documents in VACXENE is DTD specification. The generated synthetic document set will satisfy the input DTD. VACXENE can parse any DTD files provided by the user. The DTD will be validated to ensure that it conforms to the recommendations provided by the World Wide Web Consortium (W3C). If the DTD is not well-formed, an error message will inform the user of the error detected and the parsing process will be terminated. If the DTD is correctly validated, its content will be displayed. Upon completion of the parsing process, the XML Generator will generate the tree structure of the DTD. As shown in Figure 1, the tree shows the hierarchical view of the contents of the DTD. It allows the user to have a clearer picture of the DTD structure by displaying the relationship between different nodes, the cardinality of various nodes and their corresponding attributes. User can decide on the amount of details he needs to view, by expanding or collapsing the tree. From the DTD tree, the user can also specify the node parameters for each individual node.

3.2 Specifying the Level of Synthesis Input parameters control the level of complexity of the generated XML. Whenever a DTD is parsed, a set of default parameter values will be generated. User can control these default parameters through the use of the “level of synthesis” option. These options are catered to suit the different needs of various users. The three available options are “none”, “moderate” and “high”, with the “high” option producing the most complex documents. The semantics of these three types of complexity is given in Table 1. The default option is “moderate”. Table 1. Level of Synthesis Complexity Low Moderate High

Document Depth Fixed Varies Varies

Document Width Fixed Fixed Varies

Nodes Cardinality Maximum 1 Maximum 1 No Limit


359

Cardinality of Node

Root Node

Name of Node Property of Attribute Attribute of Node

Fig. 1. DTD Tree Structure

3.3 Parameters to Control Characteristics of XML Documents A range of input parameters is available to allow users to vary the characteristic of the generated XML documents. These parameters play a very important role as they affect the level of complexity and increase the randomness of the generated documents. The parameters are defined into two groups, Main Parameters and Node Parameters. Main Parameters consists of parameters, which affect the structure of the XML document as a whole. Node Parameters consists of parameters, which affect only a specified node and all its children nodes. Figure 3 shows a screenshot of the GUI for input parameter specification. The following is a list of Main Parameters, currently supported by VACXENE: o Minimum and Maximum Depth: Depth measures the number of levels of the XML document, starting from the root element. The root level is considered as level 0. The depth value of each document is a random number generated between the minimum and maximum depth value specified by the user. o Minimum and Maximum Width: Width measures the maximum number of children permitted at any level of the XML document. This width value is a random number generated between the minimum and maximum width value specified by the user. A value of zero for width is not allowed. o Scaling: This parameter specifies the number of children that the root element can have. In another words, scaling decides the number of level 1 element. Intuitively, scaling factor denotes the length of the XML document. o XML Files: This parameter is an indication of the number of synthetic XML documents to be created. o Enable Dictionary: Random dictionary words were generated to act as values for the elements in the XML documents. This parameter defines whether the generated document should contain any random dictionary words. The length of the generated values and their frequency can be controlled. This feature can be disabled.

360


Recursive Node

Recursion Rate Parameter

Fig. 2. Node Parameters of Recursive Node

o

o o

Size of Long Value: Elements with values that exceed a certain length is considered as Long Value. This parameter specifies the size of element values in order to consider them as Long Value. For example, if the value specified is 10, any element values that are of more than 10 words will be considered as Long Value. Percentage Elements with Long Value: This parameter defines the percentage number of elements in the document that will contain Long Values. Document Name: This parameter allows the user to define the file name of the generated XML documents.

We can also specify constraints at node level of XML documents by inputting Node Parameters. By clicking on a particular node in the DTD tree structure in Figure 2 5 Node Depth = 1

Node Depth = 2

Fig. 3. Example of Node Depth


361

and filling up the Node Parameters, we may control a subtree of the XML data. The following is a list of Node Parameters supported in VACXENE: Minimum and Maximum Cardinality: Cardinality measures the number times an element will appear. This cardinality value is a random number generated between the minimum and maximum cardinality parameter value. The cardinality of any element is restricted by the cardinality symbol specified in the DTD. Minimum and Maximum Fan Out: Fan out determines the maximum number of children a particular element can have. It is similar to scaling except that it can be applied to any elements in the XML. The fan out value is a random number generated between the minimum and maximum fan out parameter value. Minimum and Maximum Node Depth: Node depth determines the maximum number of levels in each sub-tree and affects the complexity of the synthetic XML document. This node depth value is a random number generated between the minimum and maximum node depth parameter value. For example in Figure 3, node 5 has a node depth of 2.

5

1

5

1

5 5

1

Recursion

Fig. 4. Example of Recursive Node Recursion

3.4 Specifying Recursive Nodes VACXENE supports the recursion of nodes. However, only recursive nodes with a cardinality symbol of “?” or “*” can be accepted. This is to avoid the possibility of infinite recursion. Recursive nodes have different node parameters as compared to ordinary nodes. They do not have fan out or node depth parameters. Instead, they have a recursion rate parameter. Recursion rate determines the number of times a recursive node can iterate. Recursion of nodes can be demonstrated by the example in Figure 4. Node 5 of the DTD tree is a recursive node and we can control the number of times it can recurse using the recursive rate parameter. Figure 2 is a screenshot of a recursive node and its node parameters. As seen in the Figure, node Staff is a recursive node. Upon the selection of node Staff, the recursion rate parameter will

362


appear as one of the node parameters and allow the user to specify the number of times this recursive node can iterate. 3.5 Conflict Detection and Rectification In certain situation the constraints imposed on a node (using Node Parameters) may conflict with the parameters set on the whole XML document (using the Main Parameters). For example if the maximum depth of the node is set to 5 and if the maximum height of the tree is set to 3 then obviously these two input parameters conflict with one another. This conflict may also occur due to the constraints imposed by the input DTD. Hence, it is necessary detect such conflicts and ask the user to rectify the problem. VACXENE can automatically detect such conflicts and inform users the conflict that may occur due to their choice of input parameters. Whenever a conflict has occurred, an interactive status message will appear to allow the user to respond to the conflict. In the previous section we discussed how to specify various input parameters in VACXENE to vary the characteristics of the synthetic XML documents. Given a well-formed DTD and a set of user specified input parameters, the XML Generator will generate synthetic XML documents. The generated documents will comply strictly to the DTD and the various input parameters. User can also specify the name of the XML documents and the number of documents to be generated as part of the input.

4 Visualization Synthetic Document Set Upon the generation of the XML documents, the user can view the contents of any of these documents and their corresponding tree structure. As in the case of DTD tree structure, the XML tree can also be expanded or collapsed. To enable the user to personalize the XML documents, the contents of the displayed documents can be edited. However if the editing of the contents leads to a conflict with the DTD, an error will occur and the changes will not be saved. VACXENE also provides the user with the option to validate the generated documents so as to ensure that the documents follow the structure specified by the DTD and satisfy all the input conditions. The screenshot in Figure 5 shows the Document Validation Panel. Document validation is an optional feature that allows the user to validate the generated documents against their DTD and the input parameters. User can choose a range of documents to be validated or use the “Validate All” checkbox to indicate that all generated documents are to be validated. The status of the validation will then be displayed on the status screen.


363

Range of XML Document to be Validated

Checkbox for Validating all XML Document

Validation Status

Button to Start Validation

Fig. 5. Document Validation Panel

5 Experimental Results In this section, we discuss the results of preliminary experiments with VACXENE. We explore how the three major factors (complexity of the documents, the number of documents generated and the use of dictionary) affect the performance of the system. For our experiment we use 8 different data sets for same input DTD. We vary the following three input parameters: level of synthesis, number of documents to be generated, and the usage of dictionary. We compute the run time of our tool. Table 2 summarizes the results. Table 2. Results Number of Documents 10 10 10 10 100 100 100 100

Complexity Moderate High Moderate High Moderate High Moderate High

Use of Dictionary No No Yes Yes No No Yes Yes

RUN-TIME Result (sec) 0.841 0.900 1.632 2.374 3.836 4.517 12.197 17.335

364


Result Graph 20 18

Run-time in seconds

16 14 Moderate, No

12

Moderate, Yes

10

High, No

8

High, Yes

6 4 2 0 10

100 Number of documents

Fig. 6. Result Graph

The results showed that the run-time of VACXENE increases as the complexity of the document is increased. The use of dictionary and an increase in the number of documents generated will also increase the run-time. From the results, we can also conclude that the use of dictionary plays a very big role in the run-time of the XML Generator. The use of the dictionary will increase the run-time by at least 2 fold. The effect of the dictionary will even be higher if the complexity of the documents is high. For moderate complexity documents, the use of dictionary will increase run-time by 94%. The use of dictionary for high complexity documents will lead to an 163% increase in run-time. The result obtained from the tests was as expected. Whenever dictionary is accessed, the XML Generator will access the dictionary file and randomly pick a word. This process will take up a certain amount of execution time. For example, if a document consists of 20 elements and each element have values of length 15 words, a total of 20 X 15 = 300 random words will have to be generated. This will lead to 300 accesses to the dictionary. If 10 documents are generated, a total of 300 X 10 = 3000 dictionary accesses will be made. The number of dictionary accesses will be further increased when the complexity of the documents increased as higher complexity documents will usually have more elements.

6 Conclusions and Future Work In this paper we introduced VACXENE, a data generator for generating synthetic complex-structured XML data, which can be of use to researchers in XML data management. VACXENE has been implemented using Java. The data generator has several input parameters that control the characteristics of the generated data. The parameters all have simple and intuitive meanings, so it is easy to understand the structure of the generated data. We presented an overview of our tool, which is based


365

on DTD, and discussed how various input parameters can easily be specified in VACXENE to generate XML documents with wide variety of characteristics. Finally, we reported on preliminary experiments we conducted with our tool. Development of VACXENE is continuing. It can easily be extended and modified to allow for different methods of data generation not covered in this paper. Areas for possible extension include, among others, generating data that conforms to a given XML schema, element sharing, and support of various probability distributions. We also intend to provide a mechanism to allow the generation of text according to different vocabularies, grammars, and character encoding schemes. This would be of great importance for generating testing data for text-intensive applications [3].

References 1. 2. 3.

4. 5. 6. 7.

IBM XML generator. http://www.alphaworks.ibm.com/tech/xmlgenerator. Aboulnaga, J. F. Naughton, and C. Zhang. Generating synthetic complex-structured XML data. In Proceedings of the Fourth International Workshop on the Web and Databases, pages 79–84, Santa Barbara, CA, USA, May 24–25 2001. Denilson Barbosa, Alberto Mendelzon, John Keenleyside and Kelly Lyons. ToXgene: a template-based data generator for XML. In Proceedings of the Fifth International Workshop on the Web and Databases (WebDB 2002). Madison, Wisconsin – June 6–7, 2002. Timo Böhme and Erhard Rahm. XMach–1: A benchmark for XML data management. In Proc. German Database Conference (BTW2001), Oldenburg, Germany, March 2001. Daniela Florescu and Donald Kossmann. Storing and querying XML data using an RDBMS. IEEE Data Engineering Bulletin, 22(3):27–34, September 1999. Feng Tian, David DeWitt, Jianjun Chen and Chun Zhang. The Design and Performance Evaluation of Alternative XML Storage Strategies. Technical Report, University of Wisconsin, Madison, USA. Albrecht Schmidt, Florian Waas, Martin Kersten, Daniela Florescu, Ioana Manolescu, Michael J. Carey, and Ralph Busse. The XML benchmark project. Technical Report INSR0103, CWI, April 2001.

A New Inlining Algorithm for Mapping XML DTDs to Relational Schemas Shiyong Lu, Yezhou Sun, Mustafa Atay, and Farshad Fotouhi Department Of Computer Science Wayne State University Detroit, MI 48202 {shiyong,sunny,matay,fotouhi}@cs.wayne.edu

Abstract. XML is rapidly emerging on the World Wide Web as a standard for representing and exchanging data. It is critical to have efficient mechanisms to store and query XML documents to exploit the full power of this new technology. While one approach is to develop native XML repositories that support XML data models and query languages directly, the other approach is to take advantage of the mature technologies that are provided by current relational or object-relational DBMSs. There is active research along both approaches and it is still not clear which one is better than the other. We continue our effort on the second approach. In particular, we have developed an efficient algorithm which takes an XML DTD as input and produces a relational schema as output for storing and querying XML documents conforming to the input DTD. Our algorithm features several significant improvements over the shared-inlining algorithm including overcoming its incompleteness, eliminating redundancies caused by shared elements, performing optimizations and enhancing efficiency.

1

Introduction

With the trend of increasing amount of XML documents on the World Wide Web, it is critical to have efficient mechanisms to store and query XML documents to exploit the full power of this new technology. As a result, various XML query languages have been proposed such as XML-QL [7], XQL [14], Lorel [12] and XML-GL [5], and more recently XQuery [6], and XML has become one of the most active research fields attracting different researchers from various communities. Currently, two approaches are being investigated for storing and querying XML data. One approach is to develop native XML repositories that support XML data models and query languages directly. This includes Software AG’s Tamino [2] and eXcelon’s XIS [1], among others. The other approach is to take advantage of the mature technologies that are provided by current relational or object-relational DBMSs. The major challenges of this approach include: (1) XML data model needs to be mapped into the target model such as the relational model; (2) queries posed in XML query languages need to be translated into ones ´ Pastor (Eds.): ER 2003 Workshops, LNCS 2814, pp. 366–377, 2003. M.A. Jeusfeld and O. c Springer-Verlag Berlin Heidelberg 2003

A New Inlining Algorithm for Mapping XML DTDs to Relational Schemas

367

in the target query languages such as SQL or OQL; and (3) the query results from the target database engines need to be published back to XML format. Recently, Kurt and Atay have performed an experimental study to compare the efficiency of these two approaches [13]. However, since both approaches are still under active research and development, it is too early to conclude which one is better than the other. Related work. Several mechanisms have been proposed to store XML data in relational or object-relational databases [8] [10] [16] [11] and publish relational or object-relational data as XML data [15] [4] [9]. Some of them use XML DTDs [16] and others consider situations in which DTDs are not available [8] [10] [11]. Two recent evaluations [19] [11] of different XML storage strategies indicate that the shared-inlining algorithm [16] overperforms other strategies in data representation and performance across different datasets and different queries when DTDs are available. In this paper, we propose a new inlining algorithm that maps XML DTDs to relational schemas. Our algorithm is inspired by the shared-inlining algorithm [16] but features several improvements over it. We will discuss these improvements in Section 3.3. Organization. The rest of the paper is organized as follows. Section 2 gives a brief overview of XML Document Type Definitions (DTDs), Section 3 describes our new inlining algorithm that maps an input DTD to a relational schema in terms of three steps: (1) simplifying input DTDs (Section 3.1); (2) creating and inlining DTD graphs (Section 3.2); (3) generating relational schemas (Section 3.3). The section ends with a discussion of the improvements we have made over the shared-inlining algorithm, which is considered as the best strategy when DTDs are available [19] [11]. A full evaluation and comparison is underway and will be presented in the near future. Section 3.4 illustrates the three steps of our algorithm using a real input DTD, and demonstrates how XML documents conforming to the DTD can be stored. Finally, Section 4 concludes the paper and provides some directions for future work.

2

XML DTDs

XML Document Type Definitions (DTDs) [3] describe the structure of XML documents and are considered as the schemas for XML documents. In this paper, we model both XML elements and XML attributes as XML elements since XML attributes can be considered as XML elements without further nesting structure. A DTD D is modeled as a set of XML element definitions {d1 , d2 , · · · , dk }. Each XML element definition di (i = 1, · · · , k) is in the form of ni = ei , where ni is the name of an XML element, and ei is a DTD expression. Each DTD expression is composed from XML element names (called primitive DTD expressions) and other DTD subexpressions using the following operators: – Tuple operator. (e1 , e2 , · · · , en ) denotes a tuple of DTD subexpressions. In particular, we consider (e) is a singleton tuple. The tuple operator is denoted by “,”.

368

S. Lu et al.

– Star operator. e∗ represents zero or more occurrences of subexpression e. – Plus operator. e+ represents one or more occurrences of subexpression e. – Optional operator. e? represents an optional occurrence (0 or 1) of subexpression e. – Or operator. (e1 | e2 | · · · | en ) represents one occurrence of one of the subexpressions e1 , e2 , · · ·, en . We ignore the encoding mechanisms that are used in data types PCDATA and CDATA and model both of them as data type string. The DOCTYPE declaration states which XML element will be used as the schema for XML documents. This XML element is called the root element. However, we assume that arbitrary XML elements defined in the DTD might be selected, inserted, deleted and updated individually. We define a DTD expression formally as follows. Definition 1. A DTD expression e is defined recursively in the following BNF notation where n range over XML element names and e1 , · · ·, en range over DTD expressions. e ::= string | n | e+ | e∗ | e? | (e1 , · · · , en ) | (e1 | · · · |en ) where the symbol “::=” should be read as “is defined as” and “|” as “or”.

3

Mapping XML DTDs to Relational Schemas

In this section, we propose a new inlining algorithm that maps an input DTD to a relational schema. The algorithm contains the following three steps: 1. Simplifying DTDs. Since a DTD expression might be very complex due to its hierarchical nesting capability, this step greatly simplifies the mapping procedure. 2. Creating and inlining DTD graphs. We create the corresponding DTD graph based on the simplified DTD, and then inline as many descendant elements as possible to an XML element. In contrast to the shared-inlining algorithm, our inlining rules eliminate the redundancy caused by shared elements in the generated relational schema and can deal with arbitrary input DTDs including those that contain arbitrary cycles. 3. Generating relational schemas. After a DTD graph is inlined, we generate a relational schema based on it. We describe these three steps in Sections 3.1, 3.2 and 3.2 respectively and conclude the section by a discussion of the improvements we have made over the shared-inlining algorithm. Finally, Section 3.4 illustrates these steps using a real XML DTD and demonstrates how XML documents conforming to this DTD can be stored based on the generated schema.


369

e+ → e∗ . e? → e. (e1 | · · · | en ) → (e1 , · · · , en ). a) (e1 , · · · , en )∗ → (e∗1 , · · · , e∗n ). b) e∗∗ → e∗. 5. a) · · · , e, · · · , e, · · · → · · · , e∗ , · · · , · · ·. b) · · · , e, · · · , e∗ , · · · → · · · , e∗ , · · · , · · ·. c) · · · , e∗ , · · · , e, · · · → · · · , e∗ , · · · , · · ·. d) · · · , e∗ , · · · , e∗ , · · · → · · · , e∗ , · · · , · · ·.

1. 2. 3. 4.

Fig. 1. DTD simplification rules

3.1

Simplifying DTDs

Most complexity of a DTD comes from the complexity of DTD expressions such as . However, as far as an XML query language is concerned, what matters is the siblings and parentchild relationships between elements. We apply the transformation rules listed in Figure 1 in the given order: 1. 2. 3. 4.

Apply rule 1 recursively and the resulting DTD will not contain +. Apply rule 2 recursively and the resulting DTD will not contain + and ?. Apply rule 3 recursively and the resulting DTD will not contain +, ? and |. Apply rules 4(a) and 4(b) recursively and the resulting DTD will take the form (e1 , e2 , · · · , en ). Each ei = e or e∗ (i = 1, · · · , n) where e is an element name. Therefore, a DTD is in some flattened form after this step. 5. Apply rules 5(a), 5(b), 5(c) and 5(d) recursively and the resulting DTD will take the form (e1 , e2 , · · · , en ) such that each ei contains distinct element name.

From an XML query language’s point of view, two pieces of information are essential: (1) The parent-child relationships between XML elements; and (2) the relative order relationships between siblings. The above transformation maintains the former but not the later. Fortunately, we can introduce an ordinal attribute for each generated relation to encode the order of XML elements when an XML element (and its containing subelements) is inserted into the database, so that any XML query conforming to the input DTD can be evaluated over the generated relational schema. Example 1. Use the above simplification procedure, one can transform to a simplified version . The following theorem indicates that our simplification procedure is complete and in addition, the resulting DTD expression is a tuple of element names or their stars.

370

S. Lu et al.

Theorem 1. Our DTD simplification procedure is complete in the sense that it accepts every input DTD and each resulting DTD expression is in the form of (e1 , e2 , · · · , en ) where ei = e or e∗ (i = 1, · · · , n), e is an element name and each ei contains a distinct XML element name. Proof. We omit the proof since it is obvious. Discussion. Compared to the transformation rules defined in the shared-inlining algorithm [16], we made several improvements over it: – Completeness. Our rules consider all possible combinations of operators and XML elements whereas the shared-inlining algorithm only lists some important combinations. For example, there is no rule that corresponds to (e1 | · · · | en )? in the shared-inlining algorithm. – Efficiency. We enforce the application of the rules in the order given. Earlier rules totally transform away some operators from the input DTD, and in each step, the number of rules to be matched is greatly reduced. This improves the efficiency of the simplification procedure significantly. – Further simplification. We observe that the role of “?” corresponds to the notion of nullable column in the relational table. We transform away “?” and this greatly simplifies the resulting DTD graph (to be described in the next subsection) since it does not contain “?” any more. 3.2

Creating and Inlining DTD Graphs

In this step, we create the corresponding DTD graph based on the simplified DTD, and then inline as many descendant elements to an element as possible. The rationale is that these inlined elements will eventually produce a relation. Therefore, we only inline a child c to a parent p when p can contain at most one occurrence of c in order to avoid introducing redundancy into the generated relation. Theorem 1 indicates that after the simplification procedure, any input DTD is now in a canonical form, i.e., each DTD expression is a tuple of distinct element names or their stars. As a result, in the corresponding DTD graph, each node represents an XML element, and each edge represents an operator of ’,’ or ’*’. Our inlining procedure considers the following three cases. 1. Case 1: Element a is connected to b by a ,-edge and b has no other incoming edges. In other words, b is a non-shared node. In this case, a can contain at most one occurrence of b, and we will combine node b into a while maintaining the parent-child relationships between b and its children. 2. Case 2: Element a is connected to b by a ,-edge but b has other incoming edges. In other words, b is a shared node. We do not combine b into a in this case since b has multiple parents. 3. Case 3: Element a is connected to b by a *-edge. In this case, each a can contain multiple occurrences of b element, and we do not combine b into a. Only case 1 allows us to inline an element to its parent. We define the notion of inlinable node as follows.

A New Inlining Algorithm for Mapping XML DTDs to Relational Schemas *

a

d

* c, d

a, b b

*

371

c

*

A

B

a b g

c

a, b, c , d d

g

e e, f f

C

D

Fig. 2. Inlining DTD graphs

Definition 2. Given a DTD graph, a node is inlinable if and only if it has exactly one incoming edge and that edge is a ,-edge. Definition 3. Given a DTD graph and a node e in the graph, node e and all other inlinable nodes that are reachable from e by ,-edge constitute a tree (since we assume a DTD graph is consistent, thus there is no ,-edge cycle in the graph). This tree is called the inlinable tree for node e (it is rooted at e). Example 2. In Figure 2.A, nodes b and d are inlinable but nodes a and c are not inlinable. The inlinable tree for a contains nodes a and b, whereas the inlinable tree for c contains nodes c and d. In Figure 2.C, nodes b, c, d and f are inlinable, but nodes a, e and g are not inlinable. The inlinable tree for a contains nodes a, b, c and d, and the inlinable tree for node e contains nodes e and f . The notion of inlinable tree formalizes the intuition of “inlining as many descendant elements as possible to an element”. We illustrate our inlining algorithm in pseudocode in Figure 3. Essentially, it uses a depth-first-search strategy to identify the inlinable tree for each node and then inline that tree to its root. A field inlinedSet of set type is introduced for each node e to represent the set of XML element nodes that has been inlined to this node e (initially e.inlinedSet = {e}). For example, in Figure 2.C, after the inlining procedure, a.inlinedSet = {a, b, c, d}. The algorithm is efficient as indicated in the following theorem. Theorem 2 (Complexity). Our inlining algorithm can be performed in O(n) where n is the number of elements in the input DTD. Proof. This is obvious since each node of the DTD graph is visited at most once.

372

S. Lu et al.

Algorithm Inline(DTDGraph G) Begin For each node e in G do If not visited(e) then InlineNode(e) End If End For End Algorithm InlineNode(Node e) Begin Mark e as “visited” For each child c of e do If not visited(c) then InlineNode(c) End If End For For each child c of e do If inlinable(c) then e.inlinedSet ∪ = c.inlinedSet; assign all children of c as the children of e and then delete c from G End If End For End Fig. 3. The inlining procedure

Example 3. Using our inlining procedure given in Figure 3, the DTD graph shown in Figure 2.A will be inlined into one shown in Figure 2.B, and the DTD graph shown in Figure 2.C will be inlined into one shown in Figure 2.D. We observe that after our inlining algorithm is applied, a DTD graph has the following property: nodes are connected by ,-edge or *-edge and ,-edge must point to a shared node. This observation is the basis of the final step of the algorithm: generating relational schemas. 3.3

Generating Relational Schemas

After a simplified DTD graph is inlined, the last step is to generate a relational schema based on this inlined DTD graph. The generated schema supports the select, insert, delete and update [18] of an arbitrary XML element declared in the input DTD. The following four steps will be performed on the inlined DTD graph to generate a set of relations. 1. For each node e, a relation e is generated with the following relational attributes.


373

a) ID is the primary key, and for each XML attribute A of e, a corresponding relational attribute A is generated with the same name. b) If | e.inlinedSet | ≥ 2, we introduce attribute nodetype to indicate the type of the XML element stored in a tuple. c) The names of all the terminal XML elements in e.inlinedSet. Since a non-terminal XML element is stored with values for ID and nodetype and the storage of the XML subelements it contains, no additional attribute is needed for it (this will become more clear later). d) If there is a ,-edge from e to node c, then introduce c.ID as a foreign key of e referencing relation c. 2. If there are at least two relations t1 (ID) and t2 (ID) generated by step 1, then we combine all the relations of the form t(ID) into one single relation table1(ID, nodetype) where nodetype indicates which XML element is stored in a tuple. 3. If there are at least two relations t1 (ID, t1 ) and t2 (ID, t2 ) generated by step 1, then we combine all the relations of the form t(ID, t) into one single relation table2(ID, nodetype, pcdata) where nodetype indicates which XML element is stored in a tuple. 4. If there is at least one ∗ edge in the inlined DTD graph, then we introduce relation edge(parentID, childID, parentType, childType) to store all the parent-child relationships corresponding to *-edges. The domains of parentType and childType are the set of XML element names defined in the input DTD. Essentially, step 1 converts each node e in the inlined DTD graph into a separate relation e. If there are some other XML element nodes that have been inlined to it (i.e., | e.inlinedSet | ≥ 2), relation e will be used to store all these XML elements, and attribute nodetype will be introduced to indicate which XML element is the root for each tuple. Since step 1 might produce a set of relations in the forms of t(ID) and t(ID, t), Step 2 and 3 optimize them by performing a horizontal combining of them into table1(ID, nodetype) and table2(ID, nodetype, pcdata). These optimizations reduce the number of target relations and will facilitate the mapping from XML operations to relational SQL operations. Finally, one single relation edge(parentID, childID, parentType, childType) stores all the many-to-many relationships between arbitrary two XML elements. Although our inlining algorithm is inspired by the shared-inlining algorithm, we made several significant improvements over it: – Completeness. Our algorithm is complete in the sense that it can deal with any input DTDs including arbitrary cyclic DTDs. The shared-inlining algorithm defines a rule to deal with two mutually recursive elements and it is not clear how a DTD with a cycle involving more than two elements is handled (see Figure 4.A for such an example). In addition, the shared-inlining algorithm checks the existence of recursion explicitly, we do not need to do this checking and cycles are dealt with naturally. – Redundancy elimination for shared nodes. A node is shared if its in-degree is more than one. Our algorithm deals with shared nodes differently from the

374

S. Lu et al. papers

paper

*

* * *

authors

conference journal *

* report, references

*

author,name,institute

A

B

literature * book * part *

telephone dept *

* faculty

*

* staff

student

chapter * section C

name

D

Fig. 4. Four inlined DTD graphs

shared-inlining algorithm. For example, for the shared node author in Figure 4.B, the shared-inlining algorithm will generate a separate relation author(authorID, author.parentID, author.parentCODE, author.name.isroot, author.name, author.institute.isroot, author.institute). This schema implies a great deal of redundancy if an author writes hundreds of conference or journal papers, In contrast, we create a relation author(ID, nodetype, name, institute) for author, and translate its parent ∗-edges (and all other ∗-edges) into another separate relation edge(parentID, childID, parentType, childType). Our strategy eliminates the above redundancy and bears the same spirit as the rule of mapping many-to-many relationships into separate relations in translating Entity-Relationship (ER) diagrams into relational schemas. – Optimizations. Two situations are very common in XML documents: (1) there are XML elements which do not have any attributes and their single purpose is to provide a tag name (e.g., Figure 4.C) for supporting nested structure; and (2) there are terminal nodes that are shared by several XML elements (such as name and telephone in Figure 4.D). If we created a separate relation for each such kind of element, then we would produce a set of relations of the form of t(ID) (case 1) or t(ID, t) (case 2). Hence, instead, we create two relations table1(ID, nodetype) and table2(ID, nodetype, pcdata) which conceptually combine all these relations. These optimizations greatly reduce the number of relations in the generated schema and facilitates the translation of XML queries into relational queries.


375

]> Fig. 5. A publication DTD

– Efficiency: The shared-inlining algorithm introduces an attribute parentID for each node under the ∗ operator while the ∗ operator itself is never translated into a separate relation. This facilitates the traversal of XML documents upwards (from children to parents) but not downwards (from parents to children). For example, in Figure 4.D, the shared-inlining algorithm will generate relations dept, faculty, staff, etc. Given a faculty, it is very easy to locate which department he is from based on an index on facultyID and faculty.parentID of relation faculty. However, it would be difficult to navigate downwards for path expressions such as dept//name (get all the names reachable from element dept), since one needs to consider the fact that dept actually has three kinds of children (faculty, staff, and student), and all these three ways of reaching a name have to be combined. In contrast, We will translate all ∗-edges into one single relation edge(parentID, childID, parentType, childType), and create two indices on parentID and childID respectively. In this way, both upward navigation and downward navigation are supported efficiently.

3.4

A Complete Example

In this section, we illustrate different steps of our algorithm with a real DTD example, and demonstrate how XML documents conforming to this DTD can be stored based on the generated schema.

376

S. Lu et al.

An XML DTD for publications is shown in Figure 5. After the simplification step (using the rules defined in Figure 1), the input DTD is simplified into one with the following new XML element definitions. The definitions for other XML elements remain the same. – – – – – –

journal (name, editors, paper*)>. conference (name, paper*)>. paper (ptitle,authors,volume,number)>. editors (person*)>. authors (person*)>. references (paper*)>.

Due to space limit, we omit the DTD graph for the simplified DTD and the inlined DTD graph and leave them as an exercise for the reader. Finally, the following eight relations will be generated. – – – – –

publication(ID) stores XML element publication. conference(ID, name.ID) stores XML element conference. journal(ID, nodetype, name.ID) stores XML elements journal and editors. name(ID, PCDATA) stores XML element name. paper(ID, nodetype, ptitle, volume, number, year) stores XML elements ptitle, authors, volume, number and year. – person(ID, nodetype, pname, institute) stores XML elements person, pname and institute. – techreport(ID, nodetype, title) stores XML elements techreport, title and references. – edge(parentID, childID, parentType, childType) stores all the parent-child relationships between two XML elements.

4


We have developed a new inlining algorithm that maps a given input DTD to a relational schema. Our algorithm is inspired by the shared-inlining algorithm but features several improvement over it including overcoming its incompleteness, eliminating redundancies caused by shared elements, performing optimizations and enhancing efficiency. Future work includes a full evaluation of the performance of our approach versus other approaches and adapting our algorithm to one that maps XML Schemas [17] (an extension to DTDs) to relational schemas. Based on this schema mapping scheme, the mappings from XML data to relational data, and from XML queries to relational queries need to be investigated.

References 1. eXtensible Information Server (XIS). eXcelon Corporation. http://www.exln.com. 2. Tamino XML Server. Software AG. http://www.softwareag.com/tamino.


377

3. T. Bray, J. Paoli, C. Sperberg-McQueen, and E. Maler. Extensible Markup Language (XML) 1.0, October 2000. http://www.w3.org/TR/REC-xml. 4. M. J. Carey, D. Florescu, Z. Ives, Y. Lu, J. Shanmugasundaram, E. Shekita, and S. Subramanian. XPERANTO: Publishing object-relational data as XML. In WebDB (Informal Proceedings), pages 105–110, 2000. 5. S. Ceri, S. Comai, E. Damiani, P. Fraternali, S. Paraboschi, and L. Tanca. XMLGL: a graphical language for querying and restructuring WWW data. In International World Wide Web Conference (WWW), Toronto, Canada, May 1999. 6. D. Chamberlin, D. Florescu, J. Robie, J. Simeon, and M. Stefanascu. XQuery: A Query Language for XML, February 2001. http://www.w3.org/TR/xquery. 7. A. Deutsch, M. Fernandez, D. Florescu, A. Levy, and D. Suciu. XML-QL: A Query Language for XML, August 1998. http://www.w3.org/TR/NOTE-xml-ql/. 8. A. Deutsch, M. Fernandez, and D. Suciu. Storing semistructured data with STORED. In Proc. of ACM SIGMOD International Conference on Management of Data, pages 431–442, Philadephia, Pennsylvania, June 1999. 9. M. Fernndez, W. Tan, and D. Suciu. SilkRoute: Trading between relations and XML. In Proc. of the Ninth International World Wide Web Conference, 2000. 10. D. Florescu and D. Kossman. Storing and querying XML data using an RDBMS. IEEE Data Engineering Bulletin, 22(3), 1999. 11. D. Florescu and D. Kossmann. A performance evaluation of alternative mapping schemes for storing xml data in a relational database. In Proc. of the VLDB, 1999. 12. R. Goldman, J. McHugh, and J. Widom. From Semistructured Data to XML: Migrating the Lore Data Model and Query Languages, 1999. 13. A. Kurt and M. Atay. An experimental study on query processing efficiency of native-XML and XML-enabled relational database systems. In Proc. of the 2nd International Workshop on Databases in Networked Information Systems (DNIS’2003), Lecture Notes in Computer Science, Volume 2544, pages 268–284, Aizu-Wakamatsu, Japan, December 2002. 14. J. Robie, J. Lapp, and D. Schach. XML Query Language (XQL), 1998. http://www.w3.org/TandS/QL/QL98/pp/xql.html. 15. J. Shanmugasundaram, E. Shekita, R. Barr, M. Carey, B. Lindsay, H. Pirahesh, and B. Reinwald. Efficiently publishing relational data as XML documents. VLDB Journal, 10(2–3):133–154, 2001. 16. J. Shanmugasundaram, K. Tufte, C. Zhang, G. He, D. DeWitt, and J. Naughton. Relational databases for querying XML documents: Limitations and opportunities. The VLDB Journal, pages 302–314, 1999. 17. C. Sperberg-MCQueen and H. Thompson. W3C XML Schema, April 2000. http://www.w3.org/XML/Schema. 18. I. Tatarinov, Z. Ives, A. Halevy, and D. Weld. Updating XML. In Proc. of ACM SIGMOD International Conference on Management of Data, Santa Barbara, CA, 2001. 19. F. Tian, D. DeWitt, J. Chen, and C. Zhang. The design and performance evaluation of alternative XML storage strategies. ACM Sigmod Record, 31(1), March 2002.

From XML DTDs to Entity-Relationship Schemas Giuseppe Psaila Universit` a degli Studi di Bergamo Facolt` a di Ingegneria Viale Marconi 5 - I-24044 Dalmine (BG), Italy [email protected]

Abstract. The need for managing large repositories of data coming from XML documents is increasing; in fact, XML is emerging as the standard format for documents exchanged over the internet. At University of Bergamo, recently we developed the ERX Data Management System, to study issues concerning the management of data coming from XML documents; its data model, called ERX (Entity Relationship for XML), being an extension of the classical ER model, allows to deal with concepts coming from XML documents at the conceptual level, and allows to reason about integration of data coming from different XML document classes. This paper focuses on the problem of automatically deriving EntityRelationship Schemas (ERX Schemas) from DTDs (Document Type Definition). In fact, the derivation of such schemas from DTDs might be a hard work to do by hand, since real DTDs are very complex and large.

1

Introduction

The need for managing large repositories of data coming from XML documents is increasing; in fact, XML is emerging as the standard format for documents exchanged over the internet. For this reason, several systems to store XML documents were developed (see Lore [6], Tamino [1], etc.). In particular, database researchers studied the problem of building such systems on top of relational databases (see [10]). At University of Bergamo, recently we developed the ERX Data Management System [9,7,8], to study issues concerning the management of data coming from XML documents, as well as the integration of different technologies, such as databases, Java and XML technology (XSLT) in the same framework. The data model provided by the system is an extension of the classical Entity-Relationship model [2], named ERX (Entity-Relationship for XML); this data model allows to deal with concepts coming from XML documents at the conceptual level. During the design of the system, we decided for an ER data model due to its independence of the particular database technology used to develop the system. The system stores data obtained from processed XML documents, and provides a query language that rebuilds XML documents [7]. ´ Pastor (Eds.): ER 2003 Workshops, LNCS 2814, pp. 378–389, 2003. M.A. Jeusfeld and O. c Springer-Verlag Berlin Heidelberg 2003

From XML DTDs to Entity-Relationship Schemas

379

Using the system, we found that this data model can be effective independently of the system. In fact, by means of ER Schemas it is possible to have an in-depth understanding of concepts described by XML documents and their correlations, as well as to reason about integration of data coming from XML documents belonging to different classes (i.e. valid for different DTDs, using the XML terminology). Thus, the ERX data model can be used independently of the actual exploitation of the ERX System. However, the derivation of an ER schema from a real DTD (Document Type Definition) might become a hard work to do by hand, since real DTDs are very complex and large. This paper focuses on the problem of automatically deriving Entity-Relationship Schemas (ERX Schemas) from DTDs. This work is the first attempt toward a technical solution to this problem: in fact, we consider (and propose a solution to deal with) basic concepts provided by DTD and ER model (e.g., XML entities are not considered, as well as hierarchies of entities in the ER model are not considered). We proceed as follows. First, we identity a set of DTD rewriting rules; by applying them, it is possible to obtain a new version of the original DTD, which is more suitable for mapping DTD concepts into ER concepts (Section 4). Then a derivation technique, and the corresponding algorithm, is defined, to derive the ER schema from the rewritten DTD (Section 5). We assume the reader familiar with XML basic concepts [3].

2

Preliminaries

Case Study. Suppose we are building the ER model for lists of products; a sample XML document might be the one reported in Figure 1. In the document, tag Product describes a single product, characterized by attributes ID (the product identifier), Description (product description), Brand (product brand). As content of tag Product, we find one Technical tag (a technical description about the product), and one Note tag (generic notes about the product). Both tags Technical and Note contains a Text tag, whose content is a mixed composition of generic text and occurrence of HyperLink tags. Document Type Definition (DTD). Suppose now that the document in Figure 1 is valid for the DTD (Document Type Definition) reported in the figure. Recall that basic tags in a DTD are the !ELEMENT tag and the ATTLIST tag: the former defines tags (also called elements) and the structure of their content; the latter defines attributes for tags. In particular, the syntax of !ELEMENT is , where T agN ame is the name of the tag under definition. Structure is a regular expression, based on the iteration operators * (zero to many repetitions) and + (one to may repetitions), the optionality operator ? (zero or once), the sequence operator (a comma) and the alternative operator (or choice operator) |. A special case of content is the mixed content, specified as (#PCDATA | T agN ame1 | . . . | T agN amen )* where #PCDATA denotes generic text. In practice, the content is a mixed combination of generic text and occurrences of the listed tag names T agN amei . Finally, the keyword EMPTY (tag without content) can be used for Structure.

380

G. Psaila

Fig. 1. XML document (on the left) and DTD (on the right) for products.

The syntax of !ATTLIST is where T agN ame is the name of the tag for which attributes are defined. In the simplest version (considered in this paper), each AttrDefi is a triple AttrN ame CDATA RI where AttrN ame is the name of the attribute under definition; CDATA denotes that the attribute value is a string; RI can be either #REQUIRED or #IMPLIED, (mandatory or optional attribute, resp.). Consider now the DTD for our example, reported in Figure 1. We can see that the structure of the document is as follows. The root element ProductList must contain a non empty list of Product tags, and has a mandatory attribute Date. Tag Product can contain an occurrence of the Technical tag, followed by a possibly empty sequence of Note tags; furthermore, Product tags have three mandatory attributes (ID, Description and Brand) and a possibly missing attribute (Price). Tags Technical and Note have the same structure, i.e. a mandatory and unique occurrence of tag Text. The content of this latter tag is a typical case of mixed content: generic text and occurrences of tag Hyperlink. Finally, tag HyperLink is an empty tag, with a mandatory attribute URL and a possibly missing attribute Text (the text associated to the URL).

3

The ERX Data Model

We now introduce the basic concepts of the ERX Data Model [9,7,8]. Entities. An entity describes a complex (structured) concept of the source XML documents. Entities are represented as solid line rectangles; the entity name is inside the rectangle. An instance of an entity X is a particular occurrence of the concept described by entity X in a source document. It is identified by a unique, system generated, numerical property named OID. The ERX Data Model provides the concept of Hierarchy as well (see [9]); for the sake of space, we do not consider this concept here.

From XML DTDs to Entity-Relationship Schemas ID (R)

381

Description (R)

Technical in Product

Product in ProductList

Technical

Product (1:1)

(1:1)

(0:1)

(1:1)

Note in Product

Note

Brand (R) Price (I) (1:N)

(0:N)

(1:1) (1:1)

Content (R)

ProductList

Contain Text (1:1)

Text

Date (R)

Par (0:N)

Text Contains

HyperLink

URL (R)

Text (I)

Fig. 2. ERX Schema for the case study

Relationships. A relationship describes correlations existing between entities X and Y . A relationship is represented as a diamond labeled with the name of the relationship. The diamond is connected to X and Y by solid lines; these lines are labeled with a cardinality constraint (l:u), which specifies for each instance of entity X (resp. Y ) the minimum number l and the maximum number u of associated instances of Y (resp. X). An instance of the relationship describes a particular association between two instances of the connected entities. A complex form of relationship is represented by a relationship with alternatives: an instance of an entity X is associated with instances of alternative entities Y1 , Y2 , ..., Yn ; the cardinality constraint for X considers all associations of an instance of X with instances of any entity among Y1 , Y2 , ..., Yn . Orthogonally, a relationship can be a containment relationship. Given two entities X and Y , a containment relationship from X to Y denotes that an instance of X structurally contains instances of Y . Containment relationships are represented as normal relationships, except for the fact that lines from X to Y are arrows, oriented from X to the relationship diamond, and from the diamond to Y . The cardinality constraint on the contained side is always (1:1) (thus, it is omitted). Instances of containment relationships have an implicit property, named order: this property denotes the position occupied by each contained entity instance (this concept is good for ordered lists, such as XML miced content). Attributes. Entities can have attributes: they represent elementary concepts associated to an entity. Attributes are represented as small circles, labeled with the name, and connected to the entity they belong to by a solid line. Entity attributes are always string valued. Furthermore, ERX does not provide the concept of key attribute. Attribute names are associated to a qualifier, which indicates specific properties of the attribute. Qualifiers (R) and (I) denotes that the attribute is required or implied (i.e. optional), respectively.

382

G. Psaila Table 1. Simple rewriting rules 1 2 3 4 5 6 7

((item)) (item?) (item*) (item+) (item+)* (item*)* (item?)*

≡ ≡ ≡ ≡ ≡ ≡ ≡

(item) (item)? (item)* (item)+ (item)* (item)* (item)*

8 9 10 11 12 13 14

(item+)+ (item*)+ (item?)+ (item+)? (item*)? (item?)? (item)op

≡ ≡ ≡ ≡ ≡ ≡ ≡

(item)+ (item)* (item)* (item)* (item)* (item)? item op

Consider now the ERX Schema in Figure 2. Observe that there is an entity for each tag in the DTD discussed in Section 2, and for each attribute defined in the DTD, there is the corresponding attribute in the schema. Furthermore, notice attribute Content on entity Par: this report the generic text specified in a mixed content (see definition for tag Text). Considering relationships, at first notice relationship Contain Text: this is a relationship with alternatives, and means that an instance of entity Text can be associated either to an instance of entity Technical or to an instance of entity Note (this corresponds to the fact that tag Text appears in the content of two distinct tags). Finally, relationship Text Contains is a containment relationship with alternative: this means that not only instances of entities Par and HyperLink are alternatively associated to instances of entity Text, but also it is necessary to keep these associations ordered, since this relationship derives from a mixed content in the DTD.

4

DTD Rewriting

We are now ready to introduce our technique to derive ER Schemas from DTDs. In this section, we introduce DTD rewriting rules. The goal of this step is the following. Given a DTD, this might be in a form not suitable for deriving ER schemas, e.g. because it is not sufficiently simplified (extra parenthesis), or because for some DTD constructs there is not a suitable ER structure. Thus the main goal of DTD rewriting rules is to obtain an equivalent (when possible) or a slightly more general version (not equivalent to the original one) which is suitable for deriving ER schemas. These rules will be illustrated in Section 4.1. However, it might not be a good idea to derive ER schemas which are too close to the original DTD, in particular when the final goal of the process is the integration of data coming from documents belonging to different classes (thus, specified by different DTDs). In practice, we want to avoid the problem known as over-fitting. To solve this problem, Section 4.2 introduces rules that perform a deeper rewriting: the rewritten DTD is significantly more general than the original one, but is also more distant from it w.r.t. the one obtained only by means of basic rewriting rules. We will show in Section 5 that ER schemas derived after deeper rewriting are simpler and more compact.


383

Table 2. Rewriting rules for choice and sequence (15 to 19) and for deeper rewriting 15 (item1 | . . . | itemi | . . . | itemn ) ≡ (item1 | . . . | itemi,1 | . . . | itemi,h | . . . | itemn ) where itemi = (itemi,1 | . . . | itemi,h ) 16 (item1 | . . . | itemi ? | . . . | itemn ) ≡ (item1 | . . . | itemi,1 ? | . . . | itemi,h ? | . . . | itemn ) where itemi = (itemi,1 | . . . | itemi,h ) 17 (item1 op1 | . . . | itemi opi | . . . | itemn opn) ≡ (item1 | . . . | itemi | . . . | itemn )opex such that: opex =? if opi = N U LL or opi =?, with at least one opi =? 18 (item1 op1 | . . . | itemi opi | . . . | itemn opn) ⇒ (item1 | . . . | itemi | . . . | itemn )opex such that: opex =* if there exists one opi =*, or one opi =+ and one opj =? opex =+ if opi = N U LL or opi =+, with at least one opi =+ opex =? if opi = N U LL or opi =?, with at least one opi =? 19 (item1 op1 , . . . , itemi opi , . . . , itemn opn)? ≡ (item1 op1 , . . . , itemi opi , . . . , itemn opn) where opi =? or opi =* 20 (item1 op1 , . . . , itemi opi , . . . , itemn opn) → (item1 | . . . | itemi | . . . | itemn )opex such that: opex =+ if every opi =+ or opi = N U LL opex =* if there exists one opi =? or opi =* 21 (item1 op1 , . . . , itemi opi , . . . , itemn opn)ope → (item1 | . . . | itemi | . . . | itemn )opex such that: opex =+ if ope =+ and every opi =+ or opi = N U LL opex =* if ope =?, or ope =*, or there exists one opi =? or opi =*

4.1

Basic Rewriting

Let us start with basic rewriting rules. We can distinguish them into three categories: simple rewriting rules, choice rewriting rules and sequence rewriting rules. Simple rewriting rules simplify the DTD, always obtaining an equivalent version. In particular, they reduce the number of parentheses and put the regular expression operator (* or + or ?) outside the external parenthesis. Table 1 shows these rules (in rule 14, and also in rules from 15 to 21, with o p we denote any operator among *, +, ?). Observe that they must be applied from left to right, so that they actually simplify the DTD. Choice rewriting rules operate on choice expressions appearing in DTDs (see Table 2, rules from 15 to 18). They replace a choice expression with another choice expression. In particular, observe that rules 15, 16, 17 obtain equivalent, but simplified, expressions in which regular expression operators (* or + or ?) appears only outside the external parenthesis.

384

G. Psaila

This result is obtained also by rule 18, but the right hand side expression is not equivalent to the left hand side expression; indeed, the right hand side expression is more general (notice that the symbol ⇒ is used in the middle of the two expressions). Although an equivalent DTD is not obtained, this is not a problem: in fact it is true that the rewritten DTD is slightly more general, but this limited loss of precision allows to obtain a simpler ER schema (at the end, a choice expression is transformed into another choice expression). Finally, rule 19 in Table 2 is the only rule for sequence expressions considered here. This rule rewrites the left hand side expression, obtaining an equivalent expression; it pushes the external regular expression operator into the parenthesis; the right hand side expression is then simplified w.r.t. the left hand side. Deeper, but not equivalent, rules for sequences are introduced in Section 4.2. Examples Consider the following element specification. It can be rewritten by applying the discussed rewriting rules. Below, the sequence of rewriting is reported, where the subscript parenthesis on the left hand side of symbols ≡ and ⇒ denotes the number of the applied rules. Rows 6. and 7. are obtained by applying rule 18, which does not produce an equivalent expression. Furthermore, the final expression is certainly simplified, but not so far away from the original one. This means that the loss of precision is minimal. 1. 2. 3. 4. 5. 6. 7. 4.2

(((E1? | E2*)?, (E3)+) | ((E4, E5)?)) ≡ (((E1? | E2*)?, E3+) | ((E4, E5)?)) (14) ≡ (((E1? | E2*)?, E3+) | (E4, E5)?) (16) ≡ ((((E1?)? | (E2*)?), E3+) | (E4, E5)?) (13,12) ≡ (((E1? | E2*), E3+) | (E4, E5)?) (18) ⇒ (((E1 | E2)*, E3+) | (E4, E5)?) (18) ⇒ (((E1 | E2)*, E3+) | (E4, E5))? (14)

Deeper Rewriting

Apart for rule 18, the above discussed rewriting rules do not change the structure of DTDs. This way, the ER schema that derives from the rewritten DTD well represents the source DTD. However, this may cause over-fitting: the ERX schema is too close to a specific DTD, and it is not general enough for integration of documents valid for several DTDs. To overcome this problem, in Table 2 we introduce two new rewriting rules, numbered as 20 and 21. Both these rules substitute sequence expressions with choice expressions; consequently, the resulting DTD is significantly more general. However, this loss of precision allows further simplification of the DTD, and allows to derive a simpler and more general ER Schema, more suitable for integration purposes. Observe that in rules 20 and 21 we used the symbol →; this way, we denote that this rewriting significantly changes the DTD. Examples. Consider the expression number 7., which is the result of the rewriting process discussed in the previous example. The rewriting process can be car-


385

ried on by applying the deeper rewriting rules, obtaining a very simplified and general expression. 8. 9. 10. 11. 12. 13.

5

→ (((E1 | E2) | E3)* | (E4, E5))? (15) ≡ ((E1 | E2 | E3)* | (E4, E5))? (20) → ((E1 | E2 | E3)* | (E4 | E5)+)? (18) ⇒ (((E1 | E2 | E3) | (E4 | E5))*)? (12) ≡ ((E1 | E2 | E3) | (E4 | E5))* (15,1) ≡ (E1 | E2 | E3 | E4 | E5)* (20)

From DTD Concepts to ERX Concepts

We are now ready to discuss how the ER schema is derived from the rewritten DTD. The main derivation rules are the following. 1. Each XML element defined in the DTD corresponds to an entity; attributes defined for each XML element are assigned to the corresponding entity, and are mandatory or optional depending on the fact that they are defined as #REQUIRED or #IMPLIED in the DTD. 2. Generic text elements (#PCDATA) correspond to entities with one single mandatory attribute named Content, whose value represents the contained text. 3. Choice expressions are translated as containment relationships. If the choice expression is a top level expression, the left hand side entity is the entity derived from the XML element whose content is defined by the choice expression. The right hand side is an alternative of all entities derived from XML elements listed in the choice expression; complex items appearing in the choice expression are dealt with by a dummy entity1 ; the process is then recursively repeated. The left hand side cardinality of the generated relationship depends on the iteration operator applied to the choice expression ((0: 1) for ?, (0: N) for *, (1: N) for +, and (1: 1) if missing). 4. Sequence expressions to which no iteration operator is applied, are translated as a series of relationships, one for each item appearing in the sequence. If the sequence expression is a top level expression, the left hand side of these relationships is the entity derived from the XML element whose content is defined by the choice expression. The right hand side is the entity derived from one element in the sequence expression; complex items appearing in the choice expression are dealt with by a dummy entity; the process is then recursively repeated. The left hand side cardinality and the right hand side cardinality are (1: 1). 5. Sequence expressions to which an iteration operator is applied are translated as a relationship, whose left hand side is the same as in point 4, while the right hand side is a dummy entity. The left hand side cardinality of the generated relationship depends on the iteration operator applied to the choice expression ((0: 1) for ?, (0: N) for *, (1: N) for +); the right hand side cardinality is (1: 1

A dummy entity is an entity that does not derive from any XML element defined in the DTD. Its name is generated by the system.

386

G. Psaila ID (R)

Description (R)

Product in ProductList Technical

Product (1:1)

(0:1)

(1:1)

(1:1) Brand (R)

Note

In Product

Price (I) (1:N)

(1:1) (1:1) Content (R)

ProductList

Contain Text (1:1)

Text

Date (R)

Par (0:N)

Text Contains

HyperLink

URL (R)

Text (I)

Fig. 3. Generalized ERX Schema for the case study

1). Then the process is recursively repeated considering the dummy entity and the sequence expression without iteration operator. 6. Finally, after all DTD specifications have been processed, it is necessary to collapse relationships. In particular, for each entity that appears in multiple right hand sides with cardinality (1: 1), these relationships are collapsed into one single relationship, with the same right hand side; the left hand side is the set of all left hand sides of relationships which are collapsed together. For example, the sample DTD described in Section 2 is already minimal w.r.t. basic rewriting rules. Hence, the ERX schema reported in Figure 2 is derived by means of our technique. Notice relationships named Technical in Product and Note in Product: they are derived from the DTD line ; notice cardinalities (0: 1) and (0: N), which derive from operators ? and *, resp.. Also note that entity Text and relationship Contain Text are obtained by applying item 6. in the derivation rules; finally, relationship Text Contains is derived from a mixed content specification (#PCDATA | HyperLynk)*. If we consider the application of deeper rewriting rules, in our sample DTD only the specification for element Product changes; it becomes: . Observe that the sequence is changed into an iterated choice expression. Figure 3 shows the resulting ERX schema: notice that this is more general, in that relationships Technical in Product and Note in Product have been substituted by a single relationship named In Product. 5.1

The Algorithm

We briefly describe the core of the derivation algorithm, which is Procedure DeriveRFel shown in Figure 4. This procedure is called for each element defini-


387

tion in the DTD, and corresponds to items 2 to 5 of the derivation rules. Here, we report some useful notation. With structure we denote a structure definition in the DTD, such as (E1, (E2 | E3))*. This is a sequence structure, which contains a simple element (E1 and a complex item (E2 | E3), which is in turn a choice structure; a structure is denoted as structure. With structure[i] we denote the i-th element in the structure (structure[1]=E1, structure[1]=(E2 | E3)). With structure.card we intend the iteration (also called cardinality) operator applied to the structure (in this case, strcture.card=*). With structure.length we denote the number of elements in the structure (structure.length = 2 in the sample). The procedure makes use of some procedures and functions. Function CreateEntityPar creates an entity for textual paragraphs (see entity PAR in Figure 2). Function CreateDummyEntity creates new entities with no attributes, whose name is system generated. Procedure CreateRelationship creates a new relationship, while procedure CreateContainmentRelationship creates containment relationships; both of them have three parameters, which specify, resp., the name of left hand side entity, the set of alternative entities in the right hand side, the iteration operator to use to establish cardinality constraints on the left hand side ((0: 1) for ?, (0: N) for *, (1: N) for +). Procedure DeriveRel is recursive, since it has to deal with nested regular expressions, as for . Figure 5.a shows the derived ERX schema (for simplicity, we do not worry about attributes); notice the presence of two relationships Rel1 and Rel2, which directly derive from each one of the two elements in the sequence structure (thus, the left hand side cardinality constraints correspond to the iteration operators, i.e.(0: N) for * and (1: N) for +. To represent the nested sequence structure, the algorithm creates a dummy entity D1, which represents the overall nested sequence structure; then, the algorithm recursively derives two relationships Rel3 and Rel4, whose left hand side is the dummy entity D1. If we apply deeper rewriting rules, we obtain the simplified element definition . from which the algorithm derives the ERX schema in Figure 5.b. Observe that this is a very simple schema.

6

Conclusions

In this paper, we considered the problem of deriving Entity Relationship schemas from XML DTDs; we adopt the ERX Data Model, a variation of the classical ER model specifically designed to cope with XML. The problem has been dealt with as follows. At first, a set of rewriting rules for DTDs has been defined; the goal of these rules is to simplify and generalize DTDs. Then a derivation technique to derive the Entity Relationship schema from the rewritten DTD has been developed. We compared the resulting schemas for our case study, obtained by applying only basic rewriting rules and deeper rewriting rules. Schemas obtained by means of deeper rewriting rules are more general, and more suitable for integrating documents valid for different DTDs.

388

G. Psaila Procedure DeriveRel(Entity, structure) begin if structure is a choice structure then list = { }; for i = 1 to structure.length do if structure[i] is a simple element then list = list ∪ structure[i].name; if structure[i] is a #PCDATA element then list = list ∪ CreateEntityPar(); if structure[i] is a sequence or a choice structure then DummyEnt = CreateDummyEntity(); DeriveRel(DummyEnt, structure[i]); list = list ∪ DummyEnt; end if end for CreateContainmentRelationship(DummyEnt, list, structure.card); end if if structure is a sequence structure then if structure.card = NULL then DummyEnt = CreateDummyEntity(); N ewStruct = structure; N ewStruct.card = NULL; DeriveRel(DummyEnt, N ewStruct); CreateRelationship(Entity, { DummyEnt }, structure.card); else for i = 1 to structure.length do if structure[i] is choice structure then DeriveRel(Entity, structure[i]); continue; if structure[i] is a simple element then Ent = structure[i].name; if structure[i] is a sequence structure then Ent = Entity; N ewStruct = structure[i]; N ewStruct.card = NULL, DummyEnt = CreateDummyEntity(); Ent = DummyEnt; DeriveRel(Ent, N ewStruct); end if CreateRelationship(Entity, { Ent }, structure[i].card); end for end if end if end Fig. 4. Procedure DeriveRel

In effect, the ERX Data Model, from which this work moved on, is very suitable to study concepts present in XML documents and their correlations. Although design guidelines convinced us to use this data model as the one provided by the ERX Data Management System, its use is not restricted within it;


389

Rel3 E1

Rel1

(1:1)

E

(1:1)

Rel1

D1 (0:N)

E1

E Rel4

(1:1)

E2

(1:1)

Rel2

(0:N)

E2

(1:1)

E3 (1:N)

E3

(1:1)

a)

b)

Fig. 5. Sample ER schemas

indeed, it is very useful to understand the content of XML documents and to perform integration tasks. Future Work. This is the first work on this topic. We plan to continue the research on two main directions. The first one is to consider the concept of hierarchy, a very useful concept provided by ERX and implicitly provided by DTDs, e.g. by means of the concept of XML entity. The second one is to move to XML Schema, the standard that will replace DTDs in the near future.

References 1. Tamino XML Database. Software AG, http://www.softwareag.com/tamino. 2. C. Batini, S. Ceri, and S. Navathe. Conceptual Database Design: An EntityRelationship Aprroach. Benjamin Cummings, Menlo Park, California, 1992. 3. T. Bray, J. Paoli, and C. M. Sperberg-McQueen. Extensible markup language (xml). Technical Report PR-xml-971208, World Wide Web Consortium, Dec. 1997. 4. M. Kay. XSLT Programmer’s Reference. Wrox Press, 2000. 5. M. Liu and T. W. Liung. A data model for semistructured data with partial and inconsistent information. In Intl. Conf. on Extending Database Technology, Konstanz, Germany, March 2000. 6. J. McHugh and J. Widom. Query optimization for xml. In Proc. 25th VLDB Conference, Edinburgh, Scotland, September 1999. 7. G. Psaila. Erx-ql: Querying an entity-relationship db to obtain xml documents. In Proceedings of DBPL-01 Intl. Workshop on Database Programming Languages, Monteporzio Catone, Rome, Italy, September 2001. 8. G. Psaila. Erx: an experience in integrating entity-relationship models, relational databases and xml technologies. In Proceedings of XMLDM-02 Intl. Workshop on XML Data Management, Prague, Czech Republic, March 2002. 9. G. Psaila and D. Brugali. The erx data management system. In Proc. of IC-2001, Second Int. Conference on Internet Computing, Las Vegas, USA, June 2001. 10. J. Shammugasundaram, K. Tufte, G. He, C. Zhang, D. DeWitt, and J. Naughton. Relational databases for querying xml documents: Limitations and opportunities. In Proc. 25th VLDB Conference, Edinburgh, Scotland, September 1999.

Extracting Relations from XML Documents Eugene Agichtein1 , C.T. Howard Ho2 , Vanja Josifovski2 , and Joerg Gerhardt2 1

Columbia University, New York, NY, USA [email protected] 2 IBM Almaden, San Jose, CA, USA {ho,vanja}@almaden.ibm.com

Abstract. XML is becoming a prevalent format for data exchange. Many XML documents have complex schemas that are not always known, and can vary widely between information sources and applications. In contrast, database applications rely mainly on the flat relational model. We propose a novel, partially supervised approach for extracting userdefined relations from XML documents with unknown schema. The extracted relations can be directly used by an RDBMS, or utilized for information integration or data mining tasks. Our method attempts to automatically capture the lexical and structural features that indicate the relevant portions of the input document, based on a few user-annotated examples. This information can then be used to extract the relation of interest from documents with schemas potentially different from the training examples. We present preliminary experiments showing that our method could be capable of extracting the target relation from XML documents even in the presence of significant variations in the document schemas.

1

Introduction

XML provides a standardized format for data exchange where relationships between entities are encoded by nesting of the elements. XML documents can have complex nested structure, while many applications prefer a simple and flat representation of the relevant information. Extracting information from XML documents into relations is of special interest, since the resulting relations would allow the use of SQL and the full power of RDBMS query processors. In such a scenario, a mapping is needed to specify the extraction of the required portions of XML documents to relations. Mapping specification is usually performed by an experienced user with knowledge of the content of the input document and resulting relations. If detailed description of the document structure is available in advance, a mapping can be defined once and used over all of the input documents. In the case when the XML documents originate from a number of different sources with variations in their schema, or when the schema evolves over time,

Work done while visiting IBM Almaden.


Extracting Relations from XML Documents

391

the mapping specification process can be long and labor-intensive. The user needs to provide a mapping for each new source, and update the queries as the document structures change. In order to relieve the user of this tedious task, we propose a system for mapping from XML to relations by generalizing from the user-provided examples, and applying the acquired knowledge on unseen documents with flexibility to handle variations in the document input structure and terminology (tag names). Such documents may be derived from HTML pages, or from business objects exported to XML. For example, consider the task of compiling a table of product prices and descriptions from different vendors, where each vendor exports their product catalogs as XML documents. These documents may encode the prices of products in a variety of ways, using different tag names and structures. With current technology, each vendor source would have to be wrapped manually to extract the tuples for the target table. Being able to extract key relations from such XML documents using a few user-specified examples would reduce the system setup time and allow for improved robustness in a case of schema change. Our partially supervised approach is an adaptation of the general nearest neighbor classification strategy [1]. In this approach, the candidate objects are compared with a set of “prototype” objects. The candidates that are the closest to a prototype p are classified into the same class as p. In our setting, the goal of the classifier is to recognize the nodes (if any), in a given XML document that are needed to extract (map) the information in the document to the given target relation. The prototype objects are constructed based on the user-annotated example XML documents, and correspond to the nodes that contain the attributes to be mapped to the target relation. The similarity between the candidate nodes and the prototype nodes is computed using signatures that represent the position, internal structure, and the data values of the document nodes. Preliminary experiments indicate that our method can be used to reliably detect relevant elements in unseen documents with similar, but different structure. The use of signatures as opposed to queries allows more flexibility in the structure captured from the training examples. For example, the terms in the signature can be related in a way that does not match the XQuery axes, or weights can be assigned to individual terms, which would allow specifying increased importance to some of the terms. Such features are not available in today’s XML query and transformation languages. Related Work Several commercial and research databases support mapping XML documents into user-defined relations. These mappings are specified by using XPath expressions and apply only to documents with schemas compatible with the expressions. If the schema is not available, a system such as XTRACT [2] can be used to infer a DTD. The documents in the collection are assumed to have the same structure, and elements can be described independently. Some systems allow building a summary of several XML documents, as for example the

392

E. Agichtein et al.

DataGuides [3] techniques that emerged from the LORE project. A related approach taken by STORED [4] uses data mining techniques to store the information in an RDBMS for efficient querying. In contrast, we assume a given user-defined relation to which we want to map XML documents with variable schema. Several interactive tools have emerged that allow mapping from XML documents to relational schemas, as for example [5]. The mappings produced by these tools are used for both shredding and storing XML documents and for view generation for online querying [6]. The techniques we use in our work draw on methods developed for the extraction of structured information from HTML and plain text, notably [7,8,9,10,11,12,13]. The rest of the paper proceeds as follows: In Section 2 we present an overview of our system and describe our data model. In Section 3 we describe our method for generating and using signatures for extracting a relation from new XML documents (Section 4). We then present preliminary experimental results in Section 5 and conclude the paper in Section 6 with a description of our current activities and future work.

2

System Overview and Data Model

We use a partially supervised approach to extract a user-specified relation from XML documents based on a few user-tagged examples. The system works in two phases, Training and Extraction, shown in Figure 1. In the Training phase, the system is trained on a set of XML documents where the user has indicated the correct mapping of the XML elements to the attributes of the target table. The result of the training stage is a set of signatures that are used in the subsequent Extraction stage. During the Extraction phase, the target table is extracted from new XML documents that may have different tag names or structure than the example documents. As the first step of the extraction stage, the nodes of the input documents are merged in order to generate a “canonical” representation of the input document. Then, the signatures generated during training are used to find the candidate nodes in the canonical representation of each input document that are most likely to contain attributes for the target relation tuples. Finally, the mapping from the descendants of this node to the target attributes is derived. The resulting mapping can be translated trivially into XPath expressions to extract the tuples from the input document, or from any document with the same structure. 2.1

Data Model

Our system extracts a single target relation, T (a1 , a2 , ..., an ), from a collection of XML documents. Representing the input XML document as a tree of nodes, each tuple t ∈ T is extracted from a document subtree rooted at a node called instance node. More formally, we define an instance node I as a document element such that:


393

Merge AS

IS /Split a1

S

a2

a1

User-tagged examples IS

AS1 AS2

a2

Merged Tree Merge

Match

Map

/Split

Nodes

Attributes

S

AS AS1 AS2

Input document

Merged Representation

Best Instance Node

a1

a2

Best Mapping

Instance and attribute signatures

Training

Extraction

Fig. 1. Overview of our system: In the Training stage the system derives instance and attribute signatures used for extracting the target relation from a new XML document.

1. Children of I contain complete information needed to extract exactly one tuple of the target table. 2. I is maximal, i.e., any ancestor node of I will contain complete information for more than one tuple in T . Figure 2 illustrates the role of the instance node in a document representing a set of books, such as one that may be exported by a book vendor. The target relation is defined as NewBooks(ISBN, BookTitle, Author, Price). The node Item in the Books category, shown, contains all the information in the attributes of its descendants that is needed to extract a tuple for the target relation. Therefore all the Item elements shown in this example are instance nodes. The extraction of the relation from a new document d consists of first identifying a node in d that corresponds to I, and then mapping descendants of I to the attributes of the target relation. We now present our approach for automatically generating flexible signatures that can be used to recognize instance nodes in new XML documents with variations in label names and structure.

3

System Training: Generating Instance and Attribute Signatures

Our approach of deriving instance and attribute signatures to extract a target relation uses as input as set of user-supplied example XML documents. First, we pre-process the input documents to derive a merged document representation (Section 3.1). As we will discuss, the merged representation will allow us to describe the example documents more completely. We then generate instance signatures (Section 3.2) that capture the position and internal structure of the instance nodes. Then we describe attribute signatures (Section 3.3) that cap-

394

E. Agichtein et al.

Ancestors

Products Books Instance Item

Item

Title

Item

Category_Desc

Author

Siblings

Publisher

Descendants New

ISBN

ISBN

BookTitle

Price

Used

Num_Copies

Author

Price

Fig. 2. Example of Extracting a Table from XML documents.

ture the structural and data characteristics of the nodes that correspond to the attributes of the tuples in the target relation. The training begins with a set of annotated example XML documents, with special tags specifying the instance nodes and the attribute mappings.1 For each example we solicit two types of input from the user: 1. Identify the instance node I (e.g., the Item node) 2. Identify the descendants of I that contain the values for the attributes in the target relation. In all machine learning approaches, one of the major problems is data sparsity, where the manually annotated examples do not fully represent the underlying distribution. Some of the nodes in the initial examples may be optional (and therefore missing), and the data values may not be repeated enough across the remaining attributes to generate a reliable signature. Therefore, we propose merging the nodes in the input documents to create the “canonical” representation of the document tree as we describe next. 3.1

Merging Nodes in the Input Document

A relational table usually represents a set of similar entities, such that each entity corresponds to one tuple. We can therefore expect that an XML document mapped to a table will also contain a set of nodes for a set of related entities. 1

In our prototype we use reserved XML element names to specify the mapping. This allows for use of XML parsing and processing over the training documents. The annotated documents can be produced from the original documents by a user using a GUI tool.

Extracting Relations from XML Documents Merge

Products (Root) Item

Book Author Title

Item

Book Author Year

Item

Item

CD Artist Length

CD Name Artist

Products (Root) Item1*

Item2*

Item3 Year

Products (Root) Item*

Book Author Title Year CD Artist Length Name

Split Node\Tag Item1 Item2

Book Author Title

395

CD

Artist

Length Name

Item4

Book

Author

Title

Year

CD

Artist

Length

Name

1 1 0 0

1 1 0 0

1 0 0 0

0 1 0 0

0 0 1 1

0 0 1 1

0 0 1 0

0 0 0 1

Fig. 3. Operation of the Merge algorithm for merging similar nodes in the input document.

Intuitively, the nodes representing the same class of entities will have similar structure and relative position in the document. Often such XML documents will be produced by a single source and will have some regularity in their structure. We can exploit this regularity within a single document by merging “similar” nodes. As a result, we will have more rich signatures and reduce the complexity of the subsequent extraction phase. More importantly, merging nodes in the input document will allow us to reduce noise (e.g., missing optional nodes), resulting in more complete signatures (Sections 3.2 and 3.3) that will later be used for extraction. The Merge Algorithm: Our procedure for merging nodes is shown graphically in Figure 3. Intuitively, sibling nodes with the same tag name, and with similar internal structure can be assumed to represent similar objects. Using this observation, we merge sibling nodes that share the same prefix path from the root, and have similar internal structure. The user-annotated instance nodes are merged just as any other nodes, resulting in the more complete examples of the instance nodes for signature generation. Our algorithm proceeds in two stages: First we Merge all nodes that share the same prefix path from the root, and then we Split the nodes in the resulting tree that are too heterogeneous.2 Merge. We traverse the input tree in a top-down fashion, recursively merging siblings with the same label into one supernode. In the example, all siblings Item that have the same label are merged into Item*. The children of Item* are the union of children of each original Item node. Currently we only merge nodes at the same level. In the future, we may want to merge nodes that have the same 2

In practice, it would be more efficient to avoid merging nodes with completely heterogeneous internal structures. For clarity and generality, we present a two-step implementation.

396

E. Agichtein et al.

label, but occur in slightly different depths in the input tree. It is not clear if this is a desired behavior and most likely depends on the application. Split. In some XML document, sibling nodes with the same tag might have completely different structure. The goal of the split phase is to correct the merged nodes generated in the previous phase. This process allows us to distinguish between nodes that are semantically equivalent but happen to have missing information, and nodes having the same label, but which are actually heterogenous. The main criterion for splitting is whether there are disjoint subsets among the set of children of the merged node. In the example above, the merged node Item* contains 2 disjoint subsets ((Book, Title, Author, Year) and ((CD, Artist, Length, Name)). Thus, Item* would be split into 2 nodes, Item1 ∗ and Item2 ∗. The split procedure splits the nodes in the merged tree in a top-down fashion. At each node, the set of the children is examined. If the set contains at least two disjoint sets of children, the current node is split, and children are allocated accordingly. Finding the disjoint sets of children can be done efficiently by using the matrix shown in Figure 3. In this matrix, a “1” in the position i, j indicates that a node i contains a child with the label j. Using the matrix we can quickly find the connected (and disjoint) entries. This approach can be extended to splitting nodes that are weakly connected, and not completely disjoint. As we discussed, the purpose of merging is to create a more complete representation of the input document. We now describe how we use this representation to generate instance and attribute signatures that we will use subsequently for extracting the target relation from new, previously unseen documents. 3.2

Instance Signatures

Recall that our goal is to generate signatures that will allow us to find instance nodes in new documents with both structure and label names potentially different from the example documents. To support such flexibility, we need to capture both the position in the document and internal structure of the instance node. Further, the representation of the signature should be such as to allow finding the instance node in documents with structure and tag names different from the example documents observed in the Training stage. To accomplish this, we divide the document tree into four regions: 1. 2. 3. 4.

A: Ancestors of I (some number of levels up the tree). S: Siblings of I. C: Descendants of I. I: Self: Tag of instance node I itself.

The Siblings and the Ancestor nodes intuitively describe the position of the instance node in the document. The Descendants component allows us to describe the internal structure of the instance node. From these tree regions, we build the instance signature S of each example. We represent S as a set of vectors S = {A, S, C, I} where each vector


397

represents the respective tree region. More specifically, we represent each tree region using the tag names of the nodes in the region just like the vector-space model of information retrieval represents documents and queries [14]. Recall, that in this representation each unique tag name corresponds to a dimension in the vector space, and therefore the order of the tag names in the input document can be ignored. For example, the vector A generated as part of signature to represent the Ancestors region for the Item node in Figure 2, would contain terms Products, Books. In future work, we plan to investigate different weighting schemes for the terms in the vector. We could also use other reported techniques for representing XML structures in a high dimensional vector space, e.g., [15], but it is not clear which representation would work best for our application. Therefore, for our initial experiments we chose the minimal representation described above.

3.3

Attribute Signatures

So far we have discussed the characterization of the position and internal structure of the instance node. These signatures would allow us to find instance nodes despite variations in the structure of the document. Similarly, we want to support variations in the internal structure of the descendants of the instance node that map to the attribute values in the target relation. To capture the characteristics of the attributes of the target relation as they appear in the example documents, we build an attribute signature AS({D}, S{A, S, C, I}), for each attribute of the target relation, which consists of two components: – 1: Data signature D for the column over all known instances of the attribute to represent the distribution of values expected to be found in each attribute. (We can use a technique similar to the one described in [5].) – 2: Structure signature S(A, S, C, I), defined equivalently to the instance signature S, where the current instance node is used as the document root, and I refers to the set of tags of all elements in the example documents that map to this attribute. We will use these signatures to map descendants of instance nodes found in test documents to attributes in the table.

3.4

Signature Similarity

We now define the similarity of signatures that we will use to extract the target relation from new documents (Section 4). Intuitively, signatures of nodes that are located in similar positions in the document tree and have similar internal structures should have a high similarity value. For this, the similarity measure should consider all components of the signature.

398

E. Agichtein et al.

More formally, we define the Similarity between signatures Sigi (A, S, C, I) and Sigj (A, S, C, I) as: Similarity(Sigi , Sigj ) = wA · Sim(Ai , Aj ) + wS · Sim(Si , Sj ) +wC · Sim(Ci , Cj ) + wI · Sim(Ii , Ij )

(1)

a·b , or cosine of the angle between vectors a and where Sim(a, b) is defined as |a|·|b| b, which is a common way to quantify similarity in information retrieval. The Similarity function combines the Sim values between the positional and structural components of the signatures. Currently, all the components of the signature are weighted equally. However, depending on the application needs, the relative importance of different tree regions (as reflected by the weights of their respective vector components, e.g., wA ), may be tuned either by the user, or by using machine learning techniques. We define similarity between attribute signatures equivalently to the way we define similarity between instance signatures (Equation 1). The only difference is that we also add the similarity of the respective data components (vector D in the attribute signature definition). Relative importance of the structural and data components of AS has been studied previously in the context of relational schema mapping in [5].

4

Extraction

Having derived the sets of instance signatures (IS) and attribute signatures (AS), we proceed to extract the target relation from the new, previously unseen XML documents. The extraction proceeds in three stages. First, similar nodes of the input document are merged using the Merge algorithm (Section 3). Then, the instances nodes in the merged document representation are identified. Finally, the descendants of the discovered instance node are mapped to the attributes of the target relation. Identifying Instance Nodes. To discover the most likely instance node, we traverse the merged document tree in a bottom-up fashion. For each node X we generate the instance signature SX . We then compute the similarity of SX and each instance signature in IS that was generated during training. The score of X is then computed as the maximum of these similarities. The node with the highest score is then chosen as the candidate instance node. Mapping Attributes. For each target column Ti , we compute the similarity between the attribute signature ASi and the value of each descendent of the candidate instance node. Since merged instance nodes are expected to have small number of descendants, and the target table a relatively small number of attributes, this exhaustive approach is feasible. The mapping that maximizes the total similarity, computed as the product of similarities over all the target attributes, is chosen as the best mapping.


(a)

399

(b)

Fig. 4. The merged representation of the training document (a), and a test document (b) document with scores for each potential instance node. The node Book, the target instance node, is assigned the highest score by the system.

We can use the results of this step as feedback to the previous step of identifying instance nodes. For example, if we cannot find a satisfactory attribute mapping from the best candidate instance node, we then try the candidate instance node with the next highest score.

5

Preliminary Experiments

For our exploratory experiments we have considered a scenario where the target relation is NewBooks, described in Section 2. We want to extract this relation from XML documents such as may be exported by book publishers and vendors. We used the same tagged example document as shown in Figure 2 for training our system. The instance node in the example is the Item node, as displayed by our system prototype in Figure 4(a). Our system uses this example to generate the instance and attribute signatures to be used for extraction. The original XML document structure and the tag names were modified significantly to create test documents. A sample modified document is shown in the merged representation (Figure 4(b)). The instance node that contains all the information needed to extract a tuple for NewBooks now has the tag name Book, and the Products node now has a new tag name Publications. Additionally, internal structure of the instance node was changed. Such variations in structure would break standard XPath expressions that depend on the element tags in the document to find the instance node. However, our system prototype consistently assigned the highest score to the correct instance nodes in all tested variations, including the test structure shown in Figure 4(b). These exploratory results are encouraging and we are currently working on a more extensive empirical evaluation of our approach.

400

6

E. Agichtein et al.


We have presented a novel approach for partially supervised extraction of relations from XML documents without consistent structures (schemas) and terminologies (tag names). These XML documents may be derived from HTML pages, or obtained from exporting business objects to XML format. Extracting relations from schema-less XML documents using the approach presented in this paper can speed deployment of web-based systems and make their maintenance easier in presence of evolving schemas. We introduced the concept of the instance node, which is crucial in identifying the target node (object) that contains information for the attributes of the target relation. Second, we partitioned the neighboring nodes of the instance node into three different regions (siblings, ancestors, and descendants) and derived their respective signatures. We then defined a classification model based on spatial proximity of these tree regions to the instance node, each region having different semantic associations with the instance node. Third, the relative influence of these regions in finding the instance node in new documents can be simply adjusted by “turning a knob”, i.e., by changing the weights of the corresponding components in the similarity calculation. Finally, the Merge algorithm described in Section 4.1 enables our system to capture the notion of semantically equivalent XML nodes, which have stronger semantics than simply having the same tag names. We are currently exploring extending our model to allow the user to identify hint nodes - those nodes that do not contain information for attributes in the target relation, yet may indicate presence of the instance node. We also plan to experiment with different signature representations and weighting schemes, and alternative similarity definitions.

References 1. Gates, G.W.: The reduced nearest neighbor rule. In: IEEE Transactions on Information Theory. (1972) 2. Garofalakis, M., Gionis, A., Rastogi, R., Seshadri, S., Shim, K.: XTRACT: a system for extracting document type descriptors from XML documents. In: Proceedings of ACM SIGMOD Conference on Management of Data. (2000) 165–176 3. Goldman, R., Widom, J.: Dataguides: Enabling query formulation and optimization in semistructured databases. In: Twenty-Third International Conference on Very Large Data Bases. (1997) 436–445 4. Deutsch, A., Fern´ andez, M., Suciu, D.: Storing semi-structured data using STORED. In: SIGMOD. (1999) 5. Miller, R.J., Hernandez, M.A., Haas, L.M., Yan, L.L., Ho, C.T.H., Fagin, R., Popa, L.: The Clio project: Managing heterogeneity. SIGMOD Record 30 (2001) 78–83 6. Josifovski, V., Schwarz, P.: XML Wrapper - reuse of relational optimizer for querying XML data. Submitted for publication. (2002) 7. Knoblock, C.A., Lerman, K., Minton, S., Muslea, I.: Accurately and reliably extracting data from the web: A machine learning approach. IEEE Data Engineering Bulletin 23 (2000) 33–41


401

8. Grishman, R.: Information extraction: Techniques and challenges. In: Information Extraction (International Summer School SCIE-97), Springer-Verlag (1997) 9. Agichtein, E., Gravano, L.: Snowball: Extracting relations from large plain-text collections. In: Proceedings of the 5th ACM International Conference on Digital Libraries. (June 2000) 10. Arnaud Sahuguet, F.A.: Building light-weight wrappers for legacy web datasources using w4f. In: Proceedings of the International Conference on Very Large Databases (VLDB). (1999) 738–741 11. Liu, L., Pu, C., Han, W.: XWRAP: An XML-enabled wrapper construction system for web information sources. In: Proceedings of ICDE. (2000) 611–621 12. Baumgartner, R., Flesca, S., Gottlob, G.: Visual web information extraction with lixto. In: VLDB. (2001) 13. Crescenzi, V., Mecca, G., Merialdo, P.: Towards automatic data extraction from large web sites. In: VLDB. (2001) 14. Salton, G.: Automatic Text Processing: The transformation, analysis, and retrieval of information by computer. Addison-Wesley (1989) 15. Kha, D.D., Yoshikawa, M., Uemura, S.: An XML indexing structure with relative region coordinate. In: ICDE. (2001) 313–320

Extending XML Schema with Nonmonotonic Inheritance Guoren Wang and Mengchi Liu School of Computer Science, Carleton University, Canada {wanggr,mengchi}@scs.carleton.ca

Abstract. Nonmonotonic inheritance is a fundamental feature of object-oriented data models. In this paper, we extend XML Schema with nonmonotonic inheritance due to its powerful modeling ability to support multiple inheritance, overriding of elements or attributes inherited from super-elements, blocking of the inheritance of elements or attributes from super-elements, and conflict handling. Another key feature of objectoriented data models is polymorphism. We introduce it into XML to support polymorphic elements and polymorphic references.

1

Introduction

Several XML schema languages have been proposed, such as DTD [2], SOX [3], XML Schema [5] to constrain and define a class of XML documents. However, they do not support inheritance at all except for XML Schema and SOX [6]. Nonmonotonic multiple inheritance is a fundamental feature of objectoriented data models [4,7]. In object-oriented languages with multiple inheritance, a class may inherit attributes and methods from more than one superclass. For example, class TA might inherit attributes and methods directly from classes teacher and student. In a multiple inheritance hierarchy, users can explicitly override the inherited attributes or methods and block the inheritance of attributes or methods from superclasses [7]. One of the problems with multiple inheritance is that ambiguity may arise when the same attribute or method is defined in more than one superclass. Therefore, conflict resolution is important in object-oriented database systems with multiple inheritance and most systems use the superclass ordering to solve the conflicts [4,7]. In this paper, we extend XML Schema with nonmonotonic inheritance due to its powerful modeling ability to support multiple inheritance, overriding of elements or attributes inherited from super-elements, blocking of the inheritance of elements or attributes from super-elements, and conflict handling. Another key feature of object-oriented data models is polymorphism. We introduce it into XML to support polymorphic elements and polymorphic references.

Guoren Wang’s research is partially supported by NSFC of China(60273079) and Mengchi Liu’s research is partially supported by NSERC of Canada.


Extending XML Schema with Nonmonotonic Inheritance (01) (02) (03) (04) (05) (06) (07) (08) (09) (10) (11) (12) (13) (14) (15) (16) (17) (18) (19) (20) (21) (22) (23) (24) (25) (26)

403

<xsd:element name=“univ” type=“univType”/> <xsd:complexType name=“univType”> <xsd:sequence> <xsd:element name=“person” type=“personType” minOccurs=“0” maxOccurs=“’unbounded’/ > <xsd:element name=“course” type=“courseType” minOccurs=“0” maxOccurs=“’unbounded’/ > <xsd:complexType name=“personType”> <xsd:sequence> <xsd:element name=“name” type=“xsd:string”/> <xsd:element name=“birthdate” type=“xsd:date”/> <xsd:element name=“addr” type=“addrType”/> <xsd:element name=“homephone” type=“xsd:string”/> <xsd:attribute name=“pid” type=“xsd:ID” use=“required”/> <xsd:complexType name=“addrType”> <xsd:sequence> <xsd:element name=“street” type=“xsd:string”/> <xsd:element name=“city” type=“xsd:string”/> <xsd:element name=“state” type=“xsd:string”/> <xsd:element name=“zip” type=“xsd:string”/>

Fig. 1. Type definitions for elements univ, person and addr

2

Extensions to XML Schema

Figure 1 shows the type definitions for elements univ, person and addr. Although they have the same syntax as the original XML Schema, some of them (for example lines (04)-(07) of Figure 1) have different semantic constraints on XML instance documents due to the introduction of the polymorphic feature. Figure 2 shows the type definition for element student that inherits personType. Because the inheritance mechanism provided by XML Schema is not flexible and powerful, we extend it as follows: (1) In a type hierarchy, a subtype may have more than one supertype to support nonmonotic multiple inheritance. Therefore, the attribute base of the extension machanism is modified as bases, e.g. in line (03) of Figure 2. (2) In the original XML Schema, a subtype inherits all elements but not attributes from its supertype. Although attributes are different from elements, they are a special kind of information from users’ point of view. Therefore, in the Extended XML Schema, a subtype inherits not only elements but also attributes from its supertypes. Note that no other ID attributes are allowed to be declared in the subtype since pid is an ID attribute inherited by the subtype. (3) In the Extended XML Schema, a specific component element or attribute in the subtype may override the element or attribute defined in the supertype. For example, the component element addr in student inherited from personType is overridden with a new simple type, as shows in line (05) of Figure 2. Note that there is no special syntax extension for overriding of element and attribute. Sometimes, it is necessary to allow a subtype to block the inheritance of attributes and elements from its supertypes. For example, teachers usually prefer

404

G. Wang and M. Liu (01) <xsd:complexType name=“studentType”> (02) <xsd:complexContent> (03) <xsd:extension bases=“personType”> (04) <xsd:sequence> (05) <xsd:element name=“addr” type=“xsd:string”/> (06) <xsd:element name=“dept” type=“xsd:string”/> (07) <xsd:element name=“takes”> (08) <xsd:attribute name=“courses” type=“IDREFS” (09) target=“courseType” use=“implied”/> (10) (11) (12) <xsd:attribute name=“sno” type=“xsd:string” use=“required”/> (13) (14) (15)

Fig. 2. Type definition for element Student (01) <xsd:complexType name=“teacherType”> (02) <xsd:complexContent> (03) <xsd:extension bases=“personType”> (04) <xsd:sequence> (05) <xsd:element name=“workphone” type=“xsd:integer”/> (06) <xsd:element name=“salary” type=“xsd:float”/> (07) <xsd:element name=“dept” type=“xsd:string”/> (08) <xsd:element name=“teaches”> (09) <xsd:attribute name=“courses” type=“IDREFS” (10) target=“courseType” use=“implied”/> (11) (12) (13) <xsd:block from=“personType”> (14) <xsd:element name=“homephone”/> (15) (16) <xsd:attribute name=“tno” type=“xsd:string” use=“required”/> (17) (18) (19)

Fig. 3. Type definition for element Teacher

to use workphone rather than homephone as their contact phone. It is reasonable that in the definition of the subtype teacherType the inheritance of homepone is blocked from its supertype personType. Therefore, the blocking mechanism is introduced as shown in lines (13)-(15) of Figure 3. The blocking mechanism has an attribute from specifying from which type the inheritance is blocked and some components specifying attributes and elements to be blocked. Another extension to XML Schema is typing of IDREF and IDREFS. The literature [1] pointed out that neither XML Schema nor DTD support typing of IDREF and IDREFS. In this case, a reference may point to any kind of element instance. One cannot require a reference to point to only an expected kind of element instances. For example, it is possible that attribute @courses of the element takes in student references a person rather than a course. So, we extend attribute declaration for specifying the type of IDREF or IDREFS, for example, in lines (08)-(09) of Figure 2 and in lines (09)-(10) of Figure 3. In Figure 4, type TAType inherits elements and attributes from both supertypes studentType and teacherType. There are two conflicts to be resolved, since elements addr and dept are declared on both supertypes studentType and

Extending XML Schema with Nonmonotonic Inheritance

405

(01) <xsd:complexType name=“TAType”> (02) <xsd:complexContent> (03) <xsd:extension bases=“studentType teacherType”> (04) <xsd:rename> (05) <xsd:element name=“dept” from=“studentType” as=“student-dept”/> (06) <xsd:element name=“dept” from=“teacherType” as=“teacher-dept”/> (07) (08) <xsd:block from=“studentType”> (09) <xsd:element name=“addr”/> (10) <xsd:block> (11) (12) (13)

Fig. 4. Type definition for element TA (01) (02) (03) (04) (05) (06) (07) (08) (09) (10) (11) (12) (13) (14) (15) (16) (17) (18) (19) (20) (21) (22) (23) (24) (25) (26) (27) (28)

<xsd:complexType name=“courseType”> <xsd:sequence> <xsd:element name=“name” type=“xsd:string”/> <xsd:element name=“desc” type=“xsd:string”/> <xsd:element name=“takenBy”> <xsd:attribute name=“students” type=”xsd:IDREFS” target=“studentType” use=“implied”/> <xsd:attribute name=“teachers” type=”xsd:IDREFS” target=“teacherType” use=“implied”/> <xsd:attribute name=“cid” type=“xsd:ID” use=“required”/> <xsd:complexType name=“underCourseType”> <xsd:complexContent> <xsd:extension bases=“courseType”/> <xsd:complexType name=“gradCourseType”> <xsd:complexContent> <xsd:extension bases=“courseType”/> <xsd:element name=“student” type=“studentType”> <xsd:element name=“teacher” type=“teacherType”> <xsd:element name=“TA” type=“TAType”> <xsd:element name=“underCourse” type=“underCourseType”> <xsd:element name=“gradCourse” type=“gradCourseType”>

Fig. 5. Type definitions for elements course, underCourse and gradCourse

teacherType. In our Extended XML Schema, three ways can be used to handle conflicts. In the first way, a conflict resolution declaration is specified explicitly to indicate from which supertype an element or attribute is inherited, for example, the block construct in lines (08)-(10) of Figure 4 indicates that the declaration of addr is inherited from the supertype teacherType rather than from studentType. In the second way, the names of elements or attributes causing conflicts are explicitly renamed in the inheriting element declaration, for example, in the subtype TAType declaration, the rename construct in line (05) of Figure 4 renames element dept inherited from supertype studentType to student-dept while the rename construct in line (06) of Figure 4 from teacherType to teacherdept. Finally, if there is a conflict and there is no conflict resolution declaration, then the element or attribute is inherited from the supertype in the order the supertypes are listed in the extension construct of the type definition.

406

G. Wang and M. Liu Jaonne Barbosa 1965-04-07 <street> 310 University Ottawa <state> Ontario K1S 5B6 5073322 <student pid=“200”> Jones Gillmann 1976-02-25 708D Somerset St 6185708 <dept> Computer Science < takes courses=“CS200 CS300” /> Alley Srivastava 1957-06-26 <street> 56 Broson Ottawa <state> Ontario K2B 6M8 <workphone> 2314343 <salary> 1200.00 <salary> <dept> Computer Science Alice Bumbulis

1976-08-29 <street> 440 Albert Ottawa <state> Ontario K1R 6P6 2915318 <workphone> 2502600 <student-dept> CS SE Introduction to CS <desc> Continuing Education Introduction to DBS <desc> Basic concepts Introduction to SE <desc> Basic concepts DBMS <desc> Impl. Techniques

Fig. 6. An XML instance document

Figure 5 shows the type definitions for elements course(lines (01)-(13)), underCourse(lines (14)-(18)) and gradCourse(lines (19)-(23)), and other element declarations(lines (24)-(28)).

3

Extensions to XML Instance Document

Consider the examples described before, type personType has three direct or indirect subtypes, studentType, teacherType and TAType, and type courseType two direct subtypes, underCourseType and gradCourseType. When polymorphism is introduced into XML, an element instance of personType in a valid instance document can be substituted with an instance of elements of its subtypes, and the instance document should still be valid. If the type of an element has at least one subtype, then the element is polymorphic. For example, element person is polymorphic since type personType has three direct or indirect subtypes. the person element instance can be substituted by instances of student, teacher,

Extending XML Schema with Nonmonotonic Inheritance

407

or TA since their types all are subtypes of personType. Similarly, the instance of course can be substituted by instance of underCourse and gradCourse. The substituting element instances are referred to as polymorphic instances. From lines (04)-(07) of Figure 1, we can see that element univ can contain a number of person and course element instances; that is univ→person∗,course∗. Therefore, element univ can contain seven component element instances due to polymorphism: person, student, teacher, TA, course, underCourse and gradCourse instances. Now we extend XML Schema with polymorphic reference, which is similar to polymorphic element. A little bit complicated example for polymorphic reference is that a teacher may teach several courses including underCourses and gradCourses as well, see the definition of element teacher in Figure 3 and its instance in Figure 6. In the definition, teaches is an IDREFS to course. If polymorphic references are supported by the system(that is, teaches can also be used to reference to either underCourse or gradCourse elements as their types all are subtypes of the type of element course), the following six combinations are valid in the instance document: (1) a teacher teaches courses; (2) a teacher teaches underCourses; (3) a teacher teaches gradCourses; (4) a TA teaches courses; (5) a TA teaches underCourses; and (6) a TA teaches gradCourses. Polymorphic reference is introduced to meet the above requirements. An IDREF or IDREFS attribute of a given element can point to instance(s) of the substituting elements of the element. It is referred to as polymorphic references.

4

Conclusions

In this paper, we extend XML Schema to support the key object-oriented features such as nonmonotonic inheritance, overriding, blocking, conflict handling. Moreover, we extend XML instance document with polymorphism, including typing of references, polymorphic elements and polymorphic references.

References 1. Lewis, P.M., Bernstein, A., Kifer, M.: Databases and transaction processing: an application-oriented approach. Addison Wesley (2002) 2. Bray, T., Paoli, J., Sperberg-McQueen, C.M., Maler, E.: Extensible Markup Language (XML) 1.0. 2nd Edn. Available at http://www.w3.org/TR/REC-xml (2000) 3. Davidson, A., Fuchs, M., Hedin, M.: Schema for Object-Oriented XML 2.0. W3C Note. Available at http://www.w3.org/TR/NOTE-SOX (1999) 4. Dobbie, G., Topor, R.W.: Resolving Ambiguities caused by Multiple Inheritance. In: Proceedings of the 4th DOOD International Conf. Singapore (1995) 265–280 5. Fallside, D.C.: XML Schema Part 0: Primer. W3C Recommendation. Available at http://www.w3.org/TR/xmlschema-0/ (2001) 6. Lee, D., Chu, W.W.: Comparative analysis of six XML schema languages. ACM SIGMOD Record. 29 (2002) 117–151 7. Liu, M., Dobbie, G., Ling, T.W.: A logical foundation for deductive object-oriented databases. ACM Transaction on Database Systems. 27 (2002) 117–151

Author Index

Adam, Emmanuel 168 Agichtein, Eugene 390 Al-Muhammed, Muhammed Albert, Manoli 40 Atay, Mustafa 250, 366 Augusto, Juan C. 17

Iacob, Ionut E. 244

Badia, Antonio 330 Bergholtz, Maria 180 Bhowmick, Sourav S 355 Bohrer, Kathy 323 Bresciani, Paolo 217 Carroll, John M. 241 Chakravarthy, Sharma 273 Chalmeta, Ricardo 65 Chirkova, Rada 297 Dedene, Guido 105 Dekhtyar, Alex 311 Dindeleux, Régis 5 Donzelli, Paolo 217 Embley, David W. 244 Erwig, Martin 342 Estrella, Florida 5 Ferreira, Carla 17 Fettke, Peter 80 Fons, Joan 40 Fotouhi, Farshad 250, 366 Garzotto, Franca 92 Gaspard, Sébastien 5 Genero, Marcela 79, 118 Gerhardt, Joerg 390 Giorgini, Paolo 167 Grangel, Reyes 65 Gravell, Andy M. 17 Guo, Zhimao 261 Hawryszkiewycz, Igor T. 195 Henderson-Sellers, Brian 167, 195 Heuvel, Willem-Jan van den 3 Ho, C.T. Howard 390

311

Jaakkola, Hannu 129 Jacob, Jyoti 273 Jayaweera, Prasad 180 Jin, Min 285 Johannesson, Paul 180 Josifovski, Vanja 390 Kim, Kibum 241 Ko, Su-Jeong 29 Leuschel, Michael A. 17 Li, Ming 261 Lin, Aizhong 195 Liu, Mengchi 402 Liu, Xuan 323 Loos, Peter 80 Lu, Shiyong 250, 366 Madria, Sanjay 249, 355 Mandiau, René 168 Mayr, Heinrich C. 3 McClatchey, Richard 5 McLaughlin, Sean 323 Michiels, Cindy 105 Miranda, David 118 Nelson, Jim 79 Ng, Karen M.Y. 17 Orriëns, Bart 52 ´ Ortiz, Angel 65 Papazoglou, Mike P. 52 ´ Pastor, Oscar 40 Pelechano, Vicente 40 Perrone, Vito 92 Piattini, Mario 79, 118 Poels, Geert 79, 152 Poler, Ra´ ul 65 Prat, Nicolas 136 Psaila, Giuseppe 378 Rosson, Mary Beth

241

410

Author Index

Sachde, Alpa 273 Schonberg, Edith 323 Shah, Ashish 297 Shin, Byung-Joo 285 Si-Said Cherfi, Samira Singh, Moninder 323 Snoeck, Monique 105 Sun, Yezhou 250, 366

Wagner, Gerd 205 Wang, Guoren 402 Weiss, Michael 229 Wohed, Petia 180 136

Thalheim, Bernhard 129 Tian, Khoo Boon 355 Tulba, Florin 205

Xu, Zhengchuan Yang, Jian

261

52

Zhou, Aoying 261 Zhou, Shuigeng 261

Conceptual Modeling for Novel Application Domains: ER 2003 Workshops ECOMO, IWCMQ, AOIS, and XSDM, Chicago, IL, USA, October 13, 2003, Proceedings

Advanced Conceptual Modeling Techniques: ER 2002 Workshops - ECDM, MobIMod, IWCMQ, and eCOMO, Tampere, Finland, October 7-11, 2002, Proceedings

Embedded Software: Third International Conference, EMSOFT 2003, Philadelphia, PA, USA, October 13-15, 2003, Proceedings

Conceptual Modeling ER 2001

Conceptual Modeling - ER 2002

Conceptual Modeling - ER 2000: 19th International Conference on Conceptual Modeling, Salt Lake City, Utah, USA, October 9-12, 2000 Proceedings

Information and Communications Security: 5th International Conference, ICICS 2003, Huhehaote, China, October 10-13, 2003, Proceedings

php|architect (October 2003)

Scientific american (October 2003)

Perspectives in Conceptual Modeling: ER 2005 Workshop AOIS, BP-UML, CoMoGIS, eCOMO, and QoIS, Klagenfurt, Austria, October 24-28, 2005, Proceedings

Conceptual Modeling - ER 2006: 25th International Conference on Conceptual Modeling, Tucson, AZ, USA, November 6-9, 2006, Proceedings

Conceptual Modeling - ER '97: 16th International Conference on Conceptual Modeling, Los Angeles, CA, USA, November 3-5, 1997. Proceedings

Conceptual Modeling - ER '96: 15th International Conference on Conceptual Modeling, Cottbus, Germany, October 7 - 10, 1996. Proceedings: 15th

Advances in Conceptual Modeling: ER '99

Agile Universe 2003: Third XP and Second Agile Universe Conference, New Orleans, LA, USA, August 10-13, 2003, Proceedings

Conceptual Modeling - ER 2007, 26 conf

Advances in Conceptual Modeling.. ER '99

ER 2001 Workshops

2003)

2003)

Computer Performance Evaluation. Modelling Techniques and Tools: 13th International Conference, TOOLS 2003, Urbana, IL, USA, September 2-5, 2003, Proceedings

2003)

2003)

Practical Aspects of Declarative Languages: 5th International Symposium, PADL 2003, New Orleans, LA, USA, January 13-14, 2003, Proceedings

2003)

2003)

Conceptual Modeling for E-Business and the Web: ER 2000 Workshops on Conceptual Modeling Approaches for E-Business and the World Wide Web and

Agent-Oriented Information Systems, 5 conf., AOIS 2003

High Performance Computing: 5th International Symposium, ISHPC 2003, Tokyo-Odaiba, Japan, October 20-22, 2003, Proceedings

Inductive Logic Programming: 13th International Conference, ILP 2003, Szeged, Hungary, September 29 - October 1, 2003, Proceedings

Distributed Computing: 17th International Conference, DISC 2003, Sorrento, Italy, October 1-3, 2003, Proceedings

Conceptual Modeling for Novel Application Domains: ER 2003 Workshops ECOMO, IWCMQ, AOIS, and XSDM, Chicago, IL, USA, October 13, 2003, Proceedings

Advanced Conceptual Modeling Techniques: ER 2002 Workshops - ECDM, MobIMod, IWCMQ, and eCOMO, Tampere, Finland, October 7-11, 2002, Proceedings

Embedded Software: Third International Conference, EMSOFT 2003, Philadelphia, PA, USA, October 13-15, 2003, Proceedings

Conceptual Modeling ER 2001

Conceptual Modeling - ER 2002

Conceptual Modeling - ER 2000: 19th International Conference on Conceptual Modeling, Salt Lake City, Utah, USA, October 9-12, 2000 Proceedings

Information and Communications Security: 5th International Conference, ICICS 2003, Huhehaote, China, October 10-13, 2003, Proceedings

php|architect (October 2003)

Scientific american (October 2003)

Perspectives in Conceptual Modeling: ER 2005 Workshop AOIS, BP-UML, CoMoGIS, eCOMO, and QoIS, Klagenfurt, Austria, October 24-28, 2005, Proceedings

Conceptual Modeling - ER 2006: 25th International Conference on Conceptual Modeling, Tucson, AZ, USA, November 6-9, 2006, Proceedings

Conceptual Modeling - ER '97: 16th International Conference on Conceptual Modeling, Los Angeles, CA, USA, November 3-5, 1997. Proceedings

Conceptual Modeling - ER '96: 15th International Conference on Conceptual Modeling, Cottbus, Germany, October 7 - 10, 1996. Proceedings: 15th

Advances in Conceptual Modeling: ER '99

Agile Universe 2003: Third XP and Second Agile Universe Conference, New Orleans, LA, USA, August 10-13, 2003, Proceedings

Conceptual Modeling - ER 2007, 26 conf

Advances in Conceptual Modeling.. ER '99

ER 2001 Workshops

2003)

2003)

Computer Performance Evaluation. Modelling Techniques and Tools: 13th International Conference, TOOLS 2003, Urbana, IL, USA, September 2-5, 2003, Proceedings

2003)

2003)

Practical Aspects of Declarative Languages: 5th International Symposium, PADL 2003, New Orleans, LA, USA, January 13-14, 2003, Proceedings

2003)

2003)

Conceptual Modeling for E-Business and the Web: ER 2000 Workshops on Conceptual Modeling Approaches for E-Business and the World Wide Web and

Agent-Oriented Information Systems, 5 conf., AOIS 2003

High Performance Computing: 5th International Symposium, ISHPC 2003, Tokyo-Odaiba, Japan, October 20-22, 2003, Proceedings

Inductive Logic Programming: 13th International Conference, ILP 2003, Szeged, Hungary, September 29 - October 1, 2003, Proceedings

Distributed Computing: 17th International Conference, DISC 2003, Sorrento, Italy, October 1-3, 2003, Proceedings

Recommend Documents